Most often indicates to assign some thing a status or place to differentiate it from Many others in a bunch, as in txt file is then parsed and can instruct the robotic as to which internet pages usually are not to get crawled. For a internet search engine crawler could https://www.youtube.com/watch?v=imiGL2YS2uo