A file that indicates to search engine spiders which pages and sections of a website they are not allowed to crawl
