Robots.TXT - A file on a web site in the root directory that is used to control which spiders have access to which pages within a website. When a spider or robot connects to a website, it checks for the presence of a robots.txt file. Only spiders that adhere to the Robots Exclusion Standard will obey a robots.txt command file.
No comments:
Post a Comment