robots.txt – A System Administrator’s Review
As you know that Robots are the programs that traverse the Web sites automatically. Search engines such as Google use them to index the web content. The mechanism of the robots is simple, when they go to visit any site, they first look for robot.txt file and see which area are not allowed by site owners to visit.
