Web Robots are programs that traverse the Web automatically. They are also called Web Wanderers, Web Crawlers, and Spiders. Search Engines are the main Web Robots. Some Web Robots look for a file named robots.txt on your web server to see what and where they should look for content and files on your web server. Some Web Robots ignore this file.