But you can use the robots.txt file to prevent the necessary duplicated content from being indexed, and therefore avoid these issues and search engine imposed penalties. If these extra pages duplicates the content elsewhere on your site, you can find yourself being penalized by some of the search engines. Maybe you even have specific landing pages that are intended for use with a variety of advertising campaigns or marketing promotions. For instance, let's say you have a high speed version of your pages with video and animations, a low speed version with simple images, and a mobile version of your site for phones. A robots file can even prevent penalties associated with duplicate content.Without the robots.txt file it gets harder to wade through all of these missing robots.txt file not found errors to find genuine errors that a webmaster needs to address each month. When your website doesn't deliver a robots file for the search engines, a '404 Not Found' error is generated each time. The robots.txt file cleans up your hosting/server logs because an error isn't generated every time a search engine visits your site and requests the robots.txt, which can happen often in one day (especially if you are a good site promoter). They would actually have to visit your site and load the url in question instead of accidentally running into it on Google, Bing, Yahoo or Teoma. Although a robots.txt file shouldn't be thought of as security, It does give you a very basic level of protection in that it will keep people from easily finding stuff you don't want easily accessible via the search engines that would otherwise find it.This assists your server as well as the search engine by reducing the overhead for the job. A robots.txt file can save on your bandwidth because when compliant spiders comes to visit, they won't crawl areas where there is no useful information (your cgi-bin, images, etc).There are probably more than the following half-dozen reasons to use a robots file to direct search engine crawler traffic on your website. The problem is that there are compliant bots that are run by respectable search engines and directories (ie: Googlebot, Bingbot, Slurp, Teoma, Robozilla), and then there are scammer bots that troll websites looking for active mail addresses and contact info to use to scam people out of their money with spam. The robots text file can function as a means to limit access to personal, private, sensitive or unnecessary directories, documents and related digital materials, but by no means should it be used as a means of securing said directories, documents or materials. The robots.txt file is a simple text file used to direct compliant robots to the important parts of your website, as well as keep them out of private areas.

Robotize file