A: A web owner will use a robots.txt file to essentially give instructions about their website to web robots. The first thing a robot will check when it is visiting a websit is the robots.txt file. You should know that this file can be viewed by anyone, so anyone can easily find out what sections on your server you dont want the robots to use. Another interesting fact is some robots especially malware robots that scan the web for security and email harvesters ignore your robots.txt file. So do not use robots.txt file to hide information. This file must be placed in the most top level directory of your web server. For example if you have a public_html folder it should be in there. For more information regarding creating and how to use a robots.txt file please see
robotstxt.org.