Robots.txt

You can put a text file in the root of your website called robots.txt. This files tells web bots what to do. You can tell them what pages or directories not to visit. You can also be specific about what bots are allowed where. Some search engines even allow you to tell them how fast to crawl your site. Here is an example robots.txt

User-agent: *
Disallow: /images/
Disallow: /page.html
Disallow: /private/
Disallow: /cgi-bin/
Disallow: /duplacate-content/

User-agent: LarbinWebCrawler #this tells the LarbinWebCrawler bot to not come to your site
Disallow:

User-Agent: msnbot #tells msn to wait 20 seconds before it visits again
crawl-Delay: 20