How do I use a Robots file?

[seointerest]

Hello Sir,

How do I use robots file. Do you have any example for the same?
I understand that robots.txt is a file in the main directory of a website which tells
the search engine spiders where they are allowed to go. How do I tell them this? Is there high level programming to be done?

[nigelzephyr]

these might help you
Here are a few good online references for information on the Robots Exclusion Protocol.

1. WebmasterWorld - robots.txt Validation
2. WebmasterWorld - robots.txt Forum
3. The Web Robots Pages - Robots Exclusion Protocol
4. Three Easy Ways to Reduce robots.txt Code Bloat
5. The Web Robots Pages - Database of Web Robots