How to block all robots from a certain folder??

[jconnel]

Sometimes you have certain sub directories that you don't want to be caught by the search engines. All major search engines that follow the REP , i.e. Yahoo, MSN and Google, and the promise not to index folders and files that are specifically mentioned on your robots.txt file. Here is an example to exclude subfolders /private/ /photos-saved/ and /images-fullsize/ from the search index,

User-agent: *
Disallow: /private/
Disallow: /photos-saved/
Disallow: /images-fullsize/
By using User-agent: * it referes to the fact that all search engines should obey this request.