Robot.txt : Subfolder exclusion

Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Robot.txt : Subfolder exclusion

Postby sorchak » Mon Nov 01, 2010 4:14 pm

Hi,
Is there a single command to prevent a crawler from going into any subfolder? The reason I am asking this is, it seems the google is picking up subfolder that don't even exist on my site. So just excluding each folder is not enough , I need a general command.

Thanks
sorchak
 
Posts: 1
Joined: Mon Nov 01, 2010 3:53 pm

Return to Search Engine Spiders

Who is online

Users browsing this forum: No registered users and 0 guests

cron