robot.txt

Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

robot.txt

Postby seoboy » Sat Aug 22, 2009 9:24 pm

I was Checking my robots.txt file on my site, and i can see my 5 static pages i have, are restricted from being crawled or indexed via dissallow of pages.php

I want all the pages indexed, and seen by spiders, but i don't know if i have to modify robots.txt file?
Any advice?
User avatar
seoboy
FORUM ADDICT
 
Posts: 57
Joined: Sat Aug 22, 2009 8:17 pm

Re: robot.txt

Postby sweetpink » Fri Aug 28, 2009 1:49 pm

Hey Seoboy, making or modifying robots.txt is easy.

Here is the code:

Please note: This will allow all robots to crawl and index all files.

This allows all robots to crawl all files.
User-agent: *
Disallow:

Or:

This allows all robots to crawl all files except the images file.
User-agent: *
Disallow: /images/

This allows all robots to crawl all files except the images file and the stats file.
User-agent: *
Disallow: /images/
Disallow: /stats/
User avatar
sweetpink
FORUM ADDICT
 
Posts: 60
Joined: Sat Aug 08, 2009 4:41 pm


Return to Search Engine Spiders

Who is online

Users browsing this forum: No registered users and 1 guest

cron