How does spiders index?

Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.
Post a reply

How does spiders index?

Postby charlie » Mon Sep 07, 2009 2:55 am

Hello all!

How does a Search Engine Spider index a website?
I guess it starts with the domains index page and then searches all the link's in it. And what if you have a subdomain that not contain a link to the main site?
Will be indexed that page?
User avatar
charlie
EIS JUNIOR
 
Posts: 22
Joined: Sun Aug 30, 2009 1:20 am
Top

Re: How does spiders index?

Postby paul755 » Mon Sep 07, 2009 2:58 am

Thats right!

It's funny how some pages take ages to get indexed, while others are indexed from a link on some test page that was only online for 15 seconds or so.

robots.txt is a good start :!:
User avatar
paul755
FORUM ADDICT
 
Posts: 60
Joined: Mon Aug 24, 2009 9:35 pm
Top

Re: How does spiders index?

Postby richardsmith » Sat Jan 28, 2012 3:21 am

Spiders are one type of the spider programme which follow the link of the website and gather information for search engine and internet users which called index. These are only computer programs, they aren't always the smartest of creatures and may get caught in endless loops built by dynamically created web-pages.

Thanks,
Online Reputation Management 8-)
richardsmith
EIS JUNIOR
 
Posts: 28
Joined: Wed Oct 20, 2010 5:39 am
Top
Post a reply

Return to Search Engine Spiders

Who is online

Users browsing this forum: No registered users and 3 guests

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Copyright © 2008-2011 Extreme Seo Internet Solution
Certified Seo Company

phpBB SEO
cron