What is a Spider

The way that search engines index your website is through the use
of computer routines which access and catalog key parts of the site.
These routines are known as spiders, because they are said to “crawl”
the “web.”

Many people have the misconception that a spider is something like
a person, and that it loads a web page into a browser just the same
as a human might. But with a little thought, we can see that this
isn’t likely or even marginally possible. And this is where amateur
SEO designers often go wrong.

You see, a spider isn’t going to see the page the same way as a
human visitor, it is going to see the page as though it were loaded
into a text editor such as notepad. Spiders will analyze a web site
looking for key items on the pages, and from those items will build
the data needed to rank your site in a search engine algorithm.

What are spiders looking for? They pay close attention to the
title and meta tags, and, of course, count the number of times your
definitive keywords are used in relation to the amount of total text.
But a spider also looks at other important tags. the “<a
href” tag tells the spider that you are linking to another site,
and it will follow that link to determine how well your website is in
line with the content of the other, and vice versa. This is known as
determining relevance, and is the single biggest factor in how well a
search engine will rate your website.

So remember, even though spiders are referred to as though they
were real things, the truth is that a spider is nothing more than a
programming script. The script processes all of the data it finds on
the site, omits information it is instructed to ignore in robots.txt
and .htaccess, and then creates a catalog entry based on the
information it has collected.

Article written by SEOnotepad.com