SEO articles & Help - Search Engine Optimization & Marketing » robots.txt

Blocking Search Engines with robots.txt

SEOnotepad — Sat, 28 Mar 2009 00:08:00 +0000

This happens to so many people: you make a website and they don’t want just anyone to be able to access them. Unfortunately, once your website goes live, many search engines like MSN, Google, Yahoo, etc., are already way ahead of you in terms of indexing your work.

You can, however, create a command that will tell search engines to not include your website. First you have to create a

robots.txt and ambiguous pages

SEOnotepad — Fri, 09 Jan 2009 10:53:23 +0000

The best use we’ve seen in a long time for robots.txt is to avoid indexing pages on your website that are ambiguous. The concept is really quite simple: You’ve got a lot of pages on your site, but some of those pages really don’t offer anything of value, or the keywords that they use don’t match the general subject matter on the rest of the site.

The most common solution to this problem is to tweak the page a

what is robots.txt and how to use it

SEOnotepad — Thu, 16 Oct 2008 05:17:30 +0000

Robots.txt what is it and how should you use it

On your website, you will want a top level file called “robots.txt.” This file is used to specifically give indexing permission to web spiders, or to take it away. As a spider begin to index your site, it will first scan the robots.txt file, and take note of all files and folders that are forbidden. This can be used to prevent search engine spiders from