This happens to so many people: you make a website and they don’t want just anyone to be able to access them. Unfortunately, once your website goes live, many search engines like MSN, Google, Yahoo, etc., are already way ahead of you in terms of indexing your work.

You can, however, create a command that will tell search engines to not include your website. First you have to create a

Continue reading " Blocking Search Engines with robots.txt "

The best use we’ve seen in a long time for robots.txt is to avoid indexing pages on your website that are ambiguous. The concept is really quite simple: You’ve got a lot of pages on your site, but some of those pages really don’t offer anything of value, or the keywords that they use don’t match the general subject matter on the rest of the site.

The most common solution to this problem is to tweak the page a

Continue reading " robots.txt and ambiguous pages "

Robots.txt what is it and how should you use it

On your website, you will want a top level file called “robots.txt.”  This file is used to specifically give indexing permission to web spiders, or to take it away.  As a spider begin to index your site, it will first scan the robots.txt file, and take note of all files and folders that are forbidden.  This can be used to prevent search engine spiders from

Continue reading " what is robots.txt and how to use it "