The best use we’ve seen in a long time for robots.txt is to avoid indexing pages on your website that are ambiguous. The concept is really quite simple: You’ve got a lot of pages on your site, but some of those pages really don’t offer anything of value, or the keywords that they use don’t match the general subject matter on the rest of the site.
The most common solution to this problem is to tweak the page a bit, add some content that is relevant to the rest of the site, change the keywords, and move on. Most websites, after all, aren’t worried about one or two inconspicuous pages that don’t quite fit in.
The next possible solution, and one that is definitely not something you should do, is to fake it. Use the same keywords that you have found useful on the rest of the site, and hope that search engines don’t notice you’ve served up a helping of useless pseudo spam.
But another solution, and one that is actually handed to us by Google itself, is to use the robots.txt to prevent crawling those pages completely. Simply add that page to your robots.txt as a no index, no follow link, and forget about it.
With some creative use of robots.txt, you can make sure that all of the pages on your site conform to the overall site relevance, at least as far as search engines are concerned. Once visitors arrive on the site, give them ample opportunity to explore those non-indexed pages, but maintain a level of relevant segregation, and only allow the spiders to crawl pages which add to the reputation you want to have.
Article written by SEOnotepad.com
1 Comment Received
March 11th, 2009 @2:09 pm
denying access to google? sounds good on paper but i have reservations about the idea of google skipping parts of your websites. isnt it better to have them pages increase their value to google in terms of keywords?
Sorry the comment area are closed for non registered users