The best use we’ve seen in a long time for robots.txt is to avoid indexing pages on your website that are ambiguous. The concept is really quite simple: You’ve got a lot of pages on your site, but some of those pages really don’t offer anything of value, or the keywords that they use don’t match the general subject matter on the rest of the site.
The most common solution to this problem is to tweak the page a
