SEO articles & Help - Search Engine Optimization & Marketing » robots http://www.seonotepad.com Taking SEO & SEM to the next level Fri, 27 Aug 2010 19:13:55 +0000 http://wordpress.org/?v=2.8.4 en hourly 1 what is robots.txt and how to use it http://www.seonotepad.com/seo/robots-txt-howto-use/ http://www.seonotepad.com/seo/robots-txt-howto-use/#comments Thu, 16 Oct 2008 05:17:30 +0000 SEOnotepad http://www.seonotepad.com/?p=79 Robots.txt what is it and how should you use it

On your website, you will want a top level file called “robots.txt.”  This file is used to specifically give indexing permission to web spiders, or to take it away.  As a spider begin to index your site, it will first scan the robots.txt file, and take note of all files and folders that are forbidden.  This can be used to prevent search engine spiders from

]]> Robots.txt what is it and how should you use it

On your website, you will want a top level file called “robots.txt.”  This file is used to specifically give indexing permission to web spiders, or to take it away.  As a spider begin to index your site, it will first scan the robots.txt file, and take note of all files and folders that are forbidden.  This can be used to prevent search engine spiders from

]]> http://www.seonotepad.com/seo/robots-txt-howto-use/feed/ 0