Robots.txt what is it and how should you use it
On your website, you will want a top level file called “robots.txt.” This file is used to specifically give indexing permission to web spiders, or to take it away. As a spider begin to index your site, it will first scan the robots.txt file, and take note of all files and folders that are forbidden. This can be used to prevent search engine spiders from indexing pages that would not be helpful or relevant.
Just because you have given instructions for a robot does not necessarily mean the robot will adhere to the command. For search engines, the commands in robots.txt are followed, but other spiders may not even check for a robots.txt before indexing the site. This is common for email skimmer robots, and other information mining uses. But for SEO purposes, the robots.txt is an excellent tool.
The robots.txt file is used to allow or disallow certain robots, or ALL robots from accessing the entire site, or specific files or directories. Fir instance, to prevent your cgi-bin directory from being indexed, you would add lines to your robots.txt file that looks like this:
User-agent: *
Disallow: /cgi-bin/
The asterisk (*) is a wildcard, and means ALL robots. The next line specificies that the /cgi-bin/ folder is the folder to which permission is not allowed. All other folder and files on the site are still available for indexing. For search engines, blocking your scripts folder can be a benefit in many cases, as well as any image folders.
Remember, disallowing the folder is a courtesy being requested. If the folder exists on a public domain, many spiders will be indexing the site regularly, without ever consulting the robots.txt for permissions at all. If you are trying to completely block a portion of your site, it is more effective to use the .htaccess file, which directs how the server will handle requests. So robots.txt is used to tell specific robots what you would prefer they ignored. It is very useful to keep certain files and folders from being indexed by search engines, which can have a large effect on how your site is rated and ranked.
Article written by SEOnotepad.com
No Comment Received
Sorry the comment area are closed for non registered users