Robots.txt what is it and how should you use it
On your website, you will want a top level file called “robots.txt.” This file is used to specifically give indexing permission to web spiders, or to take it away. As a spider begin to index your site, it will first scan the robots.txt file, and take note of all files and folders that are forbidden. This can be used to prevent search engine spiders from