Benefits of a Robots.txt
The robots.txt file is a simple text file (no HTML) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters utilize this file to help the search engines index the content of their websites. If webmasters can tell the search…
