
Robots.txt for WordPress – How To Manage Your Robots.txt File The robots.txt is a simple text file used to tell search engine robots and spiders which pages on your web site should be crawled and indexed. Pages you don’t want crawled include pages such as junk pages and duplicate content. Controlling what content is either Continue reading »





