Robots.txt for WordPress blog
There is a nice plugin that help you do “noindex, nofollow” to certain pages to remove lots of duplicate content. These plug ins donâ€™t cover pages that are not in WordPress. Here is an example WordPress robots.txt.
I went from 100 pages in the supplemental to 9. This is not for all blogs. Some blogs have different URLâ€™s. Check your supplemental index by typing
site:domain.com â€“view ***
This will show the pages you have in the supplemental index.