Robots.txt for WordPress blog
March 30th, 2007
There is a nice plugin that help you do “noindex, nofollow” to certain pages to remove lots of duplicate content. These plug ins don’t cover pages that are not in WordPress. Here is an example WordPress robots.txt.
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
I went from 100 pages in the supplemental to 9. This is not for all blogs. Some blogs have different URL’s. Check your supplemental index by typing
site:domain.com –view ***
This will show the pages you have in the supplemental index.
