≡ Menu

Robots.txt for WordPress blog

There is a nice plugin that help you do “noindex, nofollow” to certain pages to remove lots of duplicate content. These plug ins don’t cover pages that are not in WordPress. Here is an example WordPress robots.txt.

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/

I went from 100 pages in the supplemental to 9. This is not for all blogs. Some blogs have different URL’s. Check your supplemental index by typing

site:domain.com –view ***

This will show the pages you have in the supplemental index.

Comments on this entry are closed.

  • Sura Design April 13, 2007, 10:01 am

    Hi,

    I hope you enjoyed SES New York. I have tried this command a few times since the last time you posted this and it seems to no longer work. Not sure if something changed.