03.30.07

Robots.txt for WordPress blog

Posted in blogging, Tutorials, Google at 7:00 am

There is a nice plugin that help you do “noindex, nofollow” to certain pages to remove lots of duplicate content. These plug ins don’t cover pages that are not in WordPress. Here is an example WordPress robots.txt.

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/

I went from 100 pages in the supplemental to 9. This is not for all blogs. Some blogs have different URL’s. Check your supplemental index by typing

site:domain.com –view ***

This will show the pages you have in the supplemental index.

Permalink

5 Comments »

Sura Design said,

04.01.07 at 11:36 am

I tried this on my site and found 7 pages in the supplemental index. Mostly swf (flash) files. I’m not sure if there is anything I can do about those pages.

Also I copied and pasted the (Site:domain.com –view ***) from this post and it did not work.. so I tried various combinations and eventually realized it was the “S” being capital. I changed it to lowercase and it worked.

Thanks for the useful search modifier.
ogletree said,

04.01.07 at 11:50 am

Interesting I did not know the site: command was case sensitive. I type my posts in word and I guess word capitalized it for me. Thanks for the heads up.
Sura Design said,

04.13.07 at 10:01 am

Hi,

I hope you enjoyed SES New York. I have tried this command a few times since the last time you posted this and it seems to no longer work. Not sure if something changed.
ogletree said,

04.14.07 at 12:14 pm

Try this
AskApache said,

11.06.07 at 3:02 am

I finally got my site indexed near perfectly in google with this wordpress robots.txt example