A while back I wrote a post entitled “The Key To Getting Good Search Rankings” where I wrote about some good ways to boost your rankings that extra bit. At the end of the post I mentioned that I set up my robots.txt file to get rid of duplicate wordpress content.
At the time I thought about making it a permanent change, but recently I decided to change it some more. Here are the results of that little “experiment”, and why I am going back.
Traffic from Google (I still get crap from Yahoo and MSN), increased over 30%. However, very few posts ranked well and they only did so for a few days. After that their rankings diminished. I believe this occurred for one reason, once the post was kicked off the home page no other links were pointing to it. Apparently Google saw that and thought “crap, no links? Must be worthless!”
The main reason I kept the SEs from indexing the archive and feed pages was to get rid of any extra duplicate content. The problem is I forgot about one important thing, the archive pages had a ton of unique content when compared to the post’s page, so duplicate content was not an issue for the archives (for the feeds it was, as explained below).
I was kind of wrong (yeah yeah, we all get it wrong from time to time).
So, I have decided to allow Google to reindex the archive pages once more. The feed pages will still be disallowed, since there is absolutely no reason to allow those. I suspect that the archive and categories pages will get reindexed in a week or so and those pages will once again get good rankings.
However, the question still remains, why did my search traffic increase?
I suspect it occurred because of the feeds being disallowed, not the archives. Very few of my posts that are older than a month or so have any comments, so the feeds were just the content. Obviously duplicate content became a big issue with those specific feeds.
I will continue doing some more robots.txt experiments and of course I will continue writing about any changes, especially if they were positive.
Please subscribe, or else I will cry. Do you really want to make a programmer cry?
Leave a Reply