A useful post and advice from Google how to handle duplicate content:
“We now recommend not blocking access to duplicate content on your website, whether with a robots.txt file or other methods” John Mueller
John also goes on to say some good advice about how to handle duplicate content on your own site:
- Recognize duplicate content on your website.
- Determine your preferred URLs.
- Be consistent within your website.
- Apply 301 permanent redirects where necessary and possible.
- Implement the rel=”canonical” link element on your pages where you can. (Note – Soon we’ll be able to use the Canonical Tag accross multiple sites/domains too.)
- Use the URL parameter handling tool in Google Webmaster Tools where possible.
They have not updated their webmaster guidelines on duplicate content as yet.
Consider blocking pages from indexing: Rather than letting Google’s algorithms determine the “best” version of a document, you may wish to help guide us to your preferred version. For instance, if you don’t want us to index the printer versions of your site’s articles, disallow those directories or make use of regular expressions in your robots.txt file. Google
Here’s some recent official Google advice for duplicate content accross multiple sites (and some internal advice):
I’m starting to think Google keeps changing it’s mind just to keep SEOs busy — they’ll spend most of their time ‘repairing’ websites rather than optimizing them. I worry about what happens when the canonical element is no longer supported..what then? Just a thought…
@Angie yeh I know what you mean :)
OK, OK I’ll also take this recommendation as a hint, not as a directive. Specially because of this: “If you allow us to crawl these URLs, Googlebot will learn rules to identify duplicates just by looking at the URL and should largely avoid unnecessary recrawls in any case. In cases where duplicate content still leads to us crawling too much of your website, you can also adjust the crawl rate setting in Webmaster Tools.
Shame – robots.txt was a nice easy way to deal with some of these problems if you’d inherited a poorly designed site. I suppose 301s are just as easy.
Well I’m going to to continue blocking duplicate content. First, Google may be the biggest and most important search engine to focus on, however they’re not the only ones. A while back they made a deal with Adobe to claim that Flash was now more SEO friendly. Anyone who actually used that financially motivated marketing hype as an excuse to change their anti-Flash views was a fool. And Google does not state anywhere (nor can they nor will they) that content you want kept out of the SERPs is guaranteed to be kept out by the new recommended methods. In fact, they say “In cases where duplicate content still leads to us crawling too much of your website, you can also adjust the crawl rate setting in Webmaster Tools.” Well guess what – that is the most arcane and pitiful directive I’ve seen in a long time. If I want to keep the googlebot’s grubby digital hands off of certain content, I’ll do it the intelligent, proven way and not rely on Google hacks and their guestimating algorithm…
I sometimes wonder if Google would like to do away with robots.txt altogether so they can have full access to everywhere on the net. This latest advice seems lacking in any sort of considered detail (like most of their advice!). For instance if a site has pdf downloads that are largely identical to exisiting html pages then robots.txt is the only normal way to stop them being spidered and seen as duplicates- 301s aren’t relevant, canonical tags aren’t relevant, robots meta tags aren’t relevant.