SEO News

Introduction

In May 2013, Matt Cutts officially announced a new version of Google’s Penguin algorithm was set to roll out.

Google didn’t waste any time with the roll out and around a week after Matt Cutts’ announcement, the formal confirmation came that Penguin 2.0 was officially out in the wild.

With Penguin, Google is attempting a fairly ambitious effort to cut down on unnatural links, known as ‘web spam’. At the same time, there was an under the radar update referred to as ‘Domain Clustering’.

This update was never officially announced, although Matt Cutts did make a webmaster video entitled ‘Why does Google show multiple results from the same domain?’.

In the video Matt talks about users being unhappy about seeing too many results from a single domain, stating “Once you’ve seen a cluster of four results from a given domain, as you click on subsequent pages we won’t show you results again”.

This seemed to be enough of an issue to kick the search quality team into action and roll out the next generation of the domain diversity algorithm.

Although it was announced that the new algorithm change would be rolling out relatively soon, there was no formal announcement to say if the domain clustering aspect actually happened, therefore we decided to dig deep into the data to investigate whether this was a hidden release.

Our Analysis

We extracted the top 100 daily rankings for 489 major keywords (in Google UK), across multiple industries, giving us the foundations to dig a little deeper. Firstly, we selected a few highly searched generic keywords to see if there were any identifiable changes.

 
Our initial findings were quite surprising. Nobody would have expected Google to make such a drastic change with this update, seeding nearly 100 (97 in most cases) completely unique domains into each search query. Almost an entirely unique index!

Since a few keywords began to show a small amount of insight, we decided to see how this appeared across our entire keyword set. The chart below looks at the average number of unique domains across the first 10 pages of Google’s index.

These results clarify that this change definitely wasn’t limited to a handful of keywords. Pre-penguin indicates that the algorithm didn’t previously have a spread of results across multiple domains, therefore the average was relatively low.

We’re now looking at a completely transformed set of results, with the average increasing as the visitor navigates to deeper pages. The average change on deeper pages is extremely interesting, with Google previously serving 2.7 unique domains on page nine, now becoming 7.4!

Below, we can see how this changed on a page-by-page basis for each of our initial sample terms. The main insight being the minor (if any) change within Google’s first few pages, with the page-by-page change becoming more aggressive the deeper you delve into the index.

The evidence is pretty conclusive that the update has been rolled out, although it is not known why there was no formal announcement.

That being said, we are able to use this analysis to confirm Google no longer wishes to show too many resources from the same domain and instead is moving into a more diverse set of results for all queries.

Key findings and facts so far

• On average there are now 34.7 unique domains per 100 results as opposed to 19.3, meaning a number of terms weren’t fully impacted, yet.
• 1,323 sites lost all their results. Only nine of these started with 10 or more results and 121 with three or more, possibly a combined blow along with the Penguin 2.0 update.
• 451 sites lost more than 50% of their results.
• 52% of the current index is occupied by new domains with 8,892 domains that didn’t rank now displaying.

As this update caused a significant number of domains to be trimmed from search queries, the obvious question to ask is ‘who was impacted?’.

So, who are the potential losers?

The graph below shows the domains with the biggest change, essentially who lost the most results across the analysed keywords.

Based on the amount of change the biggest loser is Money.co.uk, losing 830 results. Surprisingly, all of the listed domains are what you’d class as ‘big brands’, and have mainly lost ground within the deeper pages of the index.

For instance, for the term ‘best travel insurance’, Money.co.uk began with 33 results in the top 100. After the update we saw the site had been reduced to just one!

Now, although some sites suffered a large change in difference, others suffered a much larger percentage change. This is mainly because certain sites dominated so many more results than others to begin with.

In this case Auto Trader is the biggest loser with a 91% drop in results. Again, we see a similar development with the term ‘used cars for sale’ where Auto Trader dominated the SERP (65 results out of 100).

The new algorithm soon started replacing obliterating their results in favour of unique ones, reducing them to a couple of results.

When did this happen?

Taking a few of the biggest losers as examples we’ve plotted their result loss on a daily basis over this period.

Based on this data we’re pretty certain that this update finished rolling out by the 23rd of May 2013, but the changes in results between the 20th and 23rd show a phased shift to the new index, the same date range Penguin went live.

Should Webmasters be worried?

For the majority, the level of change actually looks worse than the reality. As detailed previously, the update typically left the first two pages unaffected, focusing on cleaning up results from lower pages of the index.

Due to results residing past page two generally receiving an extremely low level of click-share, we anticipated that most of these sites won’t notice the update at all, but we wanted to verify that.

In the graph below, we can see each of the site’s visibility scores over the same period for the same keyword set. Due to the varying ranges in volumes of click-share the scores have been normalised for comparison.

Strangely, three of the biggest losers have had a mixed bag in terms of the impact on their click-share. Auto Trader suffered a loss prior to the update which could be put down to normal ranking fluctuation, but had similar levels of click-share after the dust had settled, regardless of being the biggest percentage loser.

On the other hand Money.co.uk has had a different turn of fate, taking a significant click-share hit, while MoneySupermarket turns out to be quite the opposite of a loser, even after dropping an abundance of results.

The likelihood is that neither of these site’s click-share changes were a result of the Domain Clustering update, but rather impacted positively and negatively by Penguin.

Quantity over quality?

Given our findings, it is definitive that the domain clustering algorithm is now in action and working to reduce the number of times a visitor sees results from a single domain.

But, with authoritative brands retaining their dominant positions on the top pages, does seeing a more diverse set of results through the lower layers of the index really benefit search quality?

We decided to analyse the shift in site quality since the domain clustering update. As we’re attempting to measure the type of site which now ranks we’ve looked specifically at Alexa Rank, a publicly available measure of a website’s traffic.

This gives us an indication if Google has replaced the previous highly trafficked sites with other ‘big brands’ or with lower quality sites (which typically have a higher Alexa Rank).

The above graph plots the average Alexa Rank, per ranking position (for May and June), of each site that ranked in the top 100 positions across 621 keywords.

Again, the data is extremely conclusive. In May, sites residing in the majority of the top 100 positions (typically past position 20 (page two)) had a significantly lower Alexa Rank in comparison to those ranking now.

On average, those sites ranking in May had an Alexa Rank of 1,437,153 compared with 2,681,099 in June, almost double. This signals that Google is favouring lower trafficked, and likely lower quality, websites in the lower pages of the index over established domains.

Our view is that the update has removed results simply due to the fact the results are from the same website, without any real thought or testing on Google’s part, sacrificing quality in return for greater diversity!

I for one would prefer to see more results from highly authoritative, trusted brands.

Posted by Cyrus Shepard

Have you ever redirected a page hoping to see a boost in rankings, but nothing happened? Or worse, traffic actually went down? 

When done right, 301 redirects have awesome power to
clean up messy architecture, solve outdated content problems and improve
user experience — all while preserving link equity and your ranking
power.

When done wrong, the results can be disastrous. 

In the past year, because Google cracked down hard on low quality links, the potential damage from 301 mistakes increased dramatically. There’s also evidence that Google has slightly changed how they handle non-relevant redirects, which makes proper implementation more important than ever.

Semantic relevance 101: anatomy of a “perfect” redirect

A
perfect 301 redirect works as a simple “change of address� for your
content. Ideally, this means everything about the page except the URL
stays the same including content, title tag, images, and layout.

When
done properly, we know from testing and statements from Google that a
301 redirect passes somewhere around 85% of its original link equity.

The
new page doesn’t have to be a perfect match for the 301 to pass equity,
but problems arise when webmasters use the 301 to redirect visitors to
non-relevant pages. The further away you get from semantically relevant
content, the less likely your redirect will pass maximum link equity.

For
example, if you have a page about “labrador,� then redirecting to a page
about “dogs� makes sense, but redirecting to a page about
“tacos� does not.

A clue to this devaluation comes from the manner in which search engines deal with content that changes significantly over a period of time. 

The famous Google patent, Information retrieval based on historical data, explains how older links might be ignored if the text of a page changes significantly or the anchor text pointing to a URL changes in a big way (I added the bold):

…the domain may show up in search results for queries that are no longer on topic. This is an undesirable result.

One way to address this problem is to estimate the date that a domain changed its focus. This may be done by determining a date when the text of a document changes significantly or when the text of the anchor text changes significantly. All links and/or anchor text prior to that date may then be ignored or discounted.

If these same properties apply to 301 redirects, it goes a long way in explaining why non-relevant pages don’t get a boost from redirecting off-topic pages. 

301 redirecting everything to the home page

Savvy
SEOs have known for a long time that redirecting a huge number of pages
to a home page isn’t the best policy, even when using a 301. Recent statements by Google representatives suggest that Google may go a step
further and treat bulk redirects to the home page of a website as 404s,
or soft 404s at best. 

This
means that instead of passing link equity through the 301, Google may
simply drop the old URLs from its index without passing any link equity
at all.

While
it’s difficult to prove exactly how search engines handle mass home page
redirects, it’s fair to say that any time you 301 a large number of
pages to a single questionably relevant URL, you shouldn’t expect those
redirects to significantly boost your SEO efforts.

Better alternative: When necessary, redirect relevant pages to closely related URLs. Category pages are better than a general homepage.

If
the page is no longer relevant, receives little traffic, and a better
page does not exist, it’s often perfectly okay to serve a 404 or 410
status code.

Danger: 301 redirects and bad backlinks

Before
Penguin, SEOs widely believed that bad links couldn’t hurt you, and
redirecting entire domains with bad links wasn’t likely to have much of
an effect.

Then Google dropped the hammer on low-quality links.

If the Penguin update and developments of the past year have taught us anything, it’s this:

When you redirect a domain, its bad backlinks go with it.

Webmasters
often roll up several older domains into a single website, not
realizing that bad backlinks may harbor poison that sickens the entire
effort. If you’ve been penalized or suffered from low-quality backlinks,
it’s often easier and more effective to simply stop the redirect than to try and clean up individual links.

Individual URLs with bad links

The
same concept works at the individual URL level. If you redirect
a single URL with bad backlinks attached to it, those bad links will then point to your new URL.

In this case, it’s often better to simply drop the page with a 404 or 410, and let those links drop from the index.

Infinite loops and long chains

If
you perform an SEO audit on a site, you’ll hopefully discover any
potentially harmful redirect loops or crawling errors caused by
overly-complex redirect patterns.

While
it’s generally believed that Google will follow many, many redirects,
each step has the potential to diminish link equity, dilute anchor text
relevance, and lead to crawling and indexing errors.

One or two steps is generally the most you want out of any redirect chain.

New changes for 302s

SEOs typically hate 302s, but recent evidence suggests search engines may now be changing how they handle them — at least a little. 

Google knows that webmasters make mistakes, and recent
tests by Geoff Kenyon showed that 302 redirects have potential to pass link equity. The theory is that 302s (meant to be temporary) are so
often implemented incorrectly, that Google treats them as “soft� 301s.

Duane Forrester of Bing addressed this in a recent tweet.

So, not only do search engines limit us when we try to get too clever, but
they also help to keep us from shooting ourselves in the foot.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!