Much has been written about the possible SEO impacts of A/B testing, for the most part, the following criticisms or A/B testing SEO risks have been largely debunked:
- Duplicate content
- Lost link equity
- Page load speed
But there is one factor I think many overlook: how a successful A/B test can impact your SERP Bounce Back Rate. SERP Bounce Back Rate is the number of users who visit your site from Google and hit their back button. This is a critical signal-it basically says to Google, “the user didn’t find what they wanted, now they’re back for something else.” There is strong evidence Google uses SERP Bounce Back as a ranking signal. There’s even a commercial blackhat tool out of Russia that attempts to manipulate this signal. But note, SERP Bounce Back is not the same Bounce Rate most people talk about in A/B Testing. Traditional Bounce Rate counts anyone who leaves your site after one visit as a “bounce,” even if you gave the user exactly what they wanted.
How Can A/B Test Hurt Your SERP Bounce Back Rate?
The issue isn’t so much that A/B Testing in and of itself can hurt you, it’s that by changing your page at all, you may negatively affect your SERP Bounce Back Rate. A/B testing can give you false confidence you’re making a safe change. If you focus solely on your optimization variable and Bounce Rate, you’ll miss it and roll out a change that could tank your traffic.
Why does this happen?
Many people imagine that when users visit their site, there is a limited set of things they can do, like 1) submit a lead, 2) read another page on the site, 3) bounce, or 4) close the browser. Taking this a step further, they imagine that options 1, 2, and 4 are good and bounces are bad… unless the user bounces because they found something of use. What they forget is that Google can only see two things: user searched and user came back (or not). In other words, Google only sees a portion of the bounces, and it’s that portion that you need to manage. But most sites don’t measure or manage SERP Bounce Back; when they A/B test, they manage only Bounce Rate.
So, you could improve the user experience in every other way-increase time on site, increase pages per visit, increase your average revenue per user, increase satisfaction metrics on your content (if you’re surveying on that), but if the number of visitors who come back to Google increases, you jeopardize your rankings.
Real World Example
Let’s say you run an apartment website and you decide to test a set of “city pages” that rank well for keywords like “[city] + apartments”. Today, when people come to your page, there are three things they can do: 1) submit a lead for help finding an apartment, 2) click out to an apartments’ website (and leave your site), 3) call the apartment then close the browser, or 4) bounce back to Google. From your point of view, it would be much better for your business, and (for the sake of argument) for users, to increase the number who submit leads and get free hands-on help from an apartment locator.
So you run an A/B test with a new page layout whose goal is to increase the number of leads by hovering a giant “get free help finding an apartment” button over the page. Of course you don’t want to spike your bounce rate, so you provide a “no thanks, continue” link below it. The naïve tester says, “if we generate more leads without increasing our bounce rate, we’ll go ahead.”
The good news arrives after a few weeks of running the test: leads have doubled and your bounce rate has gone down slightly.
|Number of visitors||100||100|
Awesome, right? Roll out it immediately! Not so fast. What if this is how the test looked to Google:
|Number of visitors||100||100|
Click out to apartment website
Bounce back to Google
- What you see: 2x more leads with a lower bounce rate.
- What Google sees: 2x higher SERP Bounce Back Rate.
Why? Bounce Rate hides the fact that your giant hover-over button caused 2x more users to have the blink reaction to leave the page… of those left, fewer clicked out to view apartments, so the overall bounce rate went down.
What’s worse is that before you did the test, 90% of your pages were only 10% better than the next-best ranked site sitting below you in the SERP’s, so when you rolled this change across thousands of city pages, it was a enough give them an edge. You drop from an average #1 position (with 42% of the organic clicks) to #2 (with 11% of the organic clicks). If your traffic is highly correlated with your revenue, congratulations, you’ve just destroyed 73% of your value as a company.
What Should You Do?
There are three ways to manage the risk of negatively affecting your SERP Bounce Back Rate.
- Estimate Your SERP Bounce Back Rate. Measure every possible click on your site and its source, and use the data to back into your SERP Bounce Back Rate. If you capture every possible click referred by Google, then you know the only alternative place for users to go is back to the SERP. But realize your estimate may be imperfect and it would be insufficient to simply sign off on a UI change based solely on that number. You need to test it.
- Test Changes in a Rankings Sandbox. Once you’ve tested a change and feel that it’s safe and effective, release it on a sandboxed sample of pages of your website. It’s important to hardcode the change to a set of URL’s and not to rollout the change with your A/B testing code (i.e. by sending a portion of traffic to your test page) so you can monitor rankings on the top 3 or 4 keywords for each specific page in the sandbox. You need 100% of users hitting these pages so whatever behavior effect with by pronounced and obvious. Watch ranking changes in the test group to rankings changes in a control group for 4 to 6 weeks. After that, you should have a fairly good idea whether your rankings were negatively impacted by the test.
With all things in the arena of SEO, there is no cut and dry, pat answer. Every question is always in light of competition, always balanced against other signals. But if you’re A/B testing a change that could impact thousand of pages, there’s a good chance a negative change to your SERP Bounce Back rate will translate into lost rankings and weaker traffic. And if that happens, you’ll get to explain to everyone why you need to roll back UI changes that appeared to have improved conversion rates, based on a metric you can’t see and don’t have any data about. So start managing your A/B tests more intelligently by measuring and optimizing your SERP Bounce Back rate.