41 Shades of Blue: Google’s $200 Million A/B Test

Picture a meeting room in California, in the late 2000s. On the screen, two blue rectangles. The first leans slightly green, the second slightly purple. Around the table sit some of the best designers on the planet, paid to have taste, and they cannot agree.

“The first one is easier to read.
The second one feels warmer.
Warmer? It’s an ad link, not a wool blanket.”

In most companies, this discussion ends the same way: the most senior person decides. It’s called the HiPPO method, for “Highest Paid Person’s Opinion.” At Google, the discussion ended differently. Someone suggested not choosing at all, and letting users choose instead, without them knowing.

A color quarrel that turned into an experiment

The facts, as Google told them, are fairly simple. The company displayed blue ad links next to its search results. When it launched ads in Gmail, teams noticed that the blue of the sponsored links was slightly different between the two products (Manchester Digital).

They had to harmonize. The open question was which blue to keep.

Instead of deciding, Google ran what its teams call 1% experiments: a small fraction of users sees one version, another fraction sees another, and you compare clicks. Then, to cover every base, around forty more variants were tested between the two starting blues (Manchester Digital). Hence the number that became legend: 41 shades of blue.

The effort is associated with Marissa Mayer, then Google’s vice president of search products and user experience, an engineer by training known for her data-driven decision culture. The winner? A slightly purpler blue, which got more clicks than the greener version (Manchester Digital).

So far, nothing spectacular. A company tests colors and keeps one. The spectacular part comes with the number.

$200 million, according to Google

In 2014, Dan Cobley, then managing director of Google UK, told the story at an event organized by the law firm DLA Piper. He contrasted data culture with HiPPO culture, then dropped the figure: the change of shade had brought in an extra $200 million a year in ad revenue (Manchester Digital).

The number went around the world. You’ll find it in countless conferences and digital marketing courses, usually quoted without a hint of caution.

I’ll be direct, as usual: this number deserves your suspicion.

It comes from a single source, a Google executive defending Google’s method in front of a business audience. No independent audit was ever published, and nothing tells us how the effect of the color was isolated from other changes made in the same period. The figure may be accurate. It may be generously rounded. We don’t know, and nobody outside Google knows more.

This doesn’t disqualify the experiment. Across billions of impressions, a tiny difference in click-through rate can produce enormous sums through simple arithmetic. Microsoft reportedly ran a similar calculation for Bing, estimating that the best blue could generate $80 to $90 million in ad sales (Color Matters).

Notice the paradox, though. The story is told to celebrate fact-based decisions, and most people who repeat it have never checked the central fact. Confirmation bias runs at full speed: the anecdote confirms what we want to believe about the power of data, so we swallow it whole.

The designer who walked out

There is another version of this story, told far less often.

In March 2009, Douglas Bowman, visual design lead at Google, left the company after nearly three years. He published a farewell post that made waves in the design world. In it, he confirmed the story was true: a team couldn’t choose between two blues, so it tested 41 shades in between (Zeldman).

Bowman accepted that the method works. What bothered him was what it does to a culture. In his view, design at Google lived or died by the sword of data (ITPro). He also described having to debate whether a border should be 3, 4 or 5 pixels wide (Fortune). For him, such dependence on data blocked any bold design decision, and his post sparked a heated debate across the web (TechRadar).

For some, Google proved that taste is just an opinion and that opinions can be replaced by measurement. For others, Google showed that an organization obsessed with measurement ends up daring nothing it cannot measure.

Both camps are right. That is exactly what makes the story interesting for a leader.

Optimizing a click or inventing a product

Look closely at what the 41 shades test produced. It improved an existing element of an existing product. The ad link was already there, and so were the users. The question put to the data was narrow: among these 41 options, which one maximizes clicks?

In my book, I describe the three horizons model. Horizon 1 covers established activities: the goal is to increase profits, with optimization methods and quality that must be perfect. Horizon 3 covers activities still being defined: the goal is to prove viability, with discovery methods and deliberately imperfect quality (My book, chapter 12).

This book is for CEOs, managers, and innovators ready to challenge everything and create exponential growth.

Be the first to get the english version!

The 41 shades test is a textbook Horizon 1 case. It is remarkable in its category. It says nothing about Horizon 3.

No A/B test would have invented Gmail. No A/B test would have invented the iPhone. An A/B test compares options someone has already imagined, and finds the best among the ones it is given. Optimization specialists call this the local maximum risk: you climb the hill you’re standing on very efficiently, and never see the mountain next door.

Here is the trap. An organization that discovers the power of 1% tests can be tempted to run everything through that method. It becomes formidable at improving what exists, and starts rejecting anything that can’t be proven in advance. Yet a breakthrough idea, by definition, has no data yet. It only has assumptions, which must be killed or confirmed through a different kind of experiment, faster and closer to the field.

Bowman sensed it. Mayer was right on her own turf. A leader’s job is to know which horizon they’re playing in at the moment of deciding.

Your own shades of blue

Back to your company. You have your own blue quarrels. The wording of the sign-up button. The price shown on the offer page. The subject line of the sales email. The day the newsletter goes out.

How many of these decisions are made by your executive committee’s HiPPO? How many could be settled by a simple experiment, in two weeks, on a fraction of your customers?

And the other way around: how many ambitious projects are killed early because someone demands numbers that cannot exist yet?

Cognitive biases govern us constantly. To limit their effect, favor facts and experiment results over the emotion of the moment, and delay the decision when you can (My book, chapter 6). Keep in mind, though, that the opposite excess exists. Analysis paralysis threatens teams that are always asked for more information before acting (My book).

Here is an exercise. On a scale of 0 to 5, rate your organization on two criteria:

  1. Its ability to settle optimization decisions through experimentation rather than the opinion of the most senior person.
  2. Its ability to protect breakthrough projects from premature demands for hard numbers.

If your two scores are far apart, you’ve found your work site. For my part, I long believed that protecting yourself from bad decisions with data was enough. I was wrong about half of the problem.

The essentials

Remember four things:

  • The 41 shades test is an excellent example of optimization: a debate about taste replaced by a measured experiment on a small fraction of users.
  • The $200 million figure rests on a single statement by a Google executive. When you quote it, say it is Google’s estimate.
  • An A/B test chooses among existing options and invents none. Breakthrough innovation needs other methods.
  • The danger lies on both sides: deciding by HiPPO where an experiment would do, and demanding data where it cannot exist yet.

Innovate or agonize, the choice is up to you. And this time, you can even test both.

References

Picture of Philippe Boulanger

Philippe Boulanger

Philippe Boulanger, international speaker on innovation and artificial intelligence, author, advisor, mentor and consultant.

Latest POSTS

Drone vs Camera Robot: Who Films Where the Sky Is Banned?

Banned from flying over crowds, drones run into regulation and noise. The Beni camera robot films from the ground. A textbook case of the framing bias.

Read More »

Friday the 13th: why superstition pays off for innovators

Friday the 13th destroys nothing. People who fear it change their behavior, and that change can be measured. Accidents, stock markets, the Jerusalem virus, a $900 million figure nobody verified: what superstition reveals about your innovation decisions, and how to use it.

Read More »

Naming an innovation: what Sapir showed in 1929

In 1929, Edward Sapir showed that most people link the invented word “mal” to a large table and “mil” to a small one. Almost a century later, sound symbolism still shapes how we perceive product names. What the numbers really say, what marketing took from it, and how to test a name before you pick it.

Read More »

AI-Generated Code: Who Still Understands Your Software?

A year after handing 100% of its code to AI, a company finds that nobody can explain it anymore. An anonymous testimony, the METR, GitClear and Veracode studies, and the method to stay out of the labyrinth.

Read More »

Why Coca-Cola’s Inventor Never Got Rich

John Pemberton invented Coca-Cola in 1886 and died poor two years later. Thirty-three years after the first glass, the company was worth $25 million. What this story reveals about the difference between invention and innovation, and about the Pemberton who may be sitting in your own company.

Read More »

Rare-Earth-Free Motors: France Had a 14-Year Head Start

Renault has built rare-earth-free motors since 2012. Then the stator of its next E7A motor went to China. How a technological lead gets lost in purchasing trade-offs.

Read More »

Leeuwenhoek: the cloth merchant who saw microbes first

In 1676, a Delft draper with no university education discovered microbes. The Royal Society didn’t take his word for it: it checked, then elected him. What the accurate story of Leeuwenhoek teaches leaders about ideas that come from outside.

Read More »

Edison’s salt test is a legend. His 146 questions are not.

The salted soup test attributed to Edison is a legend. His 1921 questionnaire, however, failed 686 of 718 applicants, and Einstein himself stumbled on one of its questions. From Xero’s coffee cup to Musk’s riddle, what these hiring tests really measure, and what hiring science recommends instead.

Read More »

Are you a rule breaker?

You weren’t supposed to find this.

But here you are, because you did what most people don’t: you questioned, you explored, you clicked the thing you weren’t sure you should click.

That’s Innovational Intelligence™ in action.

Most people stay inside the lines. Follow the expected path. Click the obvious buttons. Accept things as they are.

Not you.

You’re one of those rare minds that refuses to accept “this is how it’s always been done.”

We need more people who think like you.

So here’s your reward for coloring outside the lines:

Get VIP pre-release access to the next assessment on Innovational Intelligence™:

You’ll be the first to know when it’s available.

Keep breaking rules. The world needs what you see.

Your daily dose (or almost) of Innovational Intelligence™!

Receive in your inbox my latest post before everyone else!