Picture a meeting room in California, in the late 2000s. On the screen, two blue rectangles. The first leans slightly green, the second slightly purple. Around the table sit some of the best designers on the planet, paid to have taste, and they cannot agree.
“The first one is easier to read.
The second one feels warmer.
Warmer? It’s an ad link, not a wool blanket.”
In most companies, this discussion ends the same way: the most senior person decides. It’s called the HiPPO method, for “Highest Paid Person’s Opinion.” At Google, the discussion ended differently. Someone suggested not choosing at all, and letting users choose instead, without them knowing.
A color quarrel that turned into an experiment
The facts, as Google told them, are fairly simple. The company displayed blue ad links next to its search results. When it launched ads in Gmail, teams noticed that the blue of the sponsored links was slightly different between the two products (Manchester Digital).
They had to harmonize. The open question was which blue to keep.
Instead of deciding, Google ran what its teams call 1% experiments: a small fraction of users sees one version, another fraction sees another, and you compare clicks. Then, to cover every base, around forty more variants were tested between the two starting blues (Manchester Digital). Hence the number that became legend: 41 shades of blue.
The effort is associated with Marissa Mayer, then Google’s vice president of search products and user experience, an engineer by training known for her data-driven decision culture. The winner? A slightly purpler blue, which got more clicks than the greener version (Manchester Digital).
So far, nothing spectacular. A company tests colors and keeps one. The spectacular part comes with the number.
$200 million, according to Google
In 2014, Dan Cobley, then managing director of Google UK, told the story at an event organized by the law firm DLA Piper. He contrasted data culture with HiPPO culture, then dropped the figure: the change of shade had brought in an extra $200 million a year in ad revenue (Manchester Digital).
The number went around the world. You’ll find it in countless conferences and digital marketing courses, usually quoted without a hint of caution.
I’ll be direct, as usual: this number deserves your suspicion.
It comes from a single source, a Google executive defending Google’s method in front of a business audience. No independent audit was ever published, and nothing tells us how the effect of the color was isolated from other changes made in the same period. The figure may be accurate. It may be generously rounded. We don’t know, and nobody outside Google knows more.
This doesn’t disqualify the experiment. Across billions of impressions, a tiny difference in click-through rate can produce enormous sums through simple arithmetic. Microsoft reportedly ran a similar calculation for Bing, estimating that the best blue could generate $80 to $90 million in ad sales (Color Matters).
Notice the paradox, though. The story is told to celebrate fact-based decisions, and most people who repeat it have never checked the central fact. Confirmation bias runs at full speed: the anecdote confirms what we want to believe about the power of data, so we swallow it whole.
The designer who walked out
There is another version of this story, told far less often.
In March 2009, Douglas Bowman, visual design lead at Google, left the company after nearly three years. He published a farewell post that made waves in the design world. In it, he confirmed the story was true: a team couldn’t choose between two blues, so it tested 41 shades in between (Zeldman).
Bowman accepted that the method works. What bothered him was what it does to a culture. In his view, design at Google lived or died by the sword of data (ITPro). He also described having to debate whether a border should be 3, 4 or 5 pixels wide (Fortune). For him, such dependence on data blocked any bold design decision, and his post sparked a heated debate across the web (TechRadar).
For some, Google proved that taste is just an opinion and that opinions can be replaced by measurement. For others, Google showed that an organization obsessed with measurement ends up daring nothing it cannot measure.
Both camps are right. That is exactly what makes the story interesting for a leader.
Optimizing a click or inventing a product
Look closely at what the 41 shades test produced. It improved an existing element of an existing product. The ad link was already there, and so were the users. The question put to the data was narrow: among these 41 options, which one maximizes clicks?
In my book, I describe the three horizons model. Horizon 1 covers established activities: the goal is to increase profits, with optimization methods and quality that must be perfect. Horizon 3 covers activities still being defined: the goal is to prove viability, with discovery methods and deliberately imperfect quality (My book, chapter 12).
The 41 shades test is a textbook Horizon 1 case. It is remarkable in its category. It says nothing about Horizon 3.
No A/B test would have invented Gmail. No A/B test would have invented the iPhone. An A/B test compares options someone has already imagined, and finds the best among the ones it is given. Optimization specialists call this the local maximum risk: you climb the hill you’re standing on very efficiently, and never see the mountain next door.
Here is the trap. An organization that discovers the power of 1% tests can be tempted to run everything through that method. It becomes formidable at improving what exists, and starts rejecting anything that can’t be proven in advance. Yet a breakthrough idea, by definition, has no data yet. It only has assumptions, which must be killed or confirmed through a different kind of experiment, faster and closer to the field.
Bowman sensed it. Mayer was right on her own turf. A leader’s job is to know which horizon they’re playing in at the moment of deciding.
Your own shades of blue
Back to your company. You have your own blue quarrels. The wording of the sign-up button. The price shown on the offer page. The subject line of the sales email. The day the newsletter goes out.
How many of these decisions are made by your executive committee’s HiPPO? How many could be settled by a simple experiment, in two weeks, on a fraction of your customers?
And the other way around: how many ambitious projects are killed early because someone demands numbers that cannot exist yet?
Cognitive biases govern us constantly. To limit their effect, favor facts and experiment results over the emotion of the moment, and delay the decision when you can (My book, chapter 6). Keep in mind, though, that the opposite excess exists. Analysis paralysis threatens teams that are always asked for more information before acting (My book).
Here is an exercise. On a scale of 0 to 5, rate your organization on two criteria:
- Its ability to settle optimization decisions through experimentation rather than the opinion of the most senior person.
- Its ability to protect breakthrough projects from premature demands for hard numbers.
If your two scores are far apart, you’ve found your work site. For my part, I long believed that protecting yourself from bad decisions with data was enough. I was wrong about half of the problem.
The essentials
Remember four things:
- The 41 shades test is an excellent example of optimization: a debate about taste replaced by a measured experiment on a small fraction of users.
- The $200 million figure rests on a single statement by a Google executive. When you quote it, say it is Google’s estimate.
- An A/B test chooses among existing options and invents none. Breakthrough innovation needs other methods.
- The danger lies on both sides: deciding by HiPPO where an experiment would do, and demanding data where it cannot exist yet.
Innovate or agonize, the choice is up to you. And this time, you can even test both.
A corporate event, a seminar, an executive committee meeting, or a management committee meeting?
Philippe’s keynotes on innovation and AI are groundbreaking, you’ve been warned!
Complement a powerful keynote with innovative and impactful workshops.
A keynote inspires and raises awareness; workshops transform!
References
- (My book) https://philippeboulanger.com/book/
- (Manchester Digital) https://www.manchesterdigital.com/post/manchester-digital/why-google-has-200m-reasons-to-put-engineers-over-designers
- (Color Matters) https://colormatters.com/color-matters-blog/a-color-thats-worth-80000000
- (Zeldman) https://zeldman.com/2009/03/20/41-shades-of-blue
- (ITPro) https://www.itpro.com/610294/designer-quits-google-over-data-focus
- (Fortune) https://fortune.com/2009/03/21/are-googles-nerds-destroying-design/
- (TechRadar) https://www.techradar.com/news/internet/the-secrets-of-google-s-design-team-641441







