A/B testing is useful but limited. It’s great when trying to decide between 2 similar solution options. Like the the colour and placement of a button. Should it be green or should it be red? Should it be at the top or the bottom of the page?
But, what about when we’re not sure what our users want? Too often, product teams jump into A/B testing without asking the more important questions - Should we even be solving this problem? Is this really how we can provide the most possible value for our customers?
A useful mental exercise to combat the tendency to jump straight into solutions and A/B testing is to think about an A/Z test.
What would 2 completely opposite options look like?
Instead of worrying about whether the button is green or red, consider what would happen if you simply removed the button? Then consider the opposite end - what would happen if we made the button massive and the only clickable element on the screen? Think through each of the scenarios in depth.
9 times out of 10 you’ll end up at the same general conclusion you had initially. But once in a while, you’ll realise you were thinking about it all wrong. You’ll notice a massive opportunity to solve the right problem for your customers in a unique way.
You’ll realise you were comparing different varieties of apples when your customers really wanted oranges!