Expected Value

There is a cliché in experimentation that there is no losing, only learning. And, while it may be true, it ignores how you can actually maximize the expected value from what you learn in such an experiment. 

I think those unexplored components are far more valuable than the lesson from the cliché itself. 

When you take the phrase at face value, the obvious, on-surface “learning” of a “losing” test is that a specific hypothesis was invalidated; it is an insight that applies to a tactic. In such a view, the learning is actually fairly limited in its value. 

As an example: If you learn that your customer doesn’t engage with a Reels-style navigation item or your conversation rate doesn’t increase from a product guarantee badge at checkout, you’ve learned that these two tactics don’t increase micro-conversions at different points in the funnel.

While they are learnings, they have very little additional value outside of the tests themselves. They offer me few clues as to what I should test next to increase those micro-conversion points.

In other words, the expected value of the test is low. 

Expected value¹, if you’re unfamiliar, is found in statistics, and is the weighted average of all possible outcomes for a scenario. 

An example:

If there is a 50/50 raffle with $2,000 pot, a winner will receive $1,000. Say tickets cost $10 (meaning 200 tickets get sold). Each ticket has a .5% chance of winning. To calculate the expected value of the ticket, you multiply the probability of having a winning ticket (.5%) by the total winning pot value ($1,000) and then subtract the cost of your ticket ($10).

In this case, a $10 ticket has an expected value of -$5: You end up spending more to buy the ticket than you should expect to win in the raffle.

50/50 raffles are designed to operate in this fashion (it’s how they raise money). But what if you wanted to increase the expected value of participating in the raffle?

You can’t just buy more tickets. You’d have to buy more tickets at a discounted price. In such a scenario, the price needed to get to break even (or far closer to it) on expected value would be a 50% discount on the tickets: You’d need to be able to buy 3 tickets for $15.

This, I think, is a good mental model for how to actually learn from tests.

Since most tests are likely to fail (only something like 30% of experiments end up winning), you have to find ways to get future tests at a “discount.” 

The way to do that is to pick experiments where learnings can be applied in the future (either in the form of follow-on tests or as a collection of insights that might create a new hypothesis).

You might, say, test prices and learn that your product is fairly price inelastic. If that’s the case, a natural follow-on test would be to test higher prices to see if that holds. And, on the other end of the spectrum, if you find your product is price elastic, you might test just how elastic it is. Would meaningfully more demand make up for less gross margin?

The expected value in such a test is way higher than those discussed earlier, because you can begin compounding your tests to find strategic insights as opposed to tactical outcomes. 

You do, to be clear, need both. But an experimentation roadmap that has compounding benefits—even in their learnings—is likely to be worth far more than one that doesn’t.

1  I ended up on an expected value as a topic due to a fantasy football trade that was proposed to me last week. Since I’ve spent a good amount of time on sports metaphors the last month or so, I opted to avoid bringing another such metaphor to the forefront of this newsletter. I couldn’t help myself from mentioning it, though.