Big Swings

I often spend time on customer calls advocating for brands to test “bigger swings” with pretty minimal pushback. Yet I so often find folks (myself included) in an incremental mindset.

Earlier this week, I was on a customer call where we were discussing a particular test idea and someone from their side (to whom we previously advocated “bigger swings”) challenged me on whether the test in question was a big enough swing. Boom - fantastic question - is it a big enough swing?¹ The conversation found its way to “naming” a big swing test—not an easy thing to do.

Drew wrote last week about how tests often have a way of surprising you, which implies that it’s not easy to predict what will end up being a big swing. Big outcomes are identified after the fact, based on the results that a particular test shows. It’s hard, then, to make a determination on a big swing in advance.

But that’s sort of the opposite of what I’ve been trying to advocate. And this challenge from a customer to name a big swing on the spot got me thinking about how I would define the phrase “Big Swing”. 

So here’s my current thinking: a Big Swing is a change that can trigger a multiplying or compounding effect on the business.

Often I take the ‘lazy’ approach to answering this question and say that a Big Swing is anything changing the dollars and cents. Sure, it’s true that changing prices and discounts are big swings – but why? 

I firmly believe that big swings are ones that can change the underlying economic model of your business. They can come in the form of impacting willingness to buy (Do I need this product?), willingness to pay (Do I want to pay this much for this product?). They can also come in the form of discovery and what/how much people buy (I wasn’t planning on buying two, but I might as well). 

Some of the testing we see is adjacent to these vectors but not always directly tied to them. Often they are more about reducing friction or improving the site experience so that, once someone has expressed a willingness to buy and expressed a willingness to pay, they take action. Those might create some incremental wins that are beneficial to the business, but they aren’t the type of wins that are going to change your trajectory. 

Take lowering prices as a counter-example. On one hand you may have lower unit margins. On the other hand, you may sell way more of your product, offsetting lower margins and actually generating more profit. But that’s not all! You also generate more customers, each of which has a future lifetime value. And then there’s the network effect of those extra customers telling their friends about your product. And now your acquisition cost is lower, because lowering prices boosted conversion, so you can more efficiently scale your marketing spend acquiring yet more customers. And with the higher volume, higher LTV, friend networks, and increased media spending all driving more sales you’re able to increase your minimum order quantities from your supplier, reducing costs, and now your margins are back to what they were! 

I just hypothetically doubled your company overnight. (s/o Dave Rekuc; he’s a long-time supporter of Intelligems who opened my eyes to these concepts of reinvesting profits to find new, more efficient marketing-spend curves.) You’re welcome. 

Spoiler alert – it’s actually not that easy. Our intuition about pricing is very often wrong (we actually wrote a Harvard Business Review article on this last year). For most of us, especially professionals, we can look at a page and tell whether or not it’s designed well. Very few people, even professionals, can say whether or not you’re priced efficiently. Thus the potential gains available from testing here are far greater than, say, site navigations/pay layout tests. 

It’s not to say those UI-type tests aren’t worth running; they are and they’re easy to implement and will keep your test velocity high to unlock incremental gains. All good things. But the likelihood that they can unlock 20% gains overnight? Eh...

A price test, though, could actually get you there. 

And that’s where the big swing really comes in. If your economic profile changes by 20% and you’re willing to reinvest that into growth, your compounding effects become very real. Maybe you decide to go deeper into your category and win more share. You reinvest in more marketing to grab more eyeballs. Or maybe you decide to expand into new categories. You now have incremental profit to reinvest in your business. 

So if any of those bets pay off, your 20% profit gain isn’t 20% more profit; it's a new growth curve for the business funded by one test.

That feels like a pretty Big Swing to me.

1  The test in question was whether or not the product badges on collection pages were effective. While I do think that these badges are certainly impactful, my intuition is that the language and quantity of them isn’t as important as simply having them.