Sequential

One of the disconnects I’ve felt with this newsletter for a while is how much we end up talking about personalization when so much of the market is so heavily focused on testing. They end up seeming juxtaposed and compared, when in reality I think they need to be used together, and are really two sides of the same coin.

I hear from teams that they are “really good at testing,” get value from those programs, but that “we don’t really know where to start with personalization.” This is a huge limiting belief! Personalization, done well, is a chance to get outsized value by treating different audiences differently. You are already doing the technical hard part (treating different audiences differently) when you A/B test — the only difference is that, with personalization, you are doing this with purpose rather than randomly (like with a test).

Part of the problem, I think, is how personalization is often framed.

The framework most in the industry use when talking about personalization is a bottoms-up framework, where a test leads to a “losing” result but, after some analysis, you discover that the test had a positive impact on one particular segment (or visa versa). You should exploit that, and just use the experience for the segments of your traffic where it performs best.

An example: We have a customer who once tested a video banner on their website. It performed horribly. When they dug into the data, they found that the video actually worked very well on desktop — a 10% lift — but desktop traffic only represented about 15% of traffic, so the win was washed out in the aggregate data.

They spotted that, though, and rolled it out. In theory, this should boost profit by 1.5% (a 10% gain on 15% of traffic).

This is personalization. Or, at least, this is the prevailing explanation of personalization.¹

I think this framework ends up getting used for a couple reasons:

1. Testing is more common than personalization, so it’s easier to start the explanation where people are most familiar. It’s a common ground.

2. It reinforces the narrative that “experimentation is learning” through a positive lens. The example above is just that: A test might have lost, but it won somewhere, so you win.

This is all valid. But when this is your only definition of “personalization,” it can be a bit intimidating, and seem very complex to build a good personalization program. You have to start with a broad test (with its own hypothesis), you have to analyze the data, you have to find the micro segment that performs, you have to roll out the experience to just that segment, and then you have to remember all those variations that are live.

Why not just start at personalization? (I.e., come with a top down perspective.)

We all know that our traffic isn’t a monolith: We know we have new customers and returning customers; we know we have customers who buy from one section of our catalog and customers who buy from multiple sections of our catalog; we know we have new customers who convert from a specific use-case ad and new customers who convert from a different use-case ad. In a top down world, you build a hypothesis about what one of those segments would want to see, and go make that change.

E.g., you know that visitors coming from a “welcome flow” in Klaviyo are looking to purchase using their new customer offer — why not adjust the on-site experience to highlight that offer and surface products that are most popular with new customers?

You can still test this experience. You “hold back” a chunk of the traffic and give them the normal experience, and then measure the uplift to validate that treating different customers differently delivers results.

You don’t need to test, then wait, then personalize. Instead you start with the personalization and can test after to validate.

The two don’t need to be sequential. They can be run in parallel, and can get you to value a lot faster.

1  In fact, we’ve highlighted this discovery-driven approach to personalization in a few different newsletters. I have certainly contributed to my own frustration!