Context

Awhile back, we released an AI feature that would analyze your website and recommend a series of tests. It was cool, but—like most AI—it demoed better than it delivered value.

Most recommendations would be along the lines of test using lifestyle images or adding more social proof. General best practices, yes, but not really specific to the brand or what they were trying to achieve with Intelligems in the first place. 

The problem with it was the problem associated with most AI tools and most advice around testing programs: It lacked context. 

I guess I shouldn’t have been surprised by this. When I meet with merchants a common question I get is, “What do you recommend I test first?”. My answer is always, “It depends”. It depends on what your goals are (profit, volume, efficiency, …), what are the most sold products, what are common user journeys, where do people get stuck?

In the first newsletter we published, Drew laid out the idea that a brand’s experimentation roadmap is contextual. What should be tested is what needs to be improved. And the best place to find that is a brand’s P&L:

Our initial AI feature didn’t take any of this into consideration. In fact, it was sort of like advice you see on LinkedIn or the experience of using ChatGPT or Claude: You ask a question and you get a (general) answer back.

The first time you ever ask an LLM a question and get a generalized answer back, that’s novel. It’s a cool experience, but, it turns out, often as useful as the annoying best practices thought leadership you get from LinkedIn bros. In other words, it turns out that this use case is more neat than useful. 

After the initial awe of being able to write a Shakespearean sonnet about a topic of my choosing in a matter of seconds wore off, it was a while until I had my second “ah-ha” moment with AI.  Deep Research: you ask a question and you get a series of questions back. In this scenario, the LLM is looking for context, so it can answer your question more specifically. Then it gathers a ton of additional context, applies reasoning, and provides a very tailored (and helpful!) response. That’s both novel and useful.

This, I think, is where most of us are running into roadblocks with AI: How do you decouple the novelty from the usefulness, since the novelty will wear off?

For Intelligems, the answer is to invest in context. 

If we’re to invest in making recommendations, we’ll first need to understand what a brand wants to achieve generally (Drew’s first post about your P&L being your roadmap) and what a brand wants to achieve specifically (how, say, do you want to improve your AOV?).

This, in itself, is sort of meta: The context needed to recommend a specific price test is probably very different from the context needed to recommend a landing page redesign. That makes the investment challenging, both in terms of technically collecting the data and in terms of getting a brand to provide the context.

All of it, though, presents an opportunity: If a brand is able to answer these questions about what they want to achieve, they’ll operate with more clarity and we’ll be in a position to make better recommendations. 

That feels incredibly useful in the long run, even if it is novel now.