Timo here. I write about the slower Product Discovery topics. The essentials, not the latest thing; the internet already has plenty of those.

Figuring things out as I go, 100% written by a human (me), with a new post every now and then.

1. My Struggle:

Me: *sits with product team*

Team: “check out this feature we need to get done”

Feature: *not tied to an opportunity, unclear how it will make us money, no supporting evidence, needs to get done ASAP. Overall just being in a risky mood*

Me: “maybe we should figure out how this will make us money, or, at the very least, wrap it in an A/B test so we know it doesn’t hurt us”

Team: “No. Discovery or experimentation takes too long. If we ship faster, we learn faster”

Me: *nervous laughter*

So I could start rambling how this is a folly. That they confuse time to task is done with time to results, and that this, in fact, is not the right kind of faster.

But I’m tired boss.

Maybe, when a team sees no problem with shipping like this, I should take that as a signal that they’re not going to be receptive to arguments about value.

2. (What if) they’re right and I’m wrong.

What if I’d stop arguing the value, and accept the premise that doing discovery or experimentation is indeed slower than just shipping (it is, obviously, on time to task is done).

This got me thinking: from the perspective of the product team, me (or a separate team or department for that matter) helping them with discovery or experimentation is basically a service.

Some services are frictionless: ordering take out from a food delivery service. One app, a few swipes, delivered to your door within the hour.

Some services are shit: trying to get IT to fix your laptop. No ticket = no service, having to deal with ticketing systems, waiting times, more systems, losing your laptop for a few hours (or days).

Now if there’s no incentive to use the service from a value perspective (which is the situation we’re in), it had better be super easy to do.

And it’s not!

It varies per org and depends on the maturity of the teams and how discovery / experimentation is organised, but it loosely looks something like this:

1. The need for insights is captured / noticed / prompted

2. The ask or situation goes to the person or team

3. They do the necessary checks. If GO:

4. Create a plan / design the study / write the hypotheses

5. Recruit the people / build the A/B test

6. Conduct the test / gather the data

7. Make sense of the stuff they found

8. Report back with the decision to make

I’ve worked with teams where this whole cycle takes a couple of weeks. I’ve also worked with teams where this whole cycle can take up to two months.

If there’s no intrinsic motivation or incentive to experiment (or other means of gathering insights) and teams are incentivised to just ship as much as possible by the end of the quarter, asking a team to wait weeks or months to do an A/B test first is a BIG ask.

It’s a hassle, it takes too long AND there’s a BIG chance the outcome of the test is going to create even more hassle because only 1 in 10 online experiments are successful.

Statistically I’m asking them to wait to be proven wrong. Which means it’s back to the drawing board, which nobody wants because everybody just wants to get the work done ASAP.

3. There’s actually theory for this stuff, btw

Turns out, my struggle to get teams to do more discovery and experimentation can actually be explained through theory:

Source: https://www.behaviormodel.org

This framework was sitting in my notes for years. The simplest explanation, straight from the source:

“Behavior happens when Motivation, Ability, and a Prompt come together at the same time. When a behavior does not occur, at least one of those three elements is missing.”

Simple enough.

I have no idea how scientifically sound this framework is, but some common scenarios seem to fit well:

• Doing my taxes each year: I get prompted by mail each year. Not easy to do (low ability), but motivation is maxed out because NOT doing them would make me a criminal.

• Going to the gym: I’ve accepted that I’ll never enjoy going (motivation low), but the gym is on my morning commute to work. I get prompted every time I cycle past it. It’s right there plus they make my workout plan. I just have to walk in and do what the app tells me to do, which is easy (high ability).

• Cleaning out the shed: I despise it (low motivation), it’ll take up my weekend plus it means I probably have to take a few trips to the recycling centre on a busy Saturday morning (low ability). I’ve also learned to ignore it (no prompts), so yeah, it’s not happening.

Looking at my situation with some product teams, this is what seems to be happening when I want them to do more discovery and experimentation:

Their motivation is WAY below the Action line. If I want them to get into action, I need to seriously pump motivation (to be clear: this isn’t about wanting. Most teams I work with want to. It’s about what the system rewards).

Talking about the benefits of doing product discovery and experimentation (or the risks if you don’t) is essentially that: trying to increase motivation.

It’s also very naïve of me to think that me giving that talk will actually do that. Motivation comes from incentives, leadership, and how the org defines success. I, as an individual, can’t influence that. That requires bigger, more complex, organisational changes that are worth entire books on their own.

4. Focus on what’s possible: making things easier

However, what I, as an individual, CAN do is try to make things easier to do.

If it’s harder for a product team to do an A/B test or User interview than it is to get a laptop fixed by IT, then it probably IS the ability lever that needs work, not motivation.

If I accept motivation is a fixed constant, then that naturally makes ability the only lever I can pull to get teams above the Action line.

That’s my new focus: make discovery and experimentation as easy for product teams as ordering takeout (or at least easier than getting their laptops fixed).

It’s fun how frameworks can sit unused in my notes for years but suddenly click once I’ve stubbed my toe a couple dozen times.

The HOW to make discovery and experimentation easier is a whole new world on its own and probably a post for the future.

Some thoughts and rough notes for future me if I ever decide to write that post:

• The prompt part of the model is really interesting. Depending on whether the experimentation or discovery people are organised centralised or decentralised, the prompt needs to be captured before the whole thing can be set in motion. Someone needs to be there if the product teams aren’t proficient enough yet in translating a latent need for insight into a prompt for discovery.

• An interesting metric to track here is time to insight. Once a product team decides it needs insights (in whatever form), how long until useful insights are collected that help make a better informed decision? Now that might be weeks or months, but what if it’s 2 days or even 2 hours? What if it’s faster than ordering take out?

• AI is an obvious candidate for some steps in the entire process, though the biggest chunks of time and effort will always be the actual sitting down and conducting the user interview or the waiting for sample size. That explains the hype for synthetic users I guess.

Related and worth checking out:

• A keynote from Denise Visser at Conversion Hotel about making experimentation easy at bol.com

• This post from Melanie Kyrklund. She gave an excellent talk at Emerce Conversion ‘26 about this exact topic.

Timo here. I write about the slower Product Discovery topics. The essentials, not the latest thing; the internet already has plenty of those.
Figuring things out as I go, 100% written by a human (me), with a new post every now and then.

Maybe I’m lucky, but I have yet to meet a Product team that doesn’t want to do more discovery and escape the Feature Factory.

All of them wish they could talk more to users, had more autonomy over their product, and have a bigger impact on the metrics they and their bosses are looking at.

Wanting is hardly ever the issue. It’s mostly can’t, for many different reasons.

There is no need for me to write yet another post about what’s wrong with a Feature Factory and all the nuanced complexity that turns teams into them. I’m more interested in the now what part.

When I ask my favourite LLM or even do an old-school Google on how to escape the Feature Factory, I get things like:

• Change the focus from outputs to outcomes

• Define and align around a clear product vision & strategy

• Invest in continuous Product Discovery

Oof.

I see that these things are needed if I want to make a permanent and sustainable change from the factory, but they also feel a little disheartening. They can seem impossible to get done.

These things are massive and require the rest of the org. I know that that’s the entire point, but what’s the smallest thing I can do today? The thing to get the ball rolling?

The smallest ‘Total cost to get it done’

The Total cost to get it done is a made up bucket term that encapsulates all reasons I’ve heard of not doing something that helps escape the factory:

• Don’t have the budget

• No time in my day

• No time in the day of the people I need

• Don’t have the skills required to execute

• Don’t have the buy-in

• Can’t because it’s not in the strategy

• Can’t do it because it will delay the delivery timelines

The things mentioned just now have a massive Total cost to get it done. They max out on all of these factors.

Instead of investing in continuous product discovery and fully embedding that in the way of working, I could do a one-off sprint. Or even smaller, I could do a one-off activity like a single user interview, A/B test or survey.

These things STILL have some Total cost to get it done though. All of these one-off activities still require either some basic knowledge and tooling, or the time / budget to unlock them.

I have access to AI now though, which can help reduce the Total cost to get it done. Some half decent prompting and a $25 Lovable subscription will get me a working prototype that I can use and share internally to get quick feedback on the thing I’m planning to build.

Though small, it still has some Total cost to get it done. Maybe not on time in my day and budget (I can pay the $25 out of pocket if needed), but what can I do with the outcome of a prototype test if I can’t delay delivery timelines?

So, what is something with an even smaller cost that doesn’t slow down delivery either?

Add a behavioural metric

Chances are that if you’re stuck in the Feature Factory you’re looking at GMV, Conversion Rate, MRR (or none at all). Those are all important things but they’re things I get for a job well done.

They are results for the business. They’re quite hard to influence directly with the things I’m building, if at all.

A behavioural metric sits in between the thing I’m building and the result for the business. It tells me if and how it impacts the behaviour of the user.

Example
For an ecom app we expanded the searchbox to be accessible from every screen, trying to increase the conversion rate (from search). The behaviour we wanted to see is whether it was now easier for more users to search, so one of the behavioural metrics was users that search.

Ideally the research work was done upfront to figure out whether this was worth doing at all. We could’ve checked whether search helps users convert, and if it’s worth expanding it to more screens.

But that’s not the way of the Feature Factory. Too much Total cost to get it done.

I didn’t stop or change what we were planning to build to do research first. That’s the whole point! But I was able to add a valuable feedback loop by adding the behavioural metrics.

What this enables me to do and how it helps me escape

What is the feedback loop and what does it enable? First, let’s look at my options when I built a feature WITHOUT adding behavioural metrics:

There isn’t much I can learn, so there isn’t much I can do either.

When I select a behavioural metric, I’m assuming there’s a link between the user behaviour I’m tracking, and the business impact that will have. More search = more conversion (basically a very barebones hypothesis).

Tracking both enables me to learn and do a whole lot more, depending on whether the link holds. Using the search example from earlier:

Obviously having these learnings isn’t instantly going to prevent me from On to the next feature. But at least I now have some extra information to work with, which unlocks more actions:

• Double down: taking the idea a step further to see how far the link holds. More search = more conversion, even more search = even more conversion?

• Flip: try the opposite direction. If more search = less conversion, does less search = more conversion? Try decreasing the visibility of the feature, i.e. removing it from every page.

• Find the other link: an impact on a business metric is almost always caused by a change in user behaviour. If it wasn’t search, what was it? Find it. It’ll tell me what to do next.

• Iterate: try a different approach to the same link. Maybe adding search to more pages wasn’t it, but autosuggest or better placement could be.

• Investigate: somehow I achieved the opposite of what I expected. It’s worth finding out why first. Could be faulty implementation or there’s something else going on that I’m not yet seeing.

I’m stuck building things anyway (because Feature Factory); I might as well take the opportunity and learn from what I’m doing by adding this metric.

The newly unlocked actions won’t change anything directly, but they will help me make better informed decisions next.

Eventually the ball will start rolling, which will help escape the Feature Factory. Bit by tiny bit.

The learnings from this will lead to better conversations. Those conversations will lead to better questions. Those questions need answers, which will create the need for (justifies) more Discovery. More Discovery will create more learnings, and on the flywheel goes.

Not overnight and not by itself, but what can I expect with basically 0 Total cost to get it done.

Timo here. I write about the slower Product Discovery topics. The essentials, not the latest thing; the internet already has plenty of those.
Figuring things out as I go, 100% written by a human (me), with a new post every now and then.

The midwit meme. It captures a pattern that you can apply to many things:

You start simple, you overcomplicate things as you learn more, then eventually come back to simple (but with better reasoning).

I am the midwit (the person in the middle) when it comes to prioritisation, but I’m beginning to find that “fix problems” might be the better option.

1) I’ve tried a whole bunch of frameworks

At the core, I do prioritisation to achieve two things:

1. Help decide and align on what to do first. Something beyond “because I said so”, because that hardly works when working together with other people.

2. Predict success that the thing I choose to do first, delivers more value relatively consistently over the other things that now have to wait

I went through my archives and found some spreadsheets I’ve used over the last years. Very vintage:

PIE

Can’t remember which client this was but the design and A/B testing focus tells me this was at least 6+ years ago.

PXL

This was the PXL framework for a non-profit. PXL does the same as PIE/ICE/RICE but attempts to make the scoring less arbitrary.

Potential revenue impact

This felt like a big upgrade. I could show the potential revenue impact for each opportunity. Essentially it’s the same as the other ones but with an extra € step.

God knows how I came up with those lift in Conversion Rate (CVR) percentages.

This is just a selection of the things I’ve tried. There are a MANY other prio frameworks out there.

2) Why they don’t work for me and feel like a 🤡 when using them

If I’m being REALLY honest, none of these frameworks have been super helpful.

1) They don’t help decide and align

The sense of rigour and control these frameworks give might work in a perfect synthetic world where everything is controlled and life doesn’t happen.

I might score an initiative a 10/10 and prioritise it, but I’ve found that life always happens. It’s more the norm than the exception.

Here’s a short list I’ve kept over the last few months of the things that actually decided what was prioritised (life happening):

• Because the engineer that was required was pulled into a different project, so we had to change plans to properly utilise the available capacity

• Because leadership said so

• Because a stakeholder needed a favour in return for something I needed a couple of months back

• Because a competitor just launched this new feature so now there’s panic in the management team and we needed to have it too.

• Because a different new thing was raised and everyone agreed that it was an obvious problem with an obvious solution; it overruled the scoring framework

Whenever these things happened the scoring was discarded instantly. Didn’t matter if I used PIE/ICE/PXL or whatever. The spreadsheet wasn’t going to beat any of these things, and these things happen constantly.

Lack of maturity on my end and / or in the organisation? Absolutely. But I’m going to confidently assume it’s reality for more people than just me.

2) They don’t predict success

But there isn’t always something in the way. Sometimes leadership agrees, stakeholders don’t need any favours and things go exactly as planned.

I can proceed with my neat little scoring spreadsheet with a sense of control and rigour.

But here’s my problem with these things:

It accepts guessing. I should fill in a ?/10 if I’m unsure, which should prompt me to go back and test my assumptions first. In practice I start guessing a 6/10 or 7/10 and the frameworks just accepts it. The more complex the framework the more in control I feel, but that feeling is an illusion. That’s worse than just acknowledging I don’t know and go from there.

The scoring is arbitrary. How do I differentiate between a 6/10 and 7/10? Yet that tiny difference is exactly what determines what goes now and what goes next. I’ve also seen 6/10 features have much more impact than 7/10 features, multiple times*

They’re useless when it’s obvious. The times when I didn’t struggle is when I confidently filled in 10/10s across the board, but those features never needed a prioritisation matrix in the first place. It’s the mushy middle where I need guidance.

The framework adds no value in any scenario. It either accepts my guessing, struggles with the mushy middle where I need it most and things are uncertain and ambiguous, or confirms what was already obvious 🤡.

*Admittedly I don’t have any long term data about this, I should keep track of that sometime.

3) What did work for me: ditching the spreadsheets

So spreadsheets aren’t super helpful for me. But prioritisation in itself isn’t useless; I still need to decide what goes first and predict what will work.

Not everything we prioritised was pushed back by real life happening, or failed to show an impact. There were times when those two boxes were ticked quite smoothly.

Not effortlessly, but without too much pushback and with satisfactory outcomes: we expected wins and got them fairly consistently.

We didn’t use a prioritisation framework, because there was no need.

That happened when we had obvious solutions for obvious problems.

What do I mean with those?

Problems so clearly defined that anyone can see why they matter. Simple language, strong evidence that the problem is real, and a validated solution lined up that is targeted at that specific problem.

When both the problem and solution are obvious, there is no need for a framework because there is no ambiguity to resolve. Scoring spreadsheets try to combat ambiguity and create certainty through scoring, but the certainty comes from the evidence, not a number from a formula.

Some people call these “no brainers”, but I think they’re actually “max brainers” because it requires hard discovery work for something to be the simple, obvious choice.

Example
A market research company I worked with a while back had a whole list of ideas with no clear prio. I found the messaging throughout the lead site extremely vague (common issue with B2B) so I ran a quick user test to gauge messaging clarity on the most popular landing pages.

The results were terrible. When asked what they thought the site was about, 3 got it right, 19 were in the right direction (I was being very liberal) and 28 were flat out wrong (with my fav answers being “Beauty services” and “Not sure: was this an investing site?”). This was a market research company.

We ideated some new directions and ran the same test. Clarity improved ofc (else I wouldn’t have picked this example), but the thing that stood out was the ease with which the existing ideas were discarded to take on this new obvious problem.

Doing the discovery work to make the problem and solution obvious helps with both of my prioritisation goals.

(”Make the problem obvious” sounds easier than it is. It requires synthesising insights into opportunities / problems which is its own skill: check out the separate post.)

1) It decides and aligns

The framework to choose a framework article I shared earlier has this nice paragraph:

“The biggest value of the frameworks above is the ability to create a forum for conversation, a place of psychological safety, and a frame for creativity. The outcome should be less about the list of features you decide to create, and more about having a dialogue for the right internal and external stakeholders to voice what they think matters the most and why.”

I agree about the forum for conversation. But in practice, I’ve found that making sure there’s a shared understanding about the problem - so making sure the same data is available and understandable for everyone - makes this so much more productive (and fun) than talking about numbers from a scoring spreadsheet.

The data from the example showing that only 3/50 understood the messaging, and that it improved in the variant made the problem and solution obvious. That’s what got everyone to align on the prio.

I still can’t control when life happens, but I found that challenging life’s events (e.g. leadership call, shift in capacity etc.) is much easier with this data.

2) It predicts success (better than guessing numbers)

By the time we prioritised fixing the messaging, the discovery work that led up to that decision already:

• proved that the problem was real (only 3/50 got it right)

• made sure we would hit a substantial group (most popular landing pages)

• had a validated solution lined up (clarity improved in the follow-up test)

It was still a bet - everything is - but it made it much less of a prediction game. Instead of entering random numbers in the scoring spreadsheet, we already had a good feel for Reach, Impact and Confidence through the discovery work required to make the problem obvious.

Maybe that’s my problem with these things: a framework is not a bad thing in itself. There is nothing wrong thinking about factors like Reach, Impact or Confidence.

It’s just that these answers come from doing the hard discovery work, not from filling in the spreadsheet. THAT is where the effort should go when prioritisation is hard.

Borrowing a visual from my first post about the misuse of AB testing, maybe tinkering with framework and scoring spreadsheets, is yet another sign that you’re not doing enough Product Discovery work.

If you found the house is on fire, you don’t need a framework to tell you to prioritise the extinguisher. The discovery work already made that common sense.

Summary: fix problems

• Maybe I just haven’t found the right one, but scoring ideas via prioritisation frameworks makes me feel like a 🤡

• They don’t actually help align and decide: life happens + I found stakeholders align more easily around data about the problem than numbers from a scoring spreadsheet.

• They don’t actually predict success; the scoring is arbitrary, I fool myself and fill in a 6/10 instead of ?/10 when I don’t know, it struggles with the mushy middle where I need it most or it confirms what was already obvious

• I’m shelving prioritisation frameworks: I should spend my efforts on discovery work, finding obvious problems with obvious solutions. Certainty comes from the evidence itself, not from numbers in a formula

• Personal reminder for the future: if prioritisation is hard and I find myself tinkering with frameworks and scoring spreadsheets, perhaps it’s a sign I need more product discovery. The research will make prioritisation common sense: fix problems.

Is this read for you? TL;DR of these 2K words:

• If you want stakeholders to act, you need opportunities, not fragmented insights

• Synthesising insights into opportunities is similar to solving a mystery jigsaw puzzle:
  1) you need all puzzle pieces (UX, quant, qual) to complete the picture.
  2) if synthesis feels like a mess, you’re doing it right.
  3) you can’t control what’s in the picture. Your job is completing the picture.

• Framework first, some real life examples after

Opportunities are the bread and butter of Product Discovery. It’s where all the hard research work you’ve done so far finally comes together.

How do you go from insights to an opportunity though?

Individual insights are fragments of a bigger picture: the opportunity. To reveal that picture, you need to go through the process of combining, puzzling and making sense of it all: the synthesis.

Why is it so important to get this right?

Stakeholders simply don’t act on (or care about) just insights alone.

The value of all that data you collected and all those insights you drew is locked up until they lead to (better) decision making, for which you need opportunities.

Synthesising insights into opportunities is like solving a mystery jigsaw puzzle

Image source: boardwalkpuzzles.com

Do you know mystery jigsaw puzzles? It’s a regular jigsaw puzzle but with a different or even no picture on the box. You have all the pieces, but you have no idea what picture is shown until they’re all nicely fitted together.

I found that synthesising insights into opportunities works similarly. The puzzle pieces being the insights, the complete picture that is revealed once they’re all fitted the opportunity.

This is how they’re similar:

1. You need all the pieces to complete the puzzle

There are 3 types of puzzle pieces, and we need all of them to complete a puzzle and reveal the opportunity:

• The UX puzzle pieces answer the what, when and where. Without these, you have no tangible sense of the opportunity in the user’s situation

• The quantitative data puzzle pieces answer the how often or how much. Without these, you have no idea about the scale or magnitude of the opportunity

• The qualitative data puzzle pieces answer the why.Without these, you have no idea of the underlying cause

Check out the previous post if you want to know more about these data types.

Omit any of them and you’ll have:

an incomplete picture and understanding, which means

..unanswered questions and uncertainty, which means

..surprises later down the line (“oh, this feature actually only affects 1% of users?”) worst case, or a total lack of action from stakeholders best case

2. It’s a mess and that’s a good thing

Synthesising insights into opportunities is nothing like a 3-step linear process. Don’t think linear LEGO instructions, think chaotic jigsaw puzzles.

Pieces are scattered everywhere. They’re picked up, fitted, removed, rotated, picked up again and thrown back to the pile.

Entire sections are moved around based on how other sections are turning out and sometimes you just need to step away and return with fresh eyes before a piece clicks.

Synthesis is similar. One insight can affect how you interpret the other, sometimes you need to complete the qual study before you can do the quant study, and sometimes the opportunity only makes sense once you’ve fitted the final piece of a completely different puzzle.

Insights unlock each other. If it feels chaotic, you’re doing it right.

3. You can’t control what is in the picture, only that it’s complete

The whole point of a mystery puzzle is that you don’t know what’s in the picture.

Similarly, we don’t know upfront whether the insights are going to reveal a high-value shiny opportunity or a disappointing rainy-day one. That’s not our worry during the synthesis.

Our worry is completing the puzzles so we’re in a position where we can properly assess that.

The goal is better decision making, even if that means deciding not to proceed with an opportunity because it’s not as nice as you hoped. That’s ok and arguably the whole point.

Up next are 3 real life examples. One for each of the talking points we just went over:

• Example 1 is very simple and shows you need all 3 pieces to complete the puzzle

• Example 2 shows the hot mess

• Example 3 shows a complete picture that’s not so sunny, but still valuable

Example 1: Users can’t pay for their concert tickets

You’re the PM of a concert venue that sells tickets through their website.

Let’s say you’re checking your weekly analytics reports and see a drop of -29% in the Checkout to Purchase rate.

Thats your first quantitative puzzle piece. You know how much something is hurting Checkout to Purchase, but no further specifics.

You filter your survey feedback to that week and see something like this:

You now have two pieces: you know checkout conversion is dropping by -29% (how much), likely because “PayPal wasn’t working” (why).

Yet we don’t really know what “wasn’t working” means. We’re missing the what, when or where of the UX piece.

Thats the final piece of the jigsaw puzzle. We can now see what is in the picture. The latest release which made the checkout CTA sticky blocked the PayPal payment option, causing a -29% drop in Checkout to Purchase.

This is obviously a simplified example which shouldn’t happen in more mature teams (though you might be surprised how often simple things can still break). The point is that we only understood what was really happening after we nicely fit all the pieces together.

Example 2: Users can’t find anything that they like

You’re the PM of a marketplace for secondhand design stuff.

Let’s say you did a usability test focused on the cart. Funny enough, what really stood out was users trying to find a product that they like before they even got to the cart.

They just scrolled aimlessly and end up selecting a random product for the sake of completing the usability test. You’re confident that your catalogue has something that they like; so what’s the problem?

We could try fitting the quant puzzle piece and verify what we observed using analytics, but what would we be looking for? Conversion rate? Add to cart rate? Product category views?

“Users aimlessly scrolling” is a too vague to quantify for now. We need more specific questions first.

We can’t make the quant puzzle piece fit, yet.

Later that week you ask a friend to pull out their phone to try and find something that they like. You focus specifically on the UX, and how they engage with the pages and features. You observe similar aimless scrolling as in the user test, but you also notice:

• They get stuck in the top most-generic level of categories: All Products, New in, Sale. They didn’t use any of your quick links to get to deeper, more specific categories.

• They don’t care about your Style pages which could help them identify what they like and find more specific and niche products.

• They miss the filters for price, to nudge them into filtering down to a list of products specific for their budget

You have all these nice features to help users find what they like, but nobody sees or engages with them.

These UX observations give us the specific questions we missed before when trying to fit the quant puzzle piece.

Are feature engagement rates low across all users?

Yes, absolutely.

How many products do people typically view?

Only 5% of users view more than 1 product.

Do users return, or give up permanently?

Only 15% of users return within a week.

Turns out, the opportunity revealed now that all pieces are fitted is not that users don’t like what we have on offer. We do have products that they like; we just fail to surface and show it to them (fast enough).

Seeing these numbers and the UX in practice now totally explain the aimless and disinterested scrolling from the participant in the user test; we just had to cycle through the pieces:

• Qual: initial observation about aimless scrolling

• Quant: tried to fit this piece, but we missed specific questions

• UX: observed our friend not seeing / engaging with our discoverability features

• Back to Quant: confirmed this in the data thanks to the more specific questions

• Back to Qual: now the aimless scrolling makes total sense

Fitting a piece can sometimes help you fit others better, just like how fitting the outside pieces of a jigsaw first help make sense of the inward pieces.

Example 3: Users don’t care about the chatbot

You’re the PM of bookkeeping software for freelancers.

Lets say your app has a simple chatbot to answer common bookkeeping questions that has been live for a couple of months:

The founder is aboard the hype train and now wants to expand the bot with more extensive AI features. They want the bot to be able to answer more complex tax questions and enable it to give projections by integrating it with users’ invoice data.

You’re excited to give this a go. You don’t directly commit 3 sprints to building though; you do the synthesis first.

The survey results are promising: 64% of users say they want better help with bookkeeping questions. A strong initial qualitative piece.

The UX piece also fits nicely. A prototype you built was well received by users. They understood how to ask questions and considered the responses accurate.

Th quant piece delivers a reality check though. You see what’s really in the picture:

• The current chatbot works well when used (80% resolution rate), but only 2% of users have ever used it.

• Internal help pages that also answer complex tax questions hardly get any traffic: 100 views in the last 6 months

• The community forum only had 2 posts in the last 6 months about tax questions

Users need better help with bookkeeping questions, but there is no evidence they want it inside your app. They don’t care about the chatbot. They’ve already solved this problem elsewhere: google, the official site of the government and, of course ChatGPT.

Expanding and improving this feature (with AI) doesn’t magically create the demand for it. No matter how cool this feature would’ve been, it likely would’ve gone unused. It’s not worth trying to compete with ChatGPT.

Disappointing sure, but like a mystery jigsaw puzzle: we can’t control what’s in the picture.

We can only control that the picture is as complete as possible so we can properly assess whether the opportunity shown is worth chasing. Sunny, cloudy or rainy: they’re all valuable when it comes to decision making.

In this case, that means not committing to building the AI bot. A decision that saved us 3 sprints worth of work that can now be allocated to a different (more shiny) opportunity.

Recap

Individual insights are just fragments of a bigger picture: the opportunity To reveal the full picture, you need to go through the synthesis.

Synthesis is like solving a mystery jigsaw puzzle. The insights are the puzzle pieces, the complete picture that is revealed once all pieces are in place is the opportunity.

Next time you’re solving a puzzle / synthesising insights, remember this:

1. You need all the pieces to complete the puzzle

• You need the UX pieces; without them, you miss the what, when and where

• You need the quant pieces; without them, you miss the how often or how much

• You need the qual pieces; without them, you miss the why

Omit any of them and your picture is incomplete.

2. It’s a mess and that’s a good thing

Insights unlock each other. If it feels chaotic, you’re doing it right. Embrace it.

3. You can’t control what is in the picture, only that it’s complete

The whole point of a mystery puzzle is that you don’t know what’s in the picture.

Your job isn’t to improve what’s in the picture; you can’t. Your job is to complete it. Once complete, you can properly assess the value and make informed decisions, even if that means trashing the entire thing.

Is this read for you? TL;DR of these 4.8K words:

• Frameworks are great, but there are 100+ of them. It’s noisy and overwhelming.

• You really only need 5 that work really well together:

• Triple Diamond (bigger picture), Landscape of User Research Methods (better questions), Levers (identify patterns), Atomic Research (strengthen signal), and Opportunity Solution Tree (connect your work)

• This deep dive gives you a strong foundation in the discovery process and its key principles; you’ll know what (not) to do to start delivering value

We ended the previous post saying that shit ideas in = shit results out. The only way to break the cycle is doing more Product Discovery.

It’s easy to get stuck when figuring out the doing part though. There are 100+ fragmented frameworks out there. It’s hard to see which ones you really need or how they’re supposed to work together.

Collecting them all is boring, plus I think you really only need 5 to get from Research questions (the starting point) to Opportunities (the endgame).

To start delivering value, all you really need is:

1. First see the bigger picture, The Triple Diamond

2. Ask better questions for better insights, Landscape of User Research methods.

3. Make sense of those insights: identify the patterns, Levers Framework.

4. Strengthen the signal by stacking your research methods, Atomic Research

5. Manage and properly connect all your work, Opportunity Solution Trees

You’ll also see that if you really understand these 5, you probably understand 90% of all the other frameworks out there too.

We’re going in deep. By the end of this post this abomination is going to make total sense:

1. First see the bigger picture, The Triple Diamond

We’re going to take on this one first because it sets us up with the basic context in which we can place all the other things coming up. It’s also a fundamental piece: you can’t not run into this when doing Discovery, so it’s good to get familiar with it.

Source: Bootcamp on Medium

What’s going on

It tells us which phases we go through if we want to get from a problem to a solution, which is exactly what we need to do. It just describes the phases though, it doesn’t exactly tell us how we do the work. We’ll get to that later.

• The 1st phase is Problem Discovery. First you find and properly understand the problem (also called Problem Discovery), then you make the problem concrete in Opportunity Definition.

• The 2nd phase is Solution Discovery. First you discover potential solutions, then you decide which one to go for.

• The 3rd phase is Development. First you build the solution, then you deliver the solution.

Problem in, solution out. No need to further complicate things.

Everything that is coming up next in this chapter (and almost everything in this entire guide) takes place in that 1st Problem Discovery phase.

The value for us in practice

This framework is a solid reminder of some of the core principles of Product Discovery:

1. There’s a certain sequentiality in which to do things

2. There’s a point when you’ve done enough problem or solution discovery

1. There’s a certain sequentiality in which to do things

You start with a problem to solve, then you find a solution, then you build it. Sounds logical? Wait until you see teams starting at phase 2 or even phase 3 wonder why the things they’re building don’t have any impact; they skip phase 1.

But I guess that’s why you’re here.

2. There’s a point where you’ve done enough problem or solution discovery

You can discover (diverge) forever, but you need to start defining (converge) at some point if you ever want to deliver value to users.

It’s a trap to always be looking for more data and insights. A trap you ABSOLUTELY will run into when starting out. And yes, this is exactly why the framework uses diamonds- to visualise the diverging / converging cycle.

Its simplicity makes it very shareable with your colleagues. It will help get these 3 principles across. Throw this thing in a presentation as a suggested way of working when your team is skipping phase 1 for example. Just once though, no one wants to hear about your processes all the time. It’s boring.

An easy start. As I said: it’s a fundamental, and we’ll further build upon it over the next chapters. I hope you’re warmed up because the next framework is going to be a lot more detailed.

2. Ask better questions for better insights, Landscape of User Research methods

Now that we have basic context, let’s look at the first 3 steps: Research questions, Research methods and Data & Insights.

As I said, everything in this chapter takes place in that first Problem Discovery Diamond. This next framework by Christian Rohrer (NNgroup) is for that first half of that first diamond: It will help us find problems to solve.

Source: NNgroup

Just look at this three-dimensional beast. It looks a little complex, but we’re going to ignore some parts of this framework for simplicity’s sake.

This is going to be a beefy one, but if you REALLY get what’s conveyed here you’ll have an easier time understanding the things coming up later.

Stick through this part, and the rest will be a breeze.

What’s going on

It’s basically “just” a categorisation of research methods you can use. 4 areas:

1. Behavioural & Qualitative (top left)

2. Behavioural & Quantitative (top right)

3. Attitudinal & Quantitative (bottom left)

4. Attitudinal & Quantitative (bottom right)

An explanation of these are in the next bit. We’ll ignore Context of use (the colour-y and shape-y part), for now.

The area in which a research method sits is conveniently also the type of data it collects. So Usability Testing sits in the top left area, which means it collects Behavioural and Qualitative data. We’ll get to what that means and why that matters soon.

Even though this visual is about research methods, the real power is in the research questions.

I think it can be a trap to think in research methods (“We’ll use GA4 and Hotjar!”), because just browsing analytics reports or looking at session recordings without a clear research question or goal is like looking for a needle in haystack.

The quality of insight you’re able to extract from data, is generally only as good as the quality of the research question you’re asking. You need to ask better questions, which is what the second visual of this framework helps you with and where the real value is.

The value for us in practice

This framework helps you:

1. Ask better research questions

2. Pick the right tool for the job

1. Ask better research questions

Source: NNgroup

This second version follows the same 4 area categorisation as the first one, but looks at questions instead of methods.

This is where it gets good, because this tells you what kind of questions each method can help you answer:

• Behavioural data = What people do questions

• Attitudinal data = What people say questions

• Qualitative data = Why & How to fix questions

• Quantitative data = How many & How much questions

Which equals what type of insight you’re getting, which is what we’re REALLY interested in. More extensive explanation below: feel free to skip if you’re already 100% fluent in this.

SKIP if 100% fluent: The type of insight your getting from
Behavioural vs Attitudinal & Qual vs Quant

Do you want facts or opinions? Behavioural vs Attitudinal

These are questions that try to figure out What people do vs What people say.

I’m sure you are very aware that what people say they do vs what they actually do are two very different things. Do you want objective evidence of whether something is happening, and how? Bypassing biases, social desirabilities and whatnot? Then you’ll likely need a What people do question. Users are paying for your product, or they are not. That’s behavioural.

Sometimes those biases and social desirabilities is exactly what you’re interested in. You want insight in motivations, thoughts or opinions. You’ll need a What people say question. Users talking about what style of furniture they like. That’s attitudinal.

Do you want to observe directly or indirectly? Qualitative vs Quantitative

These are questions that try to figure out the Why & How to fix vs the How many & How much.

Qualitative vs quantitative tends to be seen as written text / spoken words (qualitative) vs calculated numbers (quantitative). That’s not entirely wrong but it can get a little muddy. As mentioned in the article, the distinction is clearer if we categorise these as direct (qualitative) vs indirect (quantitative).

Do you want to be able to ask follow-ups or change direction based on what the user is telling or showing you, so you need to be there in the room to observe it directly as it happens? Then that’s likely a qualitative question. These tend to be the Why? or How do we fix this? kind of questions.

If you don’t need to be there as it happens then that’s likely a quantitative question. These are questions that can be answered indirectly, like checking your analytics tool or survey results. The observation already happened instrumentally (at scale) without you being there. Because of that, these questions tend to be the How many / much? kind of questions.

Thinking with the end in mind (the insight you need) and seeing your four options side by side makes it way easier to ask better research questions.

2. Pick the right tool for the job

There is a reason why we spent some extra time on getting the types of questions and insights right, because picking the right tool for the job is now super easy.

Instead of deploying a research method and hoping for the best, you can now work backwards from the end. From insight to question to data to method:

1. Which of the 4 types of insight do I need?

2. What kind of question will help me surface that?

3. What type of data do I need?

4. Which research method or tool can help me deliver that? Which area should I choose from?

OK GREAT, that took longer than I expected. It was necessary though. You’ll recognise the things we discussed here throughout the rest of this guide, so it was time well spent.

Still with me? Good news: things are going to be a bit easier from here on out.

3. Make sense of insights: identify the patterns, Levers Framework

Next up: Levers.

Levers are fun because they can be used in two ways. They can be used both as a starting point to find problems to solve (the first half of the Diamond), and as a means to make sense of the things you found (the second half of the diamond). You’ll see shortly.

This framework from the agency Conversion is relatively new to the scene (2022?) and I LOVE it. You’ll see that there are many things out there that have a different name but do roughly the same thing: themes, heuristics, optimisation strategies etc.

Source: Conversion

What’s going on

The simplest way to put it is that levers are like a language.

It names all the different parts that when combined, form The User Experience (see top row: Cost, Trust, Usability, Comprehension, Motivation and all its subparts).

Just like how The Engine, Chassis, Transmission, Electronics and Interior combined form The Car.

It gives you a richer vocabulary, which allows you to articulate more specifically. The more specifically you can articulate what is going on, the easier it is going to be for you to take effective action.

Just like the mechanic who can point to the Fuel pump instead of the entire car as the problem and replace just that part, you can point to a sublever of Usability instead of the app as the problem and focus on just that lever.

The value for us in practice

The benefit of a richer vocabulary:

1. Levers help you identify patterns and form a discovery strategy

2. Levers are a great starting point for research questions

1. Levers help you identify patterns and form a discovery strategy

That ability to specifically articulate what the problem is (Usability > Effort > Anticipated effort VS app broke) also has a long-term benefit next to just being able to take more effective action.

You will start seeing patterns across all your data & insights. These patterns can in turn be used to help form a broader discovery strategy:

• “The Comprehension lever worked really well for X part of the product, let’s see if it also works for Y part of the product”

• “The last 5 attempts at improving the Comprehension lever didn’t yield positive results, let’s move on to a different lever”

• “Looking at last month’s feedback, we see a shift in feedback from the Comprehension lever, to the Usability lever. Let’s allocate some time next quarter to further investigate what’s going on”

It’s the vocabulary that allows you to identify these patterns.

2. Levers are a great starting point for research questions

As we said in the beginning of this part; Levers are not just for making sense of the stuff you found, they can also be a great source of inspiration for research questions.

When starting fresh, figuring out (or ruling out) whether there are any Friction issues is a very decent starting point. You’ll inevitably find more nuanced and detailed stuff as you go, but if you’re starting fresh the point is to get going.

And yes, you can combine this with the Landscape of User Research methods framework if you want. Some examples:

Apart from using this one you can also create your own version of levers. In fact, most orgs I’ve worked with do. Something that clicks with the wider strategy and language already used internally.

You’re also completely free to not use Levers at all. From an operational perspective they’re completely optional: you don’t need them to get the work done, especially if you’re a little more experienced or prefer a more pragmatic approach.

That’s fine. If you have insights indicating that something needs improvement, feel free to just go ahead and fix it. Just know that you won’t profit from any of the benefits we just discussed.

4. Strengthen the signal by stacking your research methods, Atomic Research

Let’s take a closer look at the operational process to tie up everything we talked about so far. Literally.

This next framework by Daniel Pidcock is basically a zoom-in of that first diamond.

Source: Prototypr on Medium

What’s going on

It’s actually not that different than the 5 steps we started this post with, just with slightly different language.

• Experiments = Research methods, though experiments is probably a better name if you also consider that things tend to run a little more in loops in practice

• Facts = data (not exactly, but no need to nitpick here)

• Insights = Insights

• Conclusion = what we called solutions in the previous post. Same same.

Research Methods, Levers and Opportunities are missing, but we’ll get to that shortly.

Its power is in the visualisation: the tying concepts together in the many to one relationships.

The value for us in practice

This framework teaches us something vital about the process: you gotta stack your research methods.

1. You gotta stack your research methods

Let’s say you did a user test and the participant mentioned the icons are helpful. It’s one of the many things said during the study; you jot it down and move on.

By itself it’s a nice compliment for the designer, but not a strong signal that indicates you have to take action IMMEDIATELY.

If you were to draw your insights from just the data of this user test, it’s likely this piece of data would’ve made it into an insight.

You base your conclusions on the things that did make it to an insight, ideate solutions, implement them and move on.

In a different month, maybe even in a different team, something different happened.

An A/B test was conducted that had a positive impact on all sorts of important goals. It also had a positive impact on engagement: +40% more engagement when icons are added to buttons.

There are now 2 pieces of data signalling that icons are a thing. If you see them together, you’d draw the insight “Iconography seems to aid comprehension”.

The problem is that they’re never combined because the research methods weren’t stacked; they’re separated by time or even by teams.

You did the user test and implemented all the recommendations of that study at the end of the month. By the time it’s March you’ve moved on to other things and forgot about that participant saying icons are nice.

Stacking your research methods circumvents this. It not only enables you to draw that insight in the first place, it also unlocks signal strength. If you deploy 5 methods and all 5 of them are saying icons are thing, it’s probably a BIG thing. Not only that, different methods collect different types of data (we talked about this: quant vs qual, attitudinal vs behavioural) which are like different pieces of the same puzzle. You need all of them to see the full picture.

This stacking and tying together applies to all steps of the process:

• Multiple questions can tie to a single research method: this is likely method that will yield multiple answers

• Multiple research methods can tie to a single fact: this is probably a valuable piece of information as it shows up multiple times

• Multiple facts can tie to a single insight: this is likely a strong insight

• Multiple insights can tie to a single opportunity: this is likely a strong opportunity

• Multiple opportunities can tie to a single solution: this solution potentially solves multiple problems at once

More ties = more signals = likely more prio.

Source: Prototypr on Medium

Visualising this using all the steps creates this modified Frankenstein version. It doesn’t do justice to the original work, but it’s for education purposes I guess.

If you’re seeing this: sorry for butchering your beautiful framework Daniel.

Final note: stacking methods doesn’t mean you have to do everything in the same week. It means considering the insights from different methods and moments together as one (stacking). It’s the breaking down in smaller parts that this framework does that makes stacking easy to do that in practice.

One more to go. Almost there!

5. Manage and properly connect all your work, Opportunity Solution Trees

I saved my favourite for last. It looks simple which is good. It’s also very practical; you’ll use it basically every day when you’re doing the actual work.

I’ve based this entire section on the first article I ever read on Teresa’s website, which I found eye opening and led me to use the Opportunity Solution Tree (OST) basically everywhere I work.

Her website and newsletter are both excellent and should be on your reading list. Better yet, just buy her book Continuous Discovery Habits. It’s a fundamental read and simply will make you better in Product Discovery.

Source: Producttalk

What’s going on

You’ll recognise the tree-like structure from Conversion’s Lever Framework. It helps you see the connections between all the different parts in the discovery process:

• Desired Outcome (blue) = we called this a goal up until now (e.g. sell more concert tickets). That’s fine, but outcome is better language. Outcomes (over outputs) if a whole post on its own so just roll with it for now.

• Opportunity (green) = the problems your users are experiencing. A bigger topic you’re going to commit to, to try and fix or leverage. Opportunities are super important to get right, so the entire next post is going to be just about opportunities.

• Solutions (yellow) = the solutions you’re building to help fix those problems; ideas. In the previous post we said that it helps to see ideas as solutions; this is where we stole that from.

• Experiments (orange) = the experiments you’re running to see if those solutions actually work.

It also helps you to really, really, REALLY see all of these as separate entities part of single chain. You can’t confuse one with the other, you can’t have them floating around unconnected, and you (usually) can’t proceed without having all parts fleshed out.

This may sound obvious, but this is something that goes wrong in practice basically all the time.

The value for us in practice

There’s a ton of great content about the benefits of using OSTs. I’m not going to summarise all that, so I’ll share my personal perspective here.

I’ve mulled this over much longer than healthy but I think the core value of this thing, for me, comes down to seeing when your work doesn’t connect properly (and that’s HUGE).

1. It helps you see when your work doesn’t connect properly

And with see I mean literally see. It’s the visualisation. The laying out and connecting out all your work in a neat and single overview that reveals for you when something has gone rogue.

With connect I mean how your hard discovery work ties into the things you said you’d do this quarter and to the thing you’re going to do today to achieve that. It makes discovery the bridge between the two. Its strategy made simple and visual.

Here are 3 examples to bring this to life. Sorry for the cliché topic, but it just makes things easier to explain:

1) The solution that can’t be tied to an opportunity: Buy new hiking gear

So this happens at least once a week no matter how mature the organisation is. This is the result of idea-first thinking, and always is something that is really cool and shiny, but doesn’t solve a real problem your users have. You can’t tie it to an opportunity in your tree, so chances are small it will really help you drive the outcome. I would love new hiking gear, but sadly it won’t help me lose 1KG. Not really.

I could argue that I will hike more once I have them, but unless I have data that that is actually a problem, I’m just reasoning backwards from a solution to an opportunity. That’s a slippery slope.

In practice, this ranges from homepage carousels and chatbots back in the day, to “let’s do something with AI” today. Ideas like these aren’t inherently bad, you just have to be cautious when this happens. Do the work and see if the opportunity exists and it truly helps your users or if it’s just the next cool and shiny thing to have.

2) The opportunity that can’t be tied to the outcome: Eating out too often

Working very hard on irrelevant problems. This too happens regularly and is quite the tragedy. This is the result of not understanding the opportunity and the impact that it will have on behaviour and the outcome well enough. Sure, eating out less often might indirectly help me lose 1KG, but they’re not REALLY that well connected. It’s just as easy to take in too many calories at home as it is when eating out.

You can spend weeks home cooking all your meals every day. That’s a ton of effort and you would’ve saved a lot of money doing so. That money saved is a great outcome, just not the one we’ve been tasked to work on.

In practice this means spending weeks improving the loyalty programme whilst your goal was to increase the conversion of the onboarding flow. Doing a lot of hard work without having an impact to show for it in the end is risky business. You need to fully understand what metrics will shift if you were to change something, and whether that metric is the one you’re interested in.

3) The single solution connection: Count calories

If you have an opportunity with only a single solution, chances are you’re just going with first thing you came up with. That can be OK, but there’s a ton of value in considering and trying more solutions.

Counting calories is really annoying. So if I fail, what will I then do with the problem “eating too much crap?” Will I conclude that the problem can’t be solved or isn’t really there? And what if I succeed counting calories for 2 weeks? Does that mean the problem is solved and I move on to the next?

This sounds lame yet it happens every day in practice. Teams move so quickly that there’s hardly any room for iteration, yet it takes several attempts before you understand what works and what doesn’t.

It doesn’t really matter what the result of your first solution is. If it works, you keep going to see if you can push it even further. If it doesn’t then you know you need to try something else.

Improving digital products is not a does work vs doesn’t work, but a works better vs works worse kind of thing. Your first go is hardly ever the best one, so be open and try multiple solutions.

There are more situations where this tree helps you with the process, but I hope you’re convinced. It’s incredibly powerful and has many more benefits than “just” helping you connect all your work properly. If you could only apply 1 of all these frameworks we discussed, please let it be this one.

Recap

We did it.

The Triple diamond framework helps us see the bigger picture

It teaches us there’s a certain sequentiality in which to do things (start with problem, end with solution) and that there’s a point where you’ve done enough discovery and you need to get concrete: the work flows in a diverging to converging cycle (hence the diamonds).

The Landscape of User Research helps you ask better questions for better insight

The better the research question, the better the insight. The better the insight, the better the opportunity. The better the opportunity, the better result.

The trick is knowing what question to ask as it sets the whole thing off. Seeing your options categorised in 4 areas helps you with that: are you interested in what people say and do you want to ask follow-ups, or are you interested in what they do and need numbers at scale?

The Levers framework helps make sense of the insights and identify the patterns

It’s hard to spot and identify insights in the data if you don’t have the vocabulary to call them out. Better vocabulary means better diagnosis, which means more effective problem solving. Don’t replace the car, replace the fuel pump; your app isn’t broken, the usability in your onboarding flow is.

The long term benefit of this is that you’ll start seeing patterns in the UX, which help you form a discovery strategy. These levers also form a great starting point for research questions.

Atomic research teaches us to strengthen the signal by stacking the research methods

It’s a zoomed-in version of that first Problem Discovery diamond. It ties all the individual steps together and adds many to one relationships. The stacking enables us to draw insights that otherwise wouldn’t have made the cut, and it unlocks signal strength. More ties = more signals = likely more prio.

The Opportunity Solution Tree helps manage and connect all your work

Seeing the work mapped out in front of you helps you spot when things go rogue like solutions without opportunities, opportunities not connected to the right outcomes, or opportunities with just a single solution. All the work comes together in this tree. Getting the tree right means getting your strategy right.

Look at that.

The 5 frameworks nicely fit together. There are 100+ other really nice frameworks out there, but I honestly think that with this you have everything you need to start delivering value.

Next up, a detailed look at the process of the synthesis: turning insights into opportunities

Is this read for you? TL;DR of these 2.4K words:

• 3 patterns that signal you need more Product Discovery: Solution-first thinking, Misuse of A/B testing, and Shipping & forgetting.

• They can reinforce each other in a cycle. Problem understanding helps you do better.

• AI is going to make this more urgent. Product Discovery is going to prevent you from working very efficiently on very irrelevant problems today, and it will be the differentiator tomorrow. The time to start is now.

Here’s a bad week:

• Hearing my non-tech-savvy friends say “I JUST WANT IT TO DO X” as they struggle to work their way through increasingly bloated websites and apps.

• Two sprints worth of story points spent on a feature that ends up being ignored by users, but celebrated anyway because it was delivered on time.

• Long and draining meetings refining solutions that excite nobody. Not even the people working on them.

Building digital products is a complex mess, and there are probably a hundred things at play that explain why things go the way they go.

I don’t pretend to know all of them, but I personally keep running into these destructive patterns which sometimes run in a cycle:

1. Solution-first thinking

2. The misuse of A/B testing

3. Shipping & forgetting

1. Solution-first thinking

Let’s say you’re the Product Manager of an e-commerce platform that sells concert tickets.

If you limit the things you can influence to just the changes on the platform itself, there are still a million things you can change that could potentially help you sell more concert tickets.

Join any meeting in any organisation and you’ll learn that the challenge is not coming up with these million things to change. There’s never a shortage on ideas. Never. Peoples natural reflex is to jump straight to the solutions.

It’s obvious when an org suffers from this, because they skip all the bits that have something to do with the user and their behaviour in their discussions, hypotheses and plans:

• We see a drop in X, what can we do?

• We expect that by building this, we will increase Y

• The goal is to increase Z. Lets brainstorm some ideas to help us achieve that

What’s missing in all of these is what making a change or building a feature will do to user behaviour, which in turn, might impact (financial) metrics.

Ideas generally only have an impact if they remove or fix something that’s blocking users, or if they add something nice that pushes users to complete their goal.

People don’t care about:

• new CTA colours, unless they’re actively looking for them but can’t find them due to contrast issues.

• a new search bar, unless finding items via search is both something that they do and this helps them find what they like more effectively.

• an AI-powered dashboard, unless summarised insights is something they need and this genuinely saves them time on that task.

You get the point. If you don’t add value for them, they’re not gonna care, which means no impact on your metrics.

Almost every idea is a means to an end if you think from the perspective of your users (which you should). The better you understand the end, the easier it is to find the right means.

That’s relevant because the challenge is not coming up with yet another idea, the challenge is filtering the right one from the endless stream of ideas. Discovery makes that easier for you.

Most ideas are solutions that try to solve a problem thats non-existing for your users, or don’t even pretend to solve a problem at all.

Basically this.

2. The misuse of A/B testing

So how do you decide which of all the ideas to pursue when there is no clearly defined problem to measure the effectivity of the idea as solution against? Just test it.

This is either played as diplomatic solution when teams can’t agree which idea to pursue, or when none of the proposed ideas seems the best. The problem in these two scenarios is the same. There is no notion about the underlying opportunity or problem.

This shifts the game from selecting the solution most fitting for the problem, to selecting the idea that seems the best, where the best for what? is unclear.

More efficient experimentation and delivery processes enabled us to just go and test all these ideas, and we’ll proudly call it a culture of experimentation while we’re at it. The A/B test velocity as indicator of how well we’re doing.

This is no problem if you’re in an organisation thats on the good side of The Experimentation Gap, when you have the luxury of everything that the scale you’re at brings. Traffic, sound statistics, plenty of smart people, your own testing platform, experimentation neatly embedded in the build process etc.

You’re probably like me though. Stuck on the other side of the gap, not having the luxury to A/B test bug fixes. Experimentation is expensive, you always wish you had more traffic, things break, statistics are shaky and it can take weeks if not months before you complete the idea to insight cycle.

If we’re in the same boat, you can’t disqualify yourself from doing upfront discovery work and say just test it to see if a random idea works. You’ll be too slow and burn through too many resources.

The other problem is that you get away with not knowing the problem behind the solution. You’ll find what works, and thats all that matters when A/B testing is misused like this. There’s no need to understand why something did / did not work if you’re only interested in seeing uplifts and pumping through the A/B test idea backlog as fast as you can. It perpetuates the cycle.

Don’t stop A/B testing though. It’s an amazing capability and can bring a ton of value if balanced and aligned with Discovery.

3. Ship & forget

So we’ve got a stream of solution-first ideas, of which some are just tested to see if they’ll stick.

That’s nice, but once shipped I found more often than not there’s hardly any time spent checking whether these features actually solved anything. That makes total sense because that was never the intention to begin with. Without problem understanding, there’s no idea about what it is that needs solving.

The success criteria was either: shipped on time, or when it was an A/B test: shipped on time with the statistical seal of approval. There is no real learning, because there never was a clearly defined problem to measure the effectivity of the solution against.

You’ll know whether the solution moved your outcome, but without knowing why, you’re missing out on a lot of useful feedback. Feedback that could’ve helped you replicate the win or avoid the loss next time.

Without that opportunity to tie it back to, solutions just become a list to work through as quickly as possible. This means we’re back at the 1st Solution-first thinking pattern. It’s incentivised to come up with more ideas as the system rewards pushing through them fast.

A symptom of this ship & forget issue is when A/B testing becomes a planning nuisance. You’re required to submit your topics for the next quarter while you’re still waiting for the results that shape your direction to come in. There’s such a strong emphasis on delivery that learning becomes an inconvenience.

I’m not saying to slow down; it’s nice to get the work done fast. It becomes a problem when getting the work done fast becomes a goal on its own and it starts hurting your effectivity.

The complete cycle, and breaking through doing better

1. Without problem understanding, you’ll get stuck in solution-first thinking. It’s hard to select the best idea when best for what is unclear.

2. You don’t have to decide though: just test everything. Unfortunately this is a luxury most orgs can’t really afford: it’s too slow and too expensive, but the real issue is that you falsely get away with not doing the discovery work.

3. By design, this makes the solutions a list to work through ASAP. There’s no time, need or incentive to check whether the thing you shipped solved anything. Done and on to the next. The cycle resets and requires more ideas.

Doing better with Discovery

Breaking through implies that this multi-faceted problem can be solved entirely which honestly, I haven’t seen done yet. It’s complex, context-dependant and sometimes really just not that big of a deal. It’s OK to just pump out features for a quarter sometimes. That’s just the messy reality in my experience.

Doing better is healthier and more realistic ask.You’ll find that the cycle runs much smoother with even a little bit of problem understanding that Discovery brings:

1. It becomes a whole lot easier to select the best idea, as they can now be ranked to how effective they appear to be in solving the identified problem, rather than ambiguity.

2. It not only increases the success rate of your experiments, it’s also not that big of a deal when an experiment doesn’t win. You’re now chipping away on a bigger problem and testing becomes more about learning instead of pumping velocity.

3. When you have a problem to solve instead of features to implement, you can’t really get away with just leaving it unsolved or not fully explored and just proceed to the next quarter.

Discovery doesn’t just find problems to solve; it also creates the accountability for teams to solve them. I.e. once you know the house is on fire you can’t really leave the flames unextinguished.

Why AI is only going to make this more urgent

If I try to look past the hype and try to understand what AI brings us in practice today and might bring us tomorrow, I only see an increased relevance of having problem understanding.

Today: working very efficiently on very irrelevant problems

Today, AI is already making a lot of things more accessible and easier for me. The whole idea-to-prototype cycle can now be done solo and in a matter of hours, instead of requiring help from a designer from the team (if available) and taking weeks.

With the rate things are going, I see no reason why it wouldn’t continue to make my life easier, take on more and more tasks (autonomously) and do them a lot more efficient than I could.

The emphasis here is on more efficient though, not more effective.

Source: Reddit

If you prompt nicely enough most LLMs will rave at whatever you throw at it. The shit on a stick is a silly meme example of course, but what is going on here applies to some extent to all your other interactions with AI.

It’s a powerful accelerator; the horse & carriage turned into a rocket. You still need to tell your rocket where to go though. You still need to provide proper input and judge yourself what from it’s generated output is right or not so right for you. If you don’t, you’re going to end up in the wrong direction very fast.

You’re going to work very efficiently on very irrelevant problems. If you’re already stuck in the cycle with solution-first thinking, AI will only accelerate its downward spiral.

Discovery is going to give you the fuel you need to make better use of AI’s acceleration effects. Deep problem understanding is going to be much more beneficial than yet another prompting guide. Because what’s far more important than how you formulate your question is deciding what question to ask in the first place. Shit in = shit out, shit in faster = shit out faster.

Tomorrow: discovery as the differentiator

We’ll solve the shit on a stick example eventually, though. Probably sooner than later.

LLMs will get better or this will be solved on a product / workflow level. If you’re going to use an AI landing page generator workflow, that workflow won’t let you get away with generating and using a terrible landing page that isn’t fully tailored to your needs. Quality guardrails will likely be built in.

So then what?

Source: Maggie Appleton

Maggie Appleton puts it nicely in her The Expanding Dark Forest and Generative AI article: higher standards, higher floors & ceilings. She listed this in the context of online content, but I think the principle applies here as well.

AI will help those currently stuck at terrible create something mediocre. As AI becomes more widely adopted, more people will be able to create something mediocre: higher floors. As standards of what we think is a nice experience rise, it’ll be harder to create something exceptional: higher ceilings.

Eventually, getting it done (fast) won’t be a guaranteed way to win. It’ll be hard to out-execute your competitor if everyone has equal execution power through AI.

The thing thats going to separate you from someone else who also has Loveable / whatever future AI workflow thats popular open in a separate tab, is how you’ll use it. Deep problem understanding and knowing what questions to ask (the input), will be the differentiator.

Tomorrow of course. Today we’re vibe coding shit on a stick.

Do the work

If you recognise anything from the cycle, it’s very likely you could benefit from doing more Product Discovery. The time to start is now.

With what AI is doing today and might do tomorrow, you’re either going to accelerate the cycle’s downward spiral, or you’ll break it, enable yourself to properly benefit from AI’s acceleration effects and build something exceptional.

The why is probably clear, if it wasn’t already. What might not be as clear is how. If you want to break the cycle, where do you start when you close this screen? What will you do?

There are no shortcuts to great things, and this is no exception. I don’t see any other way other than to do the work. You’ll have to learn and understand the ins and outs of Product Discovery.

That’s what I’ll help you with. First up: The only 5 Product Discovery frameworks you need to start delivering value