Big trucks and user research
In his book Antifragile, Nassim Talib writes about using data to solve problems:
“…people want more data to solve ‘problems’ …we have never had more data than we have now, yet have less predictability than ever. More data — such as paying attention to the eye colours of the people around when crossing the street can make you miss a big truck. When you cross the street, you remove data, anything but the big truck.”
There’s a great post by Leisa on the GDS User Research blog about significance – sample size and confidence: how to get your team to trust qualitative research – this is something I’ve thought about a lot. I’m regularly asked about whether research work on projects is statistically significant. Whether we should be relying on small sample sizes and qualitative insights.
Here’s what I think about significance
If I’m part of a team building a product, all that matters is what we do next. Do we see the big truck – the thing that should be obvious when we test what we’ve made with real users.
A few people have questioned my approach to only measuring what matters. How can we be certain about outcomes of design changes without a greater quantity of data, A-B testing or segmentation?
I agree with them. These things are important, just not important enough that we need them right now.
In most teams we’re still trying to find the big problems. In the early to medium lifecycle of a product, especially in the public sector, these are usually big, significant problems experienced by all, or most users. We might even be building the wrong thing.
When I was working at GDS we were lucky enough to have Steve Krug come and talk to us – this stuck with me:
“[qualitative research] won’t show you everything, [but] testing with a small group of people, will usually unearth the most significant problems quickly. Even with a small sample the most serious problems will emerge.”
User research needs to start with the most significant problems – the big trucks. Once we’ve dealt with these, products will benefit from more detailed analysis, segmentation, or larger sample sizes. This is more about refinement, and meeting the needs of more specialist groups of users – this type of focus can be important for increasing channel shift to new digital services and also meeting the needs of assisted digital users. Having fixed the big problems you should find you’ve already removed some of the barriers for users not as comfortable or unable to access digital services.
When you’re trying to build something that meets user needs, more data just means more noise. Solve the big problems first then worry about the details.
This is my blog where I’ve been writing for 18 years. You can follow all of my posts by subscribing to this RSS feed. You can also find me on Bluesky, less frequently now on X (formally Twitter), and on LinkedIn.