Well that’s not quite true. Research has shown that you can get pretty much all the usability errors from talking to just 5 people up to an 85% confidence level, talk to 15 and you’re pretty much at 100% and literally every person you talk to after that is redundant. You won’t find anything new if you interview 100, or a 1,000 you’ll just find reinforced patterns.
Also there’s a statistical formula you can do to allow you to get up to 90–100% of errors with a small percentage of people.
People are so obsessed with big data being more accurate…to those people I say do you believe that Maslow Hierarchy is true? That projection of truth was based on only 18 students, 18 students at elite universities none of them diverse or non-white. Yet I can’t go a day without researchers spouting his name.
So in product testing you seriously only need up to 15 users to get all the usability insights you want. You’d do larger sample sizes to validate but statically speaking you actually don’t need huge sample sizes.
The jury is still out for me on whether quantitative studies yield insights that a skilled design researcher couldn’t find qualitatively. That’s my bias. However, both have their place. I think self-reported data is suspect (not without the scrutiny of a skilled interviewer), but behavioral data coupled with qualitative data is powerful and truly insightful not just for product testing but designing new products.
Sometimes I think the larger the sample size the less you’re actually measuring and the more bias your study because now your confirming hypothesis. But that’s just something I’ve observed.
The fact is both qualitative research and quantitative research are “borrowed,” methods for design. They and their rules come from other sciences — social and hard sciences and we use them to build confidence in our design decisions. But design doesn’t have the same goal as physics, chemistry, psychology or marketing.
Design is a future-oriented endeavor. We only research the past and present to anchor future decisions. We need to be mindful of that when we’re justifying our methodology to people who have different goals.
If we think about the purpose of design; what we really should be doing is less hypothesizing and more understanding so we build confidence not in our current design but what we should be designing, the innovative design that doesn’t exist.
In my opinion, we’ve been too focused on validation and testing and we need to do more exploration research. We’re far too focus on do people want this experience or how people perceive this experience or is this the right experience? When we could shift our paradigm to listening instead of asking for right experience.
Exploratory research creates hypothesis not validates them.
We have centuries of data on human nature from psychology, sociology and other social sciences as foundational to anything we do in design research. Qualitative research is really about pinpointing the right patterns of behavior and motivations triggered by an experience and determining if you want to design for that or against that experience or produce another one altogether.
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
