Getting User and Customer Feedback: How Many Users Should We Talk to?

Most of us working on product development know the importance of getting user feedback, but how can we effectively drive actions based on user feedback? That’s the question.

One of the key questions I heard people asking is: How many users should we talk to?

The answer is, of course, it depends.

To Improve Usability, Interview as Few as Five Users Would Do

Based on Jacob Nielsen’s pioneering work in the field of usability testing, we don’t need to interview too many users. Interviewing 5-6 users gives us most of the possible usability issues that exist.

When I conducted usability studies, I’ve always heard non-usability people asking, how can we draw conclusions based on only a few users?

Well, a big part of the reason is  that whereas all human beings have different cultural, educational, and social background, we’re remarkably similar when comes to cognitive processes — memory, perception, learning, etc — mechanisms engaged when we try to figure out how to use a product. Therefore, one man’s meat actually is another man’s meat as well. For instance, the first generation of Windows 8 didn’t have a Start button, so that users can’t easily access programs, a feature that previous versions of Windows supported. If you interview just 5-6 users, I’m sure that this issue would surface, and you can stop there and declare that this issue is likely to bug many real-world users.

One thing to keep in mind is that whereas people are remarkably similar in their reaction to possible usability issues, prior experience as well as use cases do have an impact on how they use your product. So make sure that you only interview people that are your target users. That means, for instance, if you are designing an eCommerce site, make sure you talk to people with at least some online shopping experience. If you need to get feedback on a social news site, you’d better get people that are familiar with Reddit, Digg, Tumblr, etc.

To Understand User Preferences, Need to Listen to Many, Many Users

Then, people, particularly those trained in business schools, those that work in marketing/market research fields, and executives that need to make big business decisions based on consumer insight, will wonder: What about surveying a large group of people? Do we not need to listen to a larger group of users ever? 

Well, we do need to talk to more people — if we are studying their opinions, such as whether they find a product useful, how they would use the product to suit their needs, what background color of the screen they prefer, and so on and so forth.

The reason: Unlike using products to accomplish certain tasks, which engages basic cognitive processes, human preferences and opinions are heavily influenced by cultural and social factors and different people indeed are of different opinions — oftentimes forming stack contrast. For instance, many Chinese users love red and gold colors due to cultural influences — these colors are associated with good fortune. There’s no such association in the West.

That means, if you are working on something where preferences and opinions matter, such as choosing a logo, deciding a visual design scheme, and prioritizing product features, you do need to talk to more people, in the form of multiple interviews, focus groups, or large-scale surveys. And make sure that the people you talk to fall within the intended target user segments. I would say, at least interview 10 users for each of the user segment — e.g., young woman, stay-at-home moms, teenagers — if you’re able to get that many. You can also do surveys if your questions can be answered without probing and are relatively straightforward to answer. Read another post to learn when to use interviews and when to use surveys to get user feedback.

Finally, there’s an alternative of reaching out to a large group of users/customers — you can simply launch the product, conduct A/B tests, and learn about the metrics. Whereas this test-and-learn approach is becoming very popular due to the lean-startup movement, it does have its own drawback and we still need to do the traditional user/market research. Read my article on UX Magazine to learn why.

By Frank Guo

Google

Share

Leave a Reply

Your email address will not be published. Required fields are marked *