When it comes to the all-important issue, listening to user/customer feedback, the first thing that comes to mind for many is focus group or survey. The tendency is so persistent that even when I conducted usability studies, the stakeholders, typically product managers and business owners, kept referring to the exercise as “focus group” or “survey”!
On the other hand, designers and programmers trust usability testing much more and don’t see or understand value of focus groups and surveys.
So, what are the differences between the three and when should we apply each technique?
Thanks to Jacob Nielsen and Donald Norman’s pioneering work, usability testing is now a firmly established technique deployed by almost every tech company in developing usable products. The basic idea here is that if you want to develop products that people can easily use to accomplish intended tasks, you must conduct usability testing, in which people perform tasks and are interviewed around their experience — errors made, the ability to recover from errors, perceived ease of use, alternative ways to accomplish the same task, how user recover from mistakes, the effect of learning, etc. It’s a tried-and-true method for designing user friendly product user interface. You can read Nielsen’s classic book on this topic, Designing Usability for more information.
On the other hand, due to limited sample size — you typically interview anywhere between 4-10 users when conducting a study — and a focus mainly on improving ease of use, usability study needs to be tweaked — for example interviewing more users, combining task-based interview with open-ended questions — to answer questions beyond usability.
Most important of all, we need to realize usability is but one aspect of overall user experience — read about other aspects of user experience to learn more — therefore combining usability evaluation with other types of customer feedback techniques is essential for improving overall user experience.
For example, as add-on questions to a typical usability study, you can ask users about their personal and professional background, how a day in their lives is like, how they do shopping, socialize, use mobile devices, how they make decisions, what’s their workflows are like in relation to certain activities, and so on, in order to answer broader questions underlying your business decisions.
Re-purposed as such, these usability studies are often called “user research” among user experience professionals and are used to answer all types of business questions.
Through focus groups we interview a bunch of users, typically 6 a time, in a group setting to gather feedback. It’s often used in by political scientists and pollsters in predicting poll results (e.g., presidential elections) and by market researchers to assess products’ market viability. A typical scenario is, when developing a new product concept, we conduct focus groups to understand if the idea would stick, what features should be included, what customer pain points the product is expected to address, so on and so forth.
Whereas focus group is a quick way to talk to a large number of users relative to one-on-one interviews (e.g., usability studies are all based on one-on-one interviews), it does have issues:
1. The feedback you get is not firmly based on actual user experience. When you interview users in a focus group, they respond to your questions based on their past experience of using similar products and imagined ways of using your product (if it’s a new product), not based on actual experience with the product, because it’s very hard to have a bunch of people play with your product and give you in-depth feedback in a group setting. So you can get a general sense of their opinions of a product and how they might use it, but as you could probably imagine, without interacting with a product, such feedback lacks needed depth and details to drive product decisions.
2. Group thinking can bias feedback you gather from focus groups. Social psychologists have long since established that group thinking is a powerful force that influences what we say and do. Focus group attendees are often subject to such influence, as they say things consistent with what the majority of the group or a particularly influential person within the group say. Experienced moderators can mitigate, but can’t completely eliminate, the group thinking bias inherent in focus groups.
Mini Focus Group Could be a Better Alternative to Focus Group
I’ve applied mini focus groups, consisting of 2-3 users per group, to address the issues of conventional focus groups. In a mini focus group, you can largely avoid group thinking as the group is so small and there’s no dominating social influence at work. And you can ask each participant to use the product and observe their behavior individually, before you interview them as a whole group, something hard to do when you have a larger group. This technique is a hybrid approach of usability testing and focus group, and has proven effective in answering a mix of usability- and market-research-based questions.
Survey is perhaps the most well known customer feedback collection technique among the general public. It’s a great way to gather feedback from a large group of people, to perform quantitative analysis which allows you to compare and benchmark customer opinions, and to have many questions answered with less personal involvement than if you interview users in person.
On the other hand, we shouldn’t overuse surveys due to these reasons:
1. It’s hard to get user feedback on a complex experience-related topic. Just like focus groups, surveys don’t lend themselves well, in comparison to usability studies, to having users actually interact with a product and give you feedback on that experience.
2. You might not know the right questions to ask. When you interview users you can change your questions based on their responses, but when you reach out to them through surveys, the questions are pre-determined and could be irrelevant to users. Of course you can ask open-ended questions such as “Do you have additional suggestions for improvement? Please specify”, but it’s still not the same as person-to-person interviews, in which you can probe much deeper through follow-up questions into issues emerging during the interviews.
To learn more about this topic:
Check out a complete list of customer feedback techniques based on the business questions you want to answer
By Frank Guo