Through investigating user needs and behavior, user experience research is critical in guiding our product strategy and design. There are many techniques leveraged by user researchers. Over the many years of working in the field, I have seen a persisting tendency of choosing research methods based on techniques and not business objectives. For example, we often hear people saying things like “Why don’t we conduct an eyetracking study? It seems very exciting! For such a large re-design project, we can’t afford not doing it.” and “Let’s conduct a survey to learn about our client needs.” On the other hand, without a firm understanding of how the various research methods support the different types of business objectives, we would fail to drive business success through the methods.
Motivated by this observation, I’ll give a brief account of commonly used research methods as aligned to business objectives. Please note that this is not a detailed description of the methods and their applications, which will be discussed in my future posts.
Choosing Research Methods Based on Business Objectives
Establishing a foundational understanding of users
- Ethnographic research, a research method that allows us to perform exploratory interviews with users in their natural contexts of doing things, is suitable for establishing a foundational understanding of our users: who they are, what they do, how they carry out tasks that our products are intended to support, and so on.
Validating and prioritizing product features
- Survey is a great instrument for validating and prioritizing whether proposed product features align with user needs by asking them to rate/rank the features. In relative to qualitative research methods, the large sample size tapped by the survey method helps with communicating the results in getting executive buy-in.
- This method is not intended to support the creation of the product features in the first place, which can be better facilitated by qualitative user research methods such as ethnographic research coupled with business analysis – as a relatively close-ended research method, survey is great for supporting the validation but not the creation of ideas.
Understanding client satisfaction and attitude
- Survey might be the best method to understand overall client satisfaction, pain points, attitude, and preferences, elements that are better assessed through self-reporting than behavioral observation. It allows for the collection and analysis of a large sample size and therefore the findings are representative of the target users. The findings can be leveraged to inform the decision making of senior management, who typically resonate with quantitative insights more than with qualitative findings.
- On the other hand, in developing the survey questionnaire, we should heavily rely on previously gathered qualitative research findings to create research questions to be validated in the survey.
Evaluating and improving product concepts
- Qualitative interview methods such as contextual interviews and focus groups, in which participants are exposed to a product concept and a few usage scenarios, are suitable for getting user feedback on early-stage product ideas and concepts. Insights gathered in these studies provide in-depth, context-rich, and fresh insights around how the concept can be improved. On the other hand, these methods, qualitative in nature, should not be used as a success measurement of the concept.
Effecting user-centered design
- An iterative design-research process based on task-based user interviews and usability studies is the most effective way to deliver user-centered-design outputs. For example, one can follow this procedure: evaluating current website -> creating wireframes for the redesign -> evaluating the wireframes -> creating finalized design -> usability testing the final design
- Specialized techniques, such as RITE (Rapid Iterative Testing and Evaluation), can be leveraged to better support the iterative process.
Benchmarking design improvements
- Experiment is the method of choice to truly measure design improvements because it can isolate and pinpoint design elements one at a time. It comes with two flavors. For Benchmarking business results, we should use A/B test to experimentally compare live-product metrics (e.g., visits, sales generated, registered users). For Benchmarking user experience, we should use quantitative user evaluation methods such as survey, quantitative usability study, or eyetracking study, with experiment-method applied to the research planning, to collect user data (e.g., satisfaction rating, task completion rate).
- Longitudinal survey is an alternative that can measure design improvements by quantitatively comparing user feedback across different points in time that map to design-improvement efforts. However, the conclusions might be confounded with external factors – such as macro-economic factors – unrelated to product design and development.
Understanding competitive landscape
- Comparative survey that compares products across different companies can help us understand where the company stands relative to competitors in terms of client experience.
- Syndicated industry analysis (e.g., Forrester, Kasina, Corp Insight) provides comparisons of companies in the same industry based various evaluative criteria. This analysis offers insights around industry best practices, latest trends, unfilled gaps, and so on.
Collecting in-context user feedback
- Diary study lets users record their experience with using the product during a certain period of time in providing in-context feedback.
- Customer forums and customer reviews (e.g. Amazon.com product reviews) provide feedback platforms for users to communicate how they feel about the product. This gives us large-sample-size client feedback that reflects in-context usage experience.
- Exit survey collects user feedback right after they are done with each episode of using the product, and therefore reflects client experience with great accuracy.
Research Methods for Specialized Design Needs
- Card sort investigates how users expect the information to be structured and is a convenient tool for creating information architecture that speaks to user needs.
- Exploratory research methods such as ethnographic research and contextual interview will lend us a deep understanding of the user and serve as a robust foundation for persona development.
- Synthesis of multiple sources of client insights and metrics is key to developing personas that are both representative of target users and align with business priorities.
- Please read my published persona development paper for detailed discussions.
Content reading – web promos, search results, advertising
- Eyetracking captures users’ visual attention in a way unattainable through conventional user interview techniques such contextual inquiry and think-aloud protocol, and will help us create designs that better guide user attention. This technique is also very valuable for designing advertising, search results, and web promotions, things that critically depend on user attention.
- Please read my published online advertising paper for further details.
- Exploratory research methods such as ethnographic research and contextual interviews allow researchers to take a close look at users’ tasks and workflows in natural contexts. Such knowledge can drive the creation of an ideal UI workflow in support of user tasks.
Brainstorming design ideas
- Participatory design and specialized brainstorming focus group let users collectively generate design ideas in group settings and can help designers unlock innovative design solutions.
Customization is Key to User Research Planning
Above is a list of common user research methods and the business objectives that they respectively support. I also listed a number of methods that support specialized UI design needs. Whereas this list can be used as a rule of thumb for choosing research methods, in reality, research planning is much more complicated and method-customization is key to truly meeting business objectives.
For example, whereas card sort is traditionally viewed as the method of choice for figuring out information architecture, in some occasions we should avoid using it. Let’s say, we are designing a corporate site which is expected to represent things like products, about us, product support, jobs, education, and so on. Because this is a vast site and very few users actually have realistic use cases against all these modules and have the relevant knowledge and context, it’s very hard for users to intelligently arrange the cards that represent the proposed modules in a meaningful way.
Given that corporate sites are mainly driven by business needs such as how a company wants to project itself in front of the world, maybe the design team can start with interviewing senior management and then lay down a basic site design based on the business requirements, and then interview the different types of users such as clients, journalists, and job seekers via contextual interviews and get their reaction to the information architecture and navigation. This way, the design team will start with a basic framework first and then refine the designs based on user feedback. A stand-alone card sort is less useful than this iterative design-research approach.
By Frank Guo