Validate Your Products: A/B Testing or User Experience Research?

There are two main approaches of validating your products to see if they perform as expected and identify areas for improvements. One is user experience research, which includes usability studies as well as other forms of user interview methods, gathering user feedback through asking user questions, surveys, and direct observation of user behavior.

The other is A/B testing, also called “launch and learn”, through which we randomly present different versions of the launched product to different users and observe the differences in the resulted Web, mobile, and business metrics.

Given that both approaches are talked a lot about, based on what I’ve seen among clients I helped, there’s much confusion around choosing between the two approaches: Do we need to use both methods? Can we just do A/B testing and forget about user research? If we need to do both, then when do we use each method?

 A/B Testing

Conducting A/B Test is a wonderful way to evaluate and improve your products — because it uses real market data on already launched products, the data collected are realistic, unlike that of user research, which collects user feedback in a lab setting and typically on unfinished products.

On the other hand, two main issues exist:

1. You cannot A/B test products still being developed. By the very nature of A/B testing, you need to launch a product before you can learn about its success or failure in the marketplace. When you’re coming up with a product concept, still designing its UI, and developing the final product, you can’t A/B test it.

2. You can learn “what” but not “why” in terms of issues. A/B tests will show you metrics of the different versions and help you choose the winning version, but it sheds little light on why one version is better than the other. Without directly interviewing users, you won’t know the whys and have little insight into how to further improve the winning version.

User Research

User research, on the contrary, is most useful in informing decisions when you’re still coming up with product concept and design, and when you need to find out why users like/dislike your products and inform specific actions needed to improve the product. Furthermore, user research can also be applied when you already launched the product — combining user research with A/B test can help you understand both the whats and whys of user reaction to your product.

You Must Combine Both Approaches

To develop successful products that users love and that make you money, you have to combine both approaches. At the early stages of product development, by definition you can only rely on user research to inform product features and design. When you’ve already launched the product, you can conduct both user research and A/B testing to paint a holistic picture of what’s going on with your product.

There’re no fixed formulas of combining them, so you should just keep this in mind: Use A/B tests to measure the final behavior and business outcome and help you choose a winning product formula, and use user research to predict real-life user behavior and pain points, uncover causes for concerns, and generate in-depth insight to inform product and UX design decisions.

To learn more about this topic:

To learn more about this topic, read an article on how to conduct lean UX research to complement A/B testing published on UX Magazine.

Here, I made no distinction between user research and market research, as the boundary between them is blurred and we can do better not to sweat the nomenclature–read about differences between market research and user research.

By Frank Guo



Leave a Reply

Your email address will not be published. Required fields are marked *