Respondents define how your research will go. If you pick the wrong people or approach very few of them, you risk getting irrelevant outcomes. If you come to hasty conclusions, chances are that you’ll make a bad decision, lose time and money on a feature that no one wants.
You can’t just double check your research results. This won’t rule out mistakes. You need to learn to find respondents and develop a trusting relationship. That’s the only way to hear the truth, not just what you want to hear.
We’ve already shared our ways to find interview respondents and ways to invite them for an interview. Today, we’ll talk more about the preliminary step which is sampling.
We made a survey to find out the most frequent research challenges before even writing this article 🙂 Finding respondents was our top priority. Of all product teams that we surveyed, 61% experienced difficulties in it.
During interviews, we asked the guys about the greatest challenges they had. Most of the time, it was difficult for them to find respondents that would help with a particular task. We think the opposite way: you should look for people with a goal that you can meet.
The term speaks for itself. You may think that the more respondents, the better for surveys or product experiments. Not exactly.
You need to evaluate the universe before defining the required sample. How many people fit the parameters you set?
Let’s say the universe consists of male owners of IT companies. Derive a segment, for example, owners of B2B services, and use it as a sample for your research.
There are sample calculators, like this one for A/B tests or the one for representative samples.
They are based on statistical theory and tell whether your results will be significant.
Product managers should know the concept of confidence intervals and A/B testing rules. Let’s say you’re A/B testing a landing page. You acquired 500 visitors of which 3.8% converted, and when your traffic grew to 1000 visitors, the conversion rate became 3.2% which equaled your reference value. A good product manager always considers the sample. If your sample gets bigger, your initial outcome may vary
You can gradually increase the sample. For example, do research, release and improve the feature in several iterations, including MVP development, testing it on loyal users, and then expanding the sample depending on the feature objective.
When launching the new feature, we test it in several iterations on our users. First, we test internally on our teammates to collect feedback and improve the MVP. This usually takes one to two weeks. We use the feedback to improve UX, locate bugs, and fix them. Then, we roll the feature out for 20% of the relevant audience as part of the beta test. We add surveys inside the product to collect their feedback. After that, we release it for all users on our paid plans. Our approach to Enterprise accounts is a bit different. We test features more thoroughly and deliver the functionality at its best
You may think qualitative research is easier because you need fewer respondents. However, you can’t calculate exactly how many of them you need.
In qualitative usability testing, there’s the classical “rule of five” by Jakob Nielsen. In a nutshell, it says that five respondents find 85% of interface flaws. The rule usually works, but if you apply it everywhere, it may affect your outcomes.
If you rarely do qualitative research, engage at least 10 respondents each time. This sample will help you identify more problems per one research iteration.
You need more respondents for customer development interviews — sometimes 20, sometimes 40, or even more. Segment audience and find two or three people with different experiences in each segment.
There’s the “saturation” concept that works for qualitative research. Your sample should be enough both to identify as many experience and perception configurations as possible and not to be too big (it’s counterproductive). If your respondents start repeating what you already heard, you should probably stop looking for new ones.
It’s the user’s experience that helps you test hypotheses, not the research type. If you wonder why a user bounced during checkout, you won’t find your answers among those who ordered. They managed to handle your interface flaws. You need the ones who see your website for the first time to meet your objectives.
You need to decide on your ideal respondent before inviting them to participate. These two questions can help:
That’s how you define your selection criteria.
Dashly’s experience
We hypothesized we could grow Dashly via EdTech. One of our jobs is the lead qualification, so we wanted to know:
• what makes EdTech businesses certain that their new lead matches their idea of a perfect customer;
• what challenges companies faced during lead qualification;
• how we can improve our product to cure their pains the best way.
Then, we needed to talk to people that fit. We made up a list of respondent traits not to waste time on pointless insights. We needed the people that knew everything about lead generation and qualification.
Role: СМО, marketing director, marketing manager. Sales: over three people in a team.
Segment: EdTech.
What makes the business special: active lead qualification.
Run a screening survey to filter out unfit people. Add questions that will help you find appropriate respondents.
Dashly’s experience: These are qualification questions we used to determine if people fit our EdTech research or not:
For a qualification survey, five to ten questions are enough. If you can’t limit your survey to ten questions, add the progress sidebar to make it easier for respondents to complete it.
If a survey is your research method, separate qualification questions and only show the remaining questions if a respondent answered questions from this section in an appropriate way.
And one more thing: don’t ask for too much sensitive and contact data. Respondents are not very generous about them. If you can’t do without it, ask for it at the very end not to scare your respondents. Make sure your questions are worded properly.
Thanks! Now check your email. We’ve send your copy there
Experiment, test hypotheses, and enhance your product! We’ll be happy to accompany you along the way.