How to Organise Work on an Effective Product Research

Remember the situation, when you’ve conducted research, spent many hours on this, delivered it to the team, everybody was excited, but nobody used this? Or when you wanted to create a great product, but don’t have much resources to search for respondents and conduct interviews?

We have talked to Misha Hananashvili, UX Lead at SberAutoTech about product interviews. Misha has great experience with product interviews, and we hope you’ll find some insights in this interview.

Here are major bullet points that we discussed: 

• why product managers always need analytics;
• how you can apply usability testing methods;
• finding respondents and talking to them.

— Misha, could you please tell me more about SberAutoTech? What are you doing there as a team?

SberAutoTech is a fresh team at SBER responsible for autopilot technologies. Exploring autopilot experiences is our continuous task. I’m personally especially excited to work on phygital technologies at the edge of physical and digital worlds where ergonomics directly influences a person’s in-car behavior.

— What’s your role in this team?

My first and foremost task is to support product managers with detailed analytics. Our tasks imply different extents of engagement and expertise. There are those that you can crank out like UX tests. This tool doesn’t require deep expertise and the skill can be developed quickly with practice.

— How do you interact with product managers regarding research?

First, you should consider any incoming request. Why make research? Will it impact anything? Who else is involved? Why is it so urgent? Answering these questions can help you set the right priorities. If you can assign a task to a researcher, this is one process, but if not, it’s completely different.

If there are no researchers in a team, a product manager should have the product analytics set up as it gives a clear idea of what is happening with users. A clear-cut interpretation is half of the success. The one drawback of product analytics is that it helps you understand “who” your segment is and “where” this segment fails.

Only qualitative research can help you understand “why” this happens. Product managers should backlog questions they can’t answer on their own and then design research and tests.

Of course, there are cases where you can rely on the first impression, hallway usability testing, or similar kinds of tools, but they are rather limited and don’t give you the full picture, let alone their relevance.

— What research methods do you usually employ?

Ethnography— that is, observing users in their “natural habitat”. There’s a thing called life experience (lx). Exploring a users’ life we dive into their context and background. We see where they experience difficulties, how they handle them, and then we resolve this issue with a feature or a product. I think this is what UX research leads us to.

UX tests and problem interviews are the best options if you already have a product. These are basic tools, but keep in mind that all businesses are different. When your product is related to the digital world, you need to research how they use your app.

In qualitative research, you don’t need to calculate anything. They are widely discussed, and the entry barriers are lower. However, you still can’t do well without quantitative methods.

The surrounding context greatly impacts our work.

Read also:

• 10 best customer engagement strategies
• Client onboarding: how to engage, retain, and win users
• Customer engagement strategy: a complete guide to winning prospects in 2023
• What customer engagement platform is best for your business? 15 options to choose.

There’s another method, RITE which means Rapid Iterative Testing and Evaluation. Take 2 people, solve their matching problems, then take another 2 and repeat the cycle. Do this for a week to get the quality outcome. The only thing is that a product designer engaged in the process is experienced well to adjust your product according to the strategy and the architecture.

User Experience is a wide and comprehensive notion. And it wasn’t me who said this first! The longer you work with UX, the more often you come across it.

The bigger the company, the more teams and even people there are responsible for their section of the user experience. The value of UX research is that it makes it clear for you that poor website interfaces are the reflection of poor processes within the company.

If we don’t consider the complete picture, and several teams, product managers, or designers can’t agree on the desired UX, this will impact it as a whole.

— Does it happen when research outcomes are unexpected?

Research outcomes can be pretty sudden.

How users fail in your product is not the most captivating thing. What you should be eager to know is how users hire your product compared to what you’ve laid in it. You may find incredible things in these universes, and that’s exciting!

— How does your team use research outcomes?

The sad thing about most research is that they don’t have publication prospects, and very few data are actually applied. We used the “UX debt” to handle this.

Decompose your research outcomes, see what you will potentially change, and find out what a product manager is going to do with the issue and when. Then, follow-up with them on the specified dates and see what they managed to do. This is actually a supervision method that helps you avoid raising new issues when resolving the existing ones.

Remember that it only works when you invite people to your research, and they watch the process live.

The issue report is better perceived and accepted if a customer and a designer are present because they remember these hellish minutes that users had during a test. 

We even had the rule: no designer and product manager, no research.

— Can one evaluate outcomes of research aimed at a certain stage of product development?

No.

All research is a contribution towards something new. You never know if you succeed and what insights you will get.

There are lots of information. You never know what info you will scoop with your research. You can track it over time as you implement research outcomes.

Let’s say we have a product supported by marketing activities. We have an influx of users but we can’t see what exactly influenced our user count: research outcomes or a marketing campaign.

Of course, we can find this out, but like in any other research, we should start with a clean slate. Suppose we don’t run any campaign for 2 months. We implemented research outcomes, so there’s nothing else influencing our audience. In each qualitative research, you can track metrics like UMUX-Lite, SUS, etc., but they alone won’t give you enough info; you should track them during every research.

Read also: how to find respondents for a survey or customer interview

— Has it ever been hard for you to find respondents for your research?

No. You may always find people and ask them. Let me give you an example. Dima (Research team lead at QIWI — Editor’s note) and his team did a great thing: they created chats with users directly in Qiwi where they sent questions. You may always find people willing to give feedback.

If finding and hiring respondents is hard and costly, you may use another cool solution at the end of each user journey. Show a small pop-up asking to rate smth on a scale from 1 to 5 and include a feedback form there. If you only have a feedback form, 2% of users will respond with a particular issue. On average, these feedback forms convert at a rate of 20-30%.

— How many respondents do you need for one research?

Let’s say we have a product, a set of hypotheses, and a user journey. If we know for sure who our target audience is, we should start hiring respondents with them. If not, we may address who we consider our target audience. If you already have a product, start with 10 respondents from one segment. As you approach the tenth respondent, you’ll see major features and growth areas of your interface. Move on to the next segment and hire 5 respondents, you’ll see what their differences from your major segment are. 

If you’re only developing a local product for a particular market, region, or country, it’s a completely different thing. You may find it more difficult as mentalities are different. People act the same in some cases but breaking into a new market without getting to know it is truly insane. That’s why you may need a lot of interviews.

— How do you tell if this is a relevant respondent?

After you’ve hired respondents, make sure they are not professional “research goers”. Such respondents won’t give you any fruitful details because they are not your users. You will see it clearly when you ask questions related to particular events associated with your product and get vague replies. 

Free quantitative research where we observe website visitors or get answers from our friends is one thing. But if we’re talking about platforms, we must acknowledge that they are very concerned with the quality of respondents, though it also doesn’t mean they will not have “research goers”. 

The point is that your sample is very important in quantitative research. Don’t consider these irrelevant guys when calculating your sample. There are usually a few of them and they may count as a sampling error.

Read also: The North Star Metric and How Can This Lead to Success

— How do you get ready for research and talk to respondents right?

Research requires much cognitive effort which is no less tiring than physical exercise. You should maintain uninterrupted focus, therefore don’t schedule over 4 tests or interviews daily.

As for preparation, you should learn a mantra that you will tell your newcomers. Respondents feel uncomfortable and uptight, so ice-breakers and the first contact are very important. Small talk mastery comes with practice: the more research you do, the easier it will be for you to achieve the desired outcome. 

We once tested sales assistants that sell SIM cards. One of our questions was “What commission do competitors charge?”. One lady said she was not going to tell anything, but in 20 minutes, she came clean easily. We only needed to get her to talk. So you should know the possible barriers and fears of your respondents. 

Start a conversation in an informal manner. Ask ice-breaking questions like “How was your journey? Did it take you long to find us?”. This is mostly an HR approach when you welcome a new person and talk to them.

All people are different, but they all should feel comfortable. If they don’t, it’ll be hard for you to obtain the information.

— I’d like to bring up another topic. What do you think about interface availability for physically challenged people?

I care a lot about inclusive usability. There are permanent limitations and there are occasional and temporary ones.

Adjusting the interface to physically challenged people makes the lives of all users easier.

This is the key idea. There are tools you can use to check the contrast of your interfaces. They also show what can be improved and how.

As for tests, disabled people can greatly help you with their ideas of how you can improve your product. This is especially important. If you want to know what issues these users may face, they will be happy to help you.

— How do you find these people? How do you interact with them with due respect and care?

There are lots of relevant communities on the Internet. Even if you post a question in the UX club on Facebook, you’ll either get applications from such users or they will guide you.

— In your opinion, do companies tend to care more about challenged people when it comes to product development?

Yeah, mostly large companies and corporations. They understand the value and start investing in it, though not so fast. I’m happy that new research teams come up that work on inclusion, and there is a demand for such specialists! This means more and more companies want to make their digital interfaces inclusive.

— Wrapping up our conversation, could you please give a relatable lifehack to product managers that don’t have the capacities yet, but want to develop great products very much?

Do some research rather than no research at all. 

Steve Jobs once said a great thing that I couldn’t disagree with. “You’ve got to start with the customer experience and work backward to the technology”. All insights are in the present, even undisclosed ones.