If you build web or mobile apps, chances are you’ve heard of user testing and the benefits it can have on your product. It helps you build empathy for your users, saves you valuable time by validating (or invalidating) the usability of a feature before building it, generates data that prevents you and your team from making biased decisions, and so much more. However, for all of these benefits you still hear about many product teams that struggle to make user testing a part of their process.
Well, not too long ago this is exactly the position we found ourselves in at Dropsource. We knew we needed to do more user testing but were struggling to do it consistently. Since coming to that realization, we’ve worked hard to make user testing a central part of how we build our product. After reflecting on how far we’ve come and what we’ve learned, I wanted to take some time to talk about a few of the challenges we faced and how we overcame them. Hopefully our experience will show you that every company faces their own unique challenges when it comes to user testing and give you the confidence to start tackling yours.
Challenge 1: Finding the right users to talk to
Challenge: As an early stage startup we were on the hunt for product-market fit. Naturally, we had many theories on who our users were but still had a lot of questions to answer before we knew if they were right or not. To answer those questions we sourced different types of people and tracked their interest, activity, and success in our product. This is a great way to test a diverse market to see who finds value in your product but unfortunately it also introduces some challenges in regards to user testing. When user testing you want to talk with the right users so the empathy you build and data you gather point you in the right direction. Without a clear picture of who our users were we feared testing our product on the wrong people, which ultimately held us back from testing our product all together.
How we overcame it: For us, the trick was to embrace the startup life and work with the users we had, not avoid them. The fear of talking to the wrong users was and still is a concern, but it became clear this was not a good reason to avoid user testing. Conducting a user test with any user will always be more insightful than not doing it at all. However, we didn’t just conduct user tests and act blindly upon the results. Being aware of our situation, we made sure to segment users and understand who we were testing. For example, when testing onboarding we looked at the differences between non-technical users and technical users, when testing our new app publishing features we looked at the differences between users who had already finished an app and users who hadn’t, and so on. This allowed us to have a more informed discussion around what different users needed and more confidently decide where to allocate our resources.
Challenge 2: Finding enough users to talk to
Challenge: It’s tough to find people who are willing to participate in user testing, especially early on when you don’t have thousands of active users. Because of this, we found ourselves going back to the same users over and over again to test new features. This is a problem for multiple reasons. One, you don’t want to test users so frequently they grow tired of participating, and two, you don’t want to inform product decisions with such a small sample size.
How we overcame it: The first thing we did was prioritize what we should test and what we shouldn’t. With limited resources it’s important to pick your battles. By choosing not to test more standard features like copy/paste or log in we were able to focus our limited testing resources on the unique and complex parts of our product like onboarding or app publishing without fatiguing our users. Second, we created a test participant database. This database logs users who are willing to participate in user testing, how many times they’ve been tested, when their last user test was, and other useful pieces of data. Currently, our database is still smaller than we’d prefer but it’s growing and over time is proving to be a great asset for our product team to properly manage test frequency and have a large pool of users to recruit from.
Challenge 3: Facing the pressure to deliver
Challenge: Who doesn’t know the pressure to hit deadlines, track KPIs, and close deals? I’d be surprised to find anyone on a product team who doesn’t, and Dropsource is no different. As a fast-moving startup we’ve faced a lot of pressures, many of them self-inflicted from extremely tight deadlines, rushing features into production for demos, pivots, and so on. So where was user testing supposed to fit into this chaos? Well, it didn’t and we found ourselves self-validating features just to keep the development train moving forward. Everyone felt like something was wrong but the pressure and forward momentum made it very hard to stop.
How we overcame it: After seeing some features create friction in our product and others that were not even getting used, we started to see the real world impact of our past decisions. Using this as a catalyst for change, we made it a company priority to properly set expectations for stakeholders and slow things down so we could start making time for user testing. The most impactful changes we made was organizing our roadmap to temporally focus on behind-the-scenes features so we had time to get our user testing efforts off the ground without holding back developers and then actually conducting lightweight user testing to build a backlog of validated features ready for development. All of this was in an effort to get our user testing efforts ahead of development so we had time for it without slowing down progress.
Challenge 4: Having false confidence when building for developers
Challenge: Building a software product for developers is a double edged sword. On one hand, the developers on our team were subject matter experts and had inherent empathy for our users. On the other hand, we felt too comfortable making decisions on behalf of our users thinking we knew what they wanted. This knowledge was, and still is, very helpful in translating the complexities of software development to a visual development platform but it does not replace the need to validate ideas through user testing.
How we overcame it: To overcome this we needed to show ourselves how different we actually were from our users. To do this we tried two things. First, we made the entire team responsible for customer support. Making everyone responsible for customer support opened a direct line of communication between our team and users, giving us an unfiltered view of who they were and what problems they were facing. Secondly, when we started conducting more user testing we asked developers to sit in as note takers so they could see the experience a user had with our product. Nothing drives home the need for user testing more than seeing someone struggle with something you originally thought was obvious and easy to use.
Challenge 5: Recruiting technical participants on a budget
Challenge: The fewer filters you need to apply when recruiting for user testing participants the easier they will be to find. Unfortunately, since our target market is more technical we had to recruit a very specialized type of participant. In other words, we wouldn’t be able to walk outside and pick 5 participants off the street. Because of this challenge we decided to look at 3rd party tools like Usertesting.com or to help recruit participants but the more specialized the audience the more we had to pay.
How we overcame it: Though it may be difficult, there is always a way to recruit participants for user testing… you just need to know where to look. For us, we tried many things but ultimately found that using the targeting features of Google and Facebook we were able to run effective campaigns to recruit participants. We also used Craigslist forums, StackOverflow, and other online communities where developers would visit. We then used a screener survey to further filter the people who clicked the ad to meet our specific needs. Even though it was a lot of work, we found this to be the most time and cost effective way to recruit highly specialized participants for user testing.
And there you have it, a few of the challenges we’ve faced at Dropsource on our road to making user testing a central part of our process. There are many challenges you will have to overcome to get your organization to a point where it can consistently conduct user testing, but based on our experience I’d say it’s well worth the effort.
If you’re one of those people who thinks their team should be doing more user testing but doesn’t know where to begin, start by identifying the challenges in your organization that are acting as barriers, then decide how you will overcome them.