If you’ve read, or have even heard about, The Lean Startup, you’re probably familiar with the concept of validated learning. Validated learning is the process of trying out an initial idea and then measuring it against potential customers to validate the effect. In other words, working to figure out the right thing to build before you start building.
At Dropsource, we employ validated learning by running experiments to test unproven ideas or design directions. Unfortunately, experiments can be tricky to plan, so today I’m going to talk about a lightweight technique we’ve been trying out to help us plan our experiments, the experiment questionnaire.
The experiment questionnaire is a simple template that asks a series of questions forcing us to articulate why an experiment is worth running and how it will be run. It’s a lightweight approach, adapted from a technique found in the book, Lean Analytics, that anyone on the team can use to effectively plan an experiment and get buy-in from teammates. The questionnaire is broken into four sections: problem definition, hypothesis, experiment details, and results.
The problem definition section is the foundation of every experiment. As a startup, we need to spend our time on high impact work so this section forces us to answer the question, what will we get out of this experiment? By describing the problem itself, who is having the problem, what solving the problem will improve, and supporting all of that with real data, we can have an informed conversation about whether the experiment is worth the effort.
The hypothesis section is where we stake our claim. Like with the scientific method, we need to state what we believe will be the outcome of an experiment based on the limited data available. We then use our hypothesis to design a better experiment focused exclusively on what we want to learn.
The experiment details section is used to map out the steps required to run an effective experiment. Essentially, it’s a checklist to make sure we don’t miss anything during planning and then acts as a guide during execution. This section forces us to make critical, but often overlooked, decisions like who is responsible for what, what will be tested, how long the experiment will run, and what the success criteria will be. When filling this section out it’s important to be thorough so everyone is on the same page once we move from planning into building and testing. Be explicit, don’t rely on assumptions.
Once the first three sections of the experiment questionnaire are complete we move on to building the prototype, testing with users, and documenting results. We then go back to the questionnaire to see if the success criteria was met and mark the experiment as a success or failure in the results section. If the experiment was a success then we can confidently move forward or run another experiment to validate something else. If the experiment was a failure we must decide to abandon the idea or iterate and try again.
So far, the experiment questionnaire has helped us take a lightweight, yet structured, approach to planning our experiments and my hope is that it can do the same for you and your team. If interested, I included a download link below for the template we use at Dropsource. Feel free to use it as-is or adapt it to meet your needs, and let us know what works best for your team in the comments.
Good luck on your next experiment!
Download Experiment Questionnaire Template
Interested in learning more about testing? Here are some additional posts from the Dropsource team:
Eating Our Own Dog Food: Testing at Dropsource
Testing Support Resources
5 User Testing Challenges (and How We Overcame Them)
About The Author: Paul Berg
Paul is a UX designer by trade and currently heads the product team at Dropsource. He cofounded Dropsource out of college with Nate Frechette and has been living the startup life ever since.
More posts by Paul Berg