Once you’ve spent the time to find, recruit, and qualify beta testers, how do you motivate beta testers to provide feedback, and what should you do to keep them engaged and interested? Hint: it involves understanding their initial motivations for joining the beta and showing them that you’ve heard and understand their feedback.
Seems obvious, right? Yet it’s astounding just how often beta tests devolve and leave beta testers feeling used. Violate your beta testers’ trust and this cute limerick could become your reality:
There once was an irate beta tester
Whose feedback was just left to fester.
He was extremely perturbed,
Used some rather strong words,
And ended the rant with a singular gesture.
In truth, you’ve gotten off easy if the only repercussion is a singular gesture. More likely, you’ll end up with bad word of mouth, excessive churn, cursory feedback, sullen testers and passive-aggressive behavior.
5 Simple Steps to Keep Beta Testers Engaged
There are some simple steps you can take to ensure that your beta testers remain engaged throughout your beta program process. Treat them right and you’re likely to have a cohort that is eager to participate in future feedback sessions.
1. Have Reasonable Expectations & Recruit A LOT
This time it will be different, you tell yourself. This time, I’ve recruited the best, most motivated group of beta testers EVER. They won’t disappoint. Their feedback will be timely, precise, concise, actionable. I won’t need to coddle, remind, or badger them as the deadline for feedback approaches. This time it will be different, you say.
The beta starts and feedback rolls in. Enthusiastic, detailed feedback. Then, less. You coddle. You remind. You badger…
In general, you should expect that only a fifth of the beta testers will follow-through.
“Most beta testers will try out the program when they first get it, and then lose interest,” says Joel Spolsky. “[O]nly one out of five testers is really going to try the product and send you feedback.” He goes on to say that as a rule of thumb you should have 100 active beta testers per employee, and if the 1 in 5 ratio is to be believed, you should plan on recruiting upwards of 500 people in order to ensure 100 solid testers. “So, for example, if you have a QA department with 3 testers, you should approve 1500 beta applications to get 300 serious testers. Fewer than this and you won’t hear everything. More than this and you’ll be deluged with repeated feedback.” The takeaway: you’re going to need to recruit a lot of beta testers.
2. Understand What Motivates Beta Testers
You might be able to up the response rate from one in five by paying close attention to what motivates your particular group of beta testers.
There are 5 great reasons that people choose to become beta testers, according to Benny Luo:
1. To see something fixed
2. They love your product
3. They dislike of your competitor’s product
4. To learn something new
5. Because they love the community
Emily Hossellman largely agrees with that list and backs it up with data from a survey conducted across Centercode’s community of 80,000 testers. The top five reasons given by the more than 5,000 respondents to that survey:
1. Help Improve the Products I Use
2. A way to Learn the New Features
3. Early Access to Your Products
4. Grow Professionally
5. Earn Incentives
It’s worth noting that the #1 reason above was cited 43% of the time, more than twice as often as the #2 reason.
Hossellman’s analysis is spot on: “We found these results to be incredibly interesting. Less than 10% of respondents are involved in beta testing for the rewards we give our testers to thank them for their hard work. Over 40% joined our community to improve products they love or use. Another 37% thrive on the early access to new products and features.”
Once you understand why they chose to join your beta, tailor your approach accordingly. For example, those people who want early access might be further incented if they knew that top beta contributors would be invited to participate in future early-access, preview, or sneak-peak programs. Hossellman also points out, “If your testers are interested primarily in improving your product because they use it, you know they’ll be great candidates for testimonials and would love to be rewarded with a production version of your product at the end of the test.”
The goal here is not to manipulate your testers, but to understand their needs and design a beta program and system of incentives that is appropriately aligned. Kind of like what product managers do ALL THE TIME when creating great products.
Side note: This is one of the things that I love about being a product manager. You have the opportunity to work with some amazing people who are genuinely interested in the products you build. They care so much that they just want to help make products they love better. As a Product Manager, it’s heartening and can motivate you to do right by your users.
Side note on that side note: Some might view that ranking differently…those users might have no choice about which products they use (ahem, I’m looking at you, companies that standardize on subpar products), and they figure if they are stuck using it, they can at least try to improve the product so their own lives will be better….That’s too cynical an outlook, even for me.
I digress. Back to my original digression: The lesson here is that people are good, and given the chance, they are often more than willing to help.
3. Be a Gracious Beta Testing Host
Imagine that you invite some friends over for dinner. You do most of the cooking, and every so often you ask them to taste this or chop that. Even though you are doing most of the work, it’s a collaborative effort. Everyone contributes to the meal in their own way, and the outcome would have been different had your friends not been there. The food smells delicious and taste buds salivate in anticipation. Just as you are about to sit down to eat, you tell everyone that they need to pay you $10 for the meal.
You’d never do that to friends you invite for dinner. If you wouldn’t do it to beta tasters, don’t do it to beta testers.
Listen to Evan Hamilton, “You don’t need to pay your customers in cash for being part of a beta, as they have a vested interest in your success. But you should pay them by giving them the feature(s) they tested for free. Don’t make them work for free, tease a new feature in front of them, and then ask them to pay up. No matter what your CFO says.” According to a survey conducted by Betabound, 9% of the beta testers surveyed were “in it for the free goodies.”
No matter how you feel about the 1%, it’s ok to indulge the 9%.
4. Act On Beta Feedback
So far I’ve made the case that you should take the time to understand what motivates your beta testers and recommended that you treat them like family and not walking wallets. There is one thing that you have a responsibility to do if you are running a beta program. This ONE THING will make or break the success of your product and your current and future beta programs.
If you are asking for feedback, you need to be willing to act on it.
Tim Hampson, writing on Quora, laid it out, “Act on their feedback,” he says. “There is nothing more satisfying than recommending some new enhancement and seeing it implemented. Conversely, there is nothing more demoralizing than sending in feedback to a black hole and hearing nothing.” Hampson suggests that you “…make a point of praising all new ideas and clearly saying a) we will do this in X time frame or b) we won’t do this for the following reason…To bootstrap this, write a blog or otherwise highlight elements of the product that were the result of user feedback.”
You can even shorten the feedback loop considerably by implementing in-app messaging. In other words, rather than waiting to reply to “formal” requests, engage beta testers via in-app chat and address concerns in the here-and-now.
Regardless of the forum, if it’s a good suggestion, let them know. If it’s something you’ve heard before and decided not to address in the short term, say so. Any feedback, even feedback a user might not want to hear, is better than silence.
5. Offer Incentives to Beta Testers
Sometimes intrinsic motivators are not enough, and you need to resort to extrinsic methods.
There are plenty of ways to thank your beta testers for participating, including gift cards (Starbucks, Amazon, etc.), discounts on or access to your company’s other products, big ticket prizes, and personalized gifts (the magic of swag t-shirts!). In general, if you choose to compensate beta testers, keep the overall value of the incentive in line with the type of feedback they are providing. A couple of hours providing beta feedback in a focus group style setting may be worth some pizza and a beverage, while a months long commitment requiring frequent feedback may warrant a more substantial prize (an iDevice, for example) or accolade (“Beta Tester of the Year”.)
My two cents: while there are plenty of organizations which provide rewards, bounties and other extrinsic motivators, just because it is a common practice, does not make it a best (or required) practice. Use your best judgment.
Next up: From Beta Tester to Evangelist
The beauty of being attentive to your beta testers’ needs is that when the time comes, they’ll be more likely to reciprocate. When that product or feature is released, you’ll have given them good reasons to promote the new release (or new product) simply because they now have an increased stake in the outcome. In my next post, I’ll cover how you can encourage your group of beta testers to grow into a community of product evangelists.