How to Build an Evangelical Beta Testing Community

This is the third and final article in my beta testing series, which has included articles on How to Find Ideal Beta Testers and How to Motivate Beta Testers.


Have you ever wondered why some sites or products seem to simply explode onto the market? Some of the techniques that these companies use to seed and grow their userbase can be applied to your beta programs, helping you build a passionate, evangelical beta testing community that is willing to speak up when release time comes around. Let’s look at a few examples, and how the tactics that made them such a success can be applied to your beta program.

The User Acquisition Arc as Beta Launch Model

beta testing launch
First, let’s focus on a single foundational element: attracting and retaining users. When you are building a beta program, approach it the same as you would if building out and launching a community or userbase. Dave McClure popularized the pirate metrics as a way to define and metricize five plateaus in the user journey:

  1. Acquisition: How do you find users?
  2. Activation: Do users have a great first time experience?
  3. Retention: Do users come back?
  4. Referral: Do users tell others?

These same steps apply to the lifecycle of a beta program. As I previously wrote in the first two posts in this series, there are distinct steps you can take to acquire, activate and retain your beta testers. You can also increase the odds of a returning tester turning into a referring tester; it’s not a formulaic process, and requires deep, sustained effort to really pay off.

Download

What does it take to turn a returning beta tester into a referring beta tester?

Tweet This

Cautionary note: You might be wondering, “What happened to the fifth metric, the most important one for building a sustainable business: Revenue?” Well, a beta program is not a revenue stream. Don’t look to monetize beta participants — it sullies the feedback stream and can lead participants to feel (even more) used rather than objectively trusted. Still need convincing? Check out what I wrote about Beta Testers — or “tasters,” with an ‘a’ — in my previous post.

Use pirate metrics as a guide to build out your beta test community.

Tweet This

Case Study #1: eero: Why Beta Testing Is Fundamental for a Successful Launch

eero created a home wifi system which launched to much fanfare in 2016 and earned rave reviews. In this post on First Round Review, Kelly Neary and Paul Nangeroni discuss how important the beta program was to the overall success of launching a stupendous product: “As contained dry runs, these tests not only surface insights that help refine a product release before its launch, but they also lay the groundwork to engage early adopters and groom brand champions.” If you plan on running a beta at any point ever in your life, this article is a must-read since it explores just how much effort goes into running a successful beta.

From Beta Testers to Evangelists: Case Study #1 Learnings:

  • Declare frequency of outreach upfront. As part of the process of signing on to the beta, eero made a point of letting participants know how often they’d be in contact with each other. This had the dual benefits of further qualifying potential beta applicants who might shy away from a higher level of commitment, and also establishing a rhythm of communication. The latter was critical to maintaining engagement throughout the beta while also building rapport and trust between the people running the beta and the beta testers. Rapport and trust are important stepping stones to evangelism.
  • Seek process improvement, not just product improvement. Beta testers were expected to not only report on their experiences with the product, but also with the beta process itself. eero went so far as to institute NPS for the beta program itself and did what they could to work around the testers’ schedules to make it as convenient as possible for them to give feedback. By constantly seeking to improve the beta program, they made it clear to beta testers that the company was in it for the long haul, and wanted the beta testers to be, too.
  • Don’t close the loop; close their loop. eero would “reach out to the people who were expecting that fix or who had experienced that problem, letting them know that we were solving the bug. Open lines of communication are super helpful only if they’re relevant to the users who initiate them,” according to Neary. This level of responsiveness and white glove treatment of the beta testers was yet another visible expression to them of just how valuable they were.
  • Cycle in fresh perspectives. Especially if a beta program is running for a while (months, let’s say), it’s important to maintain a mix of beta testers, including some who may be newer to the product in the later testing stages. This helps to counteract fatigue and complacency, not only on the part of the beta testers, but also of those running the program. This benefits new and returning testers alike, because if forces the people running the program to remain engaged when onboarding the newer folks.
  • Pepper in postmortems. Because their beta program had such a long runway, the team held intermittent retrospectives and adjusted their approach to running the beta itself based on lessons learned. The incremental adjustments allowed them to be responsive to the needs of their beta testers and to any quirks they discovered in the overall beta program.
  • Give your beta testers a great product to test. eero set a high bar for the quality of the product even before they were ready to drop it on beta testers; “polished, not perfect.” This upped the odds that beta testers would have at least a good experience with the product. They did not rely on their beta testers to be the first line of QA. They also made sure the company as a whole was prepared to support the beta testers throughout the beta program. They respected their beta testers’ time enough to not force them to slog through a half-assed product release. I was not even a part of this beta, and I really appreciate that sentiment!
  • Find early beta testers by sourcing referrals from friends and family. “The prospective betas filled out an application that allowed eero to select a group that had very diverse representation of what it believed its true customer set would be.” They defined their key criteria up front to ensure they ended up with a diverse set of beta testers. The added benefit of using friends and family? You have to treat them well, or it will quickly come back to haunt you. You’d better be nice to my mom.
  • Incorporate Feedback During Beta. Take testers’ feedback seriously and incorporate it back into the product during the beta period. So, it goes without saying that you should allow enough time in your beta to incorporate the feedback. According to Nangeroni, “Give yourself a reasonable amount of time to collect feedback and funnel it back into the product prior to launch. Ideally, the beta should end when you’ve made the necessary adjustments to hit the level of performance, stability or speed that your metrics demand. Your beta customers — who are living the day-in-the-life with your hardware and software — need time to help you get that last mile.”
  • Feedback Loops > One-Time Feedback Events. I love this quote from the post: “A beta test is more ping pong than rocket launch.” What a visceral visual. The most insightful comments come from the ongoing back and forth with beta testers. This back and forth also reinforces the relationship, which increase the likelihood that those beta testers will….want….to….become….evangelists! (Ping Pong Rocket Launch is also the name of my next album. The first track is a ballad: Beta Evangelists.)
  • Know your beta testers. “First we reviewed each beta tester’s personal and home profile, regardless of who referred them. Then we clearly communicated the process and end goal, which we emphasized required their discretion,” says Neary.
  • Closure and more exposure. “If you’ve done your beta correctly, your early users are invested not only in your technology, but also with your brand on an emotional level. Says Nangeroni, “Part of success is developing a sense of community with your beta customers. It takes a lot of effort to reach out, but over time you start to develop rapport and relationships with these individuals. When you’ve established that trust, people are not only satisfied with what your product does, but excited for where it’s going.” (There it is again: “trust” and “rapport”.)

As First Round writes, “Nangeroni and Neary recommend closing the chapter with the beta and offering options for early users to stay engaged. ‘Start by sharing the milestone with them. Launch day was a defined achievement to celebrate together as well as another opportunity to check in. It’s very much like a graduation,’ says Neary.”

Case Study #2: Product Hunt: Why An Overnight Success Usually Isn’t

beta testing time

Ben Gelsey wrote a fantastic teardown analysis of the Product Hunt community and the steps co-founder Ryan Hoover took to build it out. It’s a long read, and well worth spending the time to digest. The subtitle sums it up: “An Overnight Success 1,834 Days in the Making.” He dissects Product Hunt’s approach as a way to answer “What does it take to successfully launch a community site?”  The short answer, as Ryan Hoover outlined in an early blog post:

  1. Target Influencers
  2. Indulge Early Adopters & Listen
  3. Make it Useful Without Users
  4. Create Exclusivity, Scarcity, Urgency
  5. Give Users Tools to Evangelize
  6. Seed Content & Communities

If one of the goals of your beta program is to convert some number of the testers into product ambassadors and build out an ongoing community, consider following this model.

From Beta Testers to Evangelists: Case Study #2 Learnings:

How does this apply to creating a passionate groups of beta testers? Points 1, 2, and 4 are all techniques you can use to attract, nurture and grow a community, while point 5 is a way to amplify your community’s comments. A bit more detail on each:

  • Target Influencers. Seek out beta testers who are a good fit for your beta test…yes, that’s a no-brainer. One additional criteria you may want to consider is the beta tester’s sphere of influence. When reviewing their profile, do you have a sense of just how active they might be on social media or whether their network includes the types of people you’re targeting? It’s these early people who can set the tone for the community, especially if you allow them to get involved in recruiting other people to the beta tests.
  • Indulge Early Adopters & Listen. When just starting out, nurture those beta testers who are superlative, recognize them, promote them, respond to their feedback. Let them know they are being heard. Jon Perino at Centercode agrees, and shares some concrete tips including showing gestures of gratitude, public accolades, and radically open communication.
  • Create Exclusivity, Scarcity, Urgency. Many will apply, only a few will be asked to join…or at least that’s the impression you want to create. Ask your top beta testers to invite their friends.
  • Give Users Tools to Evangelize. When all is said and done, make it easy for your most passionate testers (and users) to broadcast their love for your product and your company. (I hope it’s love, but lust and even friendship are good, too.) Consider including tools in the beta and release products which allow users to distribute content wherever and to whomever they desire. As Hoover wrote: “Make sharing easy, fun, and intrinsic.” This is an ideal place for you to leverage [Insert your favorite social media property here].

Building out a superlative evangelical beta testing community requires playing the long game. For example, the Ryan Hoover post referenced above was written nearly 2.5 years before Product Hunt’s earliest release, and evidence suggests Hoover was laying the seeds for launch that entire time. Short term thinking may get you immediate results, but not an ongoing relationship. Why go through the effort of creating a one-and-done beta program when you can build something with sustained longer term value?

Building out an evangelical beta testing community requires playing the long game.

Tweet This

Case Study #3: Kickstarter and Quirky: Engaging the Crowd Early and Often

Much has been said and written about Quirky (oh, sad, sad Quirky.). Much continues to be said about Kickstarter[a][b]. A brief primer: Crowdsourced product development, when done well, is a phenomenal way to engage potential users early on and to keep them involved throughout the process of defining, developing and launching a product. Quirky and Kickstarter are the best examples of the ethos of allowing the crowd to have a hand in all phases of bringing a product from nothing to reality. Both platforms encourage radical openness which in turn gives participants the option to follow its progress and keep some skin in the game throughout. By the end of the process, participants may feel that they helped birth this particular product, and like a proud parent, will want to sing its praises.

From Beta Testers to Evangelists: Case Study #3 Learnings

The techniques used by these two platforms echo many of the points above, and as someone running a beta program, you can learn valuable lessons from these businesses:

  • Engage the crowd early and provide consistent feedback. This is one ideal way to create products the market wants.
  • Enlist a group of committed beta testers who will willingly promote the final product. This point can not be emphasized often enough. The core group of committed beta testers is crucial. How do you find such a group? I’d suggest you refer so some of the previous posts in this series. (“Will Willingly” would be the name of my Jane Austen tribute band, if I had a band, or played an instrument, or had any musical talent whatsoever. Or even enjoyed reading Jane Austen novels.)
  • Embed native tools to both serve as echo chamber and social media amplifier. The echo chamber is undeniably effective at amping up sentiment related to a project or feature, yet it is critical that there are embedded tools to reach external audiences. The internal echo chamber is great during a beta period when the company may want to maintain confidentiality and generate pre-launch excitement.

To Sum Up

Ultimately, the quality, and evangelical fervor of your beta testers will be a direct reflection of how you run your beta program. Quality and transparency beget loyalty. The Golden Rule applies equally well to kindergarteners as it does to those of us who run beta programs: Treat others the way to want to be treated. If you respectfully listened to beta testers’  feedback, provided timely and accurate responses, incorporated their feedback back into the product when it made sense to do so, and ultimately let them know how much you appreciated their participation in the program, odds are pretty high that many of your beta program participants will become brand ambassadors and will sign up to beta test your next release.

About the Author

Steven Telio
Steven is a Product Management Consultant who specializes in defining and delivering stellar digital products. He has held senior level Product Management roles with a number of startups, including 4 which had successful exits. He has led projects in a variety of industries for organizations that include EMC, Pfizer, Eli Lilly, Syngenta, Boeing, NASA, and Harvard Medical School, and began his career doing technical support for a medical device start-up, where he answered “patient-on-the-table” service calls from neurosurgeons.

Follow on

Get our best stuff delivered directly to your inbox.

No thanks

Want to build swoon-worthy products?

At UserVoice, we help you turn customer feedback into a better product. We aggregate incoming product ideas and measure the business value of each idea so you can prioritize what to build next. Uncover your next big feature today!

Start Free Trial