How I'm Beta Testing Postulate: Hypotheses, Metrics, and Personas

A week ago, I wrote a huge Twitter thread about Postulate.

It wasn't an announcement or reveal in any way; most of the ideas in that thread I had written up and shared elsewhere before. But there's something about a Twitter thread that lends legitimacy and catches people's attention like no other medium in the tech world (maybe the staking of my entire primary digital identity on my words, as opposed to a one-off blog post). Posting that thread marked the turning point when Postulate went from a random side project in others' eyes, to a startup.

At the end of it, I wrote: "If you're interested, follow @PostulateApp or DM me to beta test 😉". Waitlist signups skyrocketed from a dozen to 50 the day after and now 71, which isn't that many people, but keep in mind that I haven't made any other effort to get signups. I received a bunch of requests to beta test, and three people scheduled onboardings through my Calendly in the first week.

The thing is, I had no idea what beta testing actually entailed. I saw Dive offer early access and get hype, as well as Typedream and a bunch of other software startups; with the Postulate MVP getting in shape, it seemed natural that I open up beta testing too, if only as a forcing function to continue building out the MVP at fast pace. When real onboardings were being booked, though, I knew I would have to be far more intentional, so I consulted Ascend founder Hannah Le and TKS director Nadeem Nathoo for advice. From their experiences and resources they shared, I put together a framework for what I want to get out of onboarding beta testers to Postulate, and how I would go about doing it.

Beta test to test hypotheses

The point of beta testing is to validate specific hypotheses. The overarching hypothesis is that your product solves the pain point it set out to, and you can achieve product-market fit. Validating this hypothesis is any startup's primary and more or less only goal at early MVP stages.

Such a broad hypothesis is impossible to directly test, however. Hypotheses must be clearly measurable for you to know whether to validate or reject them. Thus, the product-market fit hypothesis should be broken down into specific ways that users will use your product. For Postulate, for example, here are the three that I came up with:

1. Users will jot down their thoughts as snippets. (Metric: snippets/day)

2. Users will turn snippets into public posts. (Metric: posts/day and snippet conversion rate)

3. Users will share public posts elsewhere. (Metric: views on posts)

Collectively, if all three hypotheses prove to be true, then I've validated my intended overall user flow.

Validating whether such a flow really solves users' problems is less quantifiable. With regards to this goal, Hannah outlines the following approach: "Aim for making users emphatic about your product. This means: they naturally spread words of mouths, they use the product with high frequency, and the product enables them to do new things."

Nadeem offers another framework for setting good hypotheses: optimize for key ideas that reflect the success of your product. For an app like Postulate, the most relevant ones are engagement and growth. Are existing users spending time on the platform? Are they bringing new users on to the platform? If both of these conditions can be proven true, then we've validated the ability of our product to succeed. In the three hypotheses I listed above, 1 and 2 measure engagement, while 2 and 3 measure potential for growth, so we're right on track.

Define specific user personas

Results from tested hypotheses are only useful if they can be contextualized against types of users being targeted. This is simply the need to eliminate or be aware of cofounding variables in an experiment; without this contextualization, your results will be very hard to interpret, and often misleading.

Specifically, you want your beta testers to be your perfect customers. "What groups of users would resonate with the problem you’re solving the most?" Hannah puts it. If your test results are bad, you don't want to be guessing at whether it was because you didn't find the right users; you want to know that it's because of your product, and that you should devote effort to fixing aspects of it related to that test.

Nadeem further pushed me to define specific customer personas, advising that it's fine if not all beta testers are ideal customers, I just need to be aware of which testers aren't and consider their advice accordingly.

For Postulate, I realized that I had never even rigorously thought about specific user personas. Man, all those months working in PM and UX did not come through here 😪 I set about defining what kind of people Postulate would be useful towards, identifying the general pattern of those actively upskilling or learning, particularly those who are early in their careers. I picked out a couple of groups I had on my mind: TKS kids, young engineers/students in tech, and young founders.

Here are two specific personas (just real people who fit lol) that emerged from this:

Laura Gao, student: a high school sophomore and TKS Innovate, Laura's primary goals are learning and self-development. The pressure to make money, choose a career path, or even get into college hasn't fully set in yet, but Laura is ambitious and curious and pushes herself to continuously take in useful knowledge and develop mental models in the most effective way possible. In school, she takes standard high school classes; in TKS, she self-learns quantum computing. She publishes content on her blog and Medium, writes a monthly newsletter, and maintains a public presence on Twitter in order to create serendipitous connections and train her skills.

Ben Laufer, young founder: dropping out of Minerva before the start of his junior year, Ben runs a creator analytics startup called Gensight and a network of community houses called Edyfi. His energy is focused on developing the product/experience for each of these startups, and reaching out to potential customers and collaborators. He continuously collects external information about the creator space, co-living, and entrepreneurship in general, but learning isn't the primary intention; applying them to his work is. He also generates many of his own insights, sometimes as material for content marketing for Gensight, and other times as private notes to keep for future reference. He actively maintains and grows a public presence, primarily on Twitter with content in threads and linked Medium posts, in order to establish the legitimacy of Gensight and connect with others in the creator space.

Create the ideal experience

Lastly, Nadeem urges me to provide the best experience possible to beta testers. For the pilot of TKS, for example, tuition was priced extremely low, and Nadeem and Navid put in a huge amount of effort to help kids however possible. Students' positive experiences provided the validation needed to continue growing the program.

Going above and beyond in regards to user experience is another means of eliminating cofounding variables. You don't want to doubt if negative test results are because of sub-optimal customers, nor because of sub-optimal non-hypothesis-related experience delivered to ideal customers.

Concretely, this means establishing multiple touchpoints, and an easy way for users to share their experience/report any bugs or feature requests. It might mean creating extra delight, like reading and giving feedback on content produced, or promoting it online.

Conclusion

With all these considerations in mind, I'm really excited for Postulate to finally get out there, and work towards finding emphatic customers and product-market fit! It's a step that I've never taken before, so there's a ton of learning for me too along the way. Success here will take us one step closer to the vision of open-source knowledge 😁