Making Safer Bets by Optimising Your Product with Analytics

As product managers (PMs), we essentially place bets on the best path forward for our products. But it’s not enough to just throw all your money at one thing. PMs need to understand how to make the smartest bets in the first place, and then determine what worked and what didn’t to improve the odds for the next round.

At Jumbo Interactive, we tend to think about product strategy in terms of offensive and defensive bets. Jumbo develops solutions to enable the world’s best lottery experiences, and I’ve been with Jumbo for a little over a year now as a PM in the growth space. On the growth side of the business, almost all our bets will be offensive, so I explore, add, or optimise our growth loops to get them spinning better and faster.

PMs need to understand how to make smart bets, and then determine what worked and what didn’t to improve the odds for the next round.

One of our products is Lotto Party, which is a way for friends to pool their money together for a better chance at the big jackpot. Lotto Party keeps track of who contributed what for a particular draw. That way, if a party wins big, everyone is notified and automatically gets their slice of the pie.

Jumbo’s approach to growth is to go after the viral growth loops provided by products like Lotto Party. It’s important to grow not just linearly, but exponentially. Lotto Party provides compounding growth: someone invites their friends to take part, and then a subset of those friends will create their own lotto parties and invite more friends.

Jumbo’s growth engine includes several of these viral growth loops, which are then complemented by paid ad loops: The paid ads generate leads, which generate capital, which we then spend on finding more leads. Paid and viral—the two work in tandem.

To confidently make these bets, drive discovery and measure our success, we rely on Amplitude. We’ve seen a 40% increase in monthly active users in Lotto Party over the past year, a lot of which can be attributed to the insights we’ve gained from Amplitude Analytics, Amplitude Experiment, and Amplitude Audiences.

Analytics ensures everyone is on the same page

The main thing we do with Analytics is measuring our leading and lagging indicators. So one key metric we monitor for Lotto Party is how many groups have been created, because a new group signifies both the end of a loop and a start of a new one. Was the person who started the new group a recent party joiner or not? That helps us understand the companion growth mechanism.

But it’s not just me looking at this information in Analytics: it’s company-wide. Every engineering team I work with has access to my dashboards and charts, and I encourage them to look at them often. I manage their team spaces, and I create a Notebook for every bet we place. I’ll throw all the analytics into that notebook for the engineering team to review.

I also make short Loom videos for a specific dashboard or a chart in Analytics to succinctly explain what can otherwise be complicated information. I share these with everyone, including engineers, data analysts, other PMs, my bosses, and my boss’ bosses. Instead of trying to schedule eight different meetings, I can share a four-minute video of the analytics with everyone. Everyone can see what I’m doing, the results of that work, and it helps make our gut assumptions more accurate about how users actually use the product.

Experiment tells us which investments are worth going all-in on

What we see in Analytics naturally leads to the A/B testing in Experiment. These tests allow us to create hypothesis statements, determine whether they are true or false, and then know where to go all-in (or where to pull back). Experiment increases our confidence in placing a particular bet because it gives us a clear risk-benefit analysis.

We start by creating an experiment design doc where we explain the problem and supply as much supporting evidence as possible on the problem and the possible opportunity. We then write a hypothesis and call out the primary metric, which we generally describe as a trade-off metric. In most experiments, we also list a few secondary metrics. That info will determine what success or failure looks like, as well as the sample size we need for statistical significance, which then determines the experiment duration.

The experiment design doc gives us strong definitions of what we’re testing in Experiment. It also helps us prioritize experiments. A three-month-long experiment is pretty slow. If we know that an experiment’s going to take that long, it’s better to get a couple of week-long experiments out of the way before we start on the longer one.

Experiment design matters, because we want to build everything quick, dirty, and cheap. We want to get something out as soon as possible so we can collect more data, and making as small an initial investment as possible allows us to test extensively and build smartly.

Making as small an initial investment as possible allows product teams to test extensively and build smartly.

One example from Lotto Party was the lack of discoverability around the product. The majority of users on a platform are in what we call our “play” experience, and Lotto Party was fairly divorced from that experience. If we brought Lotto Party more into the play experience, what impact would that have? That move would’ve required a significant lift from engineering and a significant investment as a result.

Our experiment mindset brings everyone—product, engineering, Q&A, whoever’s involved in the process—to think about streamlining this experiment to gain the greatest insights from the smallest investment. In Experiment, we saw that incorporating Lotto Play into the play experience almost doubled the sales for Lotto Party, and more than doubled the number of groups created. Investing further in this was a no-brainer.

Audiences increases transaction value with more personalised carts

We use Audiences to make smarter cart recommendations, which we include in email outreach and also the purchase flow. Whether it’s something the user has bought in the past or is about to buy, we offer additional products they might like, or a suggestion such as, “Replay this ticket you’ve played before.” It’s a smarter twist on that classic question, “Do you want fries with that?” Audiences knows this particular customer doesn’t want fries—but they do like milkshakes. It would be pretty hard to build our own algorithmic framework to figure out the user’s desires, but Amplitude supplies it for us.

The more personalised the recommendation, the more likely the user is to select it.

The recommendations in Audiences, which are powered by Amazon Personalize, easily pay for themselves. The more personalised the recommendation, the more likely the user is to select it. Jumbo knows this because we had cart placements before we started using Audiences, and those placements have performed much better since.

About eight months ago, I had an idea about the products in our cart placements. Previously, the cart placements focused on a subset of products related to charity raffles, but I wanted to expand it to all our products and let Audiences decide what’s best for each customer. Lo and behold, we saw an uplift in clicks and transaction values. With Audiences, the more options you provide, the more people you’re going to hit.

Retain existing users and make smart bets to win new ones

That personalisation is also significant to Amplitude’s integration with Braze, which we use for messaging. If you think about our growth engine, we have all these acquisition loops generating customers, but the other side of it—the core of our engine—is retention and engagement, and that’s where Braze sits. Being smarter about knowing when to send a message to a user makes all the difference, and it’s just as important to know the right channel. If the person uses a particular app frequently, it probably makes more sense to send them a push notification or an in-app message than an email.

Braze is there to help us continue habit loops. We define a habit loop as a customer being an active player for the two months after activation. If they play for two months, they’ve become a habitual player. We use Braze to trigger those habit loops and keep them spinning to drive our retention. But to do this, we need to have the best data flowing into Braze. Analytics helps us do that by making smarter cohorts and personalizing the content. The more specific we can be about this content, its timing and delivery, the better the results of those campaigns.

I’ve learned a lot from working at Jumbo, and I have a whole new perspective on growth. Working with Amplitude has also been a huge win. We wouldn’t have seen the increase in monthly active users for Lotto Party without Amplitude. The platform enables us to place the smartest bets.