From Correlation to Causation: Improving Conversions by Embracing Self-Service Analytics

At Brainly, we love to learn. It comes with the territory—we a leading global education app, with 300 million monthly users. Our sites and apps allow students, parents, and teachers to ask and answer homework questions, and we encourage members of our online community to answer each other’s questions, too.

Behind the scenes, we are curious and data-driven, and believe in asking as many questions as possible to move our business forward. We use data to spur product growth, but understanding user events, interpreting analytics, and tying both to our bottom line can be tricky.

Driven by curiosity but hobbled by our data analytics tools

As Brainly’s Manager of Data Analytics, I help our teams understand trends, explore correlations, and establish causation. It was challenging to extract actionable insights from our data with our previous setup, as we were a Google Analytics shop. We would ask our data analysts to write queries in SQL, Python, and R, and then present their findings as charts and other visualisations that product managers and internal stakeholders could understand. We are a fast-paced environment, and the answers were coming too slowly. Our analysts were overloaded, and we often found ourselves waiting a week or two for the answer to a pressing question.

Extracting data from Google Analytics was a burden. Our funnels are elaborate, and we segment our users into various behavioural cohorts. While hand-coded queries yielded results (albeit slowly), Google Analytics’ UI wasn’t user-friendly and added a layer of complexity. We couldn’t visualise our data without struggling through a series of counterintuitive menus, and even then, our segmenting and visualisation options were extremely limited. We had a wealth of information but couldn’t dive into our data without jumping through hoops.

From 10 to 80 weekly users in a few months

We started to look at alternative analytics platforms. We invited several vendors to demo their products, but my boss always had Amplitude Analytics in mind, and the team didn’t disappoint. They showed us use cases we hadn’t considered and created dummy data to demonstrate how we could segment and visualise Brainly user data.

A user-friendly analytics tool removes the pressure from analytics teams and makes user data available to anyone who needs it.

Most importantly, these demos helped the company realise that our teams could use Amplitude’s self-service analytics tools to pull user data instead of asking our overburdened analysts to do it. We had found a collaborative tool that took some of the pressure off our analytics team and made user data available to anyone who needed it.

We launched Analytics in February 2022 with 10 active weekly users. By November, that number had swelled to 80 active weekly users and 140 registered users. We’ve seen the self-service element of the platform change how teams interact with data. Our various teams, including our product managers, use Analytics to extract funnel and conversion metrics without waiting for analysts. Our mobile engineers, who create user events, use it for implementation purposes, and our QA team employs the platform to verify events are working correctly in real time.

However, the biggest impact has been on our content creation.

Transforming our core content

The core of Brainly is our Community Q&A product. Learners log in and ask questions, and other learners can answer. Every response earns points, encouraging users to ask their own questions, unlock pages, and improve their community status. Our content team verifies these answers, rewarding correct ones with green checks. These expert-verified answers are then presented as instant answers, so users don’t have to search our database for a solution. We suspected that instant answers improved the user experience and led students to sign up for a free trial, but we didn’t have the numbers to confirm whether this was correlation or causation.

Thanks to Analytics, we determined that users who see more instant answers in their first seven days on the site or app are significantly more likely to sign up for a free trial than those who see none. The conversion rate was fantastic and much higher than we expected. This realisation made improving our verified answers and increasing our instant answer match rate our top content priorities. To do this, we built a database of instant answers that our content team has verified.

We now leverage artificial intelligence (AI) and machine learning (ML) to automatically verify answers that are liked or given a positive rating by a certain number of people because there’s a greater probability they are high-quality answers.

Supercharging A/B tests with Amplitude Experiment

We recently adopted Amplitude Experiment to simplify A/B testing and run concurrent experiments without exposing users to more than one testing scenario at a time. We were previously using Google Optimize to run these tests on our website and Firebase to test our app. We found a hack that pushed Google Optimize test data to Analytics, but there was no way to push Firebase data. Experiment lets us integrate testing and analytics within a single ecosystem, which makes it a no-brainer.

In the first month of using Experiment, we launched six experiments to test various hypotheses, one of which demonstrates the power of the product. After we began optimising and prioritising our verified answers, the percentage of instant answers in search results rose from 5% to 10%. Using Experiment, we confirmed that an increased number of users were seeing more than five instant answers and that conversion rates had risen correspondingly.

We hope to roll out the platform to everyone at Brainly by the end of 2022.

Powerful features and seamless integrations

The most used features within Analytics are the behavioural cohorts and segmentation charts. Our product managers, among others, create highly customised user segments to understand how different behavioural cohorts interact with new and existing features. Retention charts are equally popular, and we use them to see how often users return to create new content and whether they engage in weekly learning interactions. These include reading, answering or asking a question, and having a tutoring session. If somebody engages in at least one such action every week, it means they are having a positive user experience, which leads to cumulative learning.

I use the formulas in the segment section of Analytics to compare user segments. I also find data tables incredibly useful because I can look at specific metrics like conversions and assemble different pages, features, and users into visualisations that provide deeper insights into our funnels.

Analytics integrates seamlessly with other analytics tools, specifically Branch.io and Snowflake. We use Branch.io to track unlogged users who migrate from our website to our mobile app. It’s a big win because we can see what experiments they were exposed to on the web and determine what content and features best drive users to the app.

We use Snowflake for all our backend data, but the integration with Amplitude allows us to swap user events. For example, when User B comments on User A’s answer to a question, Analytics generates an event for User B’s comment—but it can’t generate an event that says User A received a comment because User B initiated the session. With Snowflake, we can flip the equation and create a passive event that indicates User A received a comment from User B, and feed that information back into Analytics. This action gives us a better picture of how User A’s behaviour changed after receiving a comment. It’s amazing how much more we can learn with this simple shift in perspective.

Our North Star and next steps

In just seven months, we grew our active user base to nearly 120 people and built an Amplitude community within Brainly. We have a Slack channel where Amplitude users can engage in real-time discussions about the platform and a Confluence page with how-to articles, tips and tricks, and other important links. We also created a group of ‘Ampliteers’, power users who serve as ambassadors and mentors to Brainly employees who are new to Amplitude. We always had a question-based culture, but now it’s democratised and not confined to our data analysts.

By segmenting users based on their activity instead of their demographic, you can create a more personalised experience.

Amplitude helped us find our North Star. Once we realised that instant answers drove conversions, we pivoted and adopted a new growth strategy based on the network effect. The more questions our users answer, the more people are drawn to our site. Increased user-generated content also improves our SEO ranking.

That’s not the only change. We are creating new behavioural cohorts in Amplitude to segment users by the type and frequency of actions they perform on our site. By segmenting them based on their activity instead of their demographic, we hope to create a more personalised experience that will guide them through their education this year and in the years to come. We want to establish a long-term educational journey instead of simply providing immediate individual help. It’s an exciting development, and I’m confident it will evolve into new features that further distinguish Brainly from other educational sites and apps.

Amplitude has transformed Brainly’s ability to leverage data. We have easier and faster access to user and event data, and we can visualise it in myriad ways that allow us to verify the information and generate actionable insights. Our people have the tools to ask pertinent questions that satisfy their curiosity and lead to better outcomes for the learners that depend on our platform.