See How Product Changes Affect Engagement Behavior with Impact Analysis

The best Product teams are obsessed with building experiences that delight and deliver value to their customers. Businesses that are smart about using quantitative and qualitative means to understand their customers’ needs build happy, loyal user bases and outperform their competition. In today’s digital age, it’s this kind of impact-driven product strategy that is the foundation of systematic growth.

Because of this, Product Managers are always trying to develop a deeper understanding of their users’ product journey, identify new opportunities to offer value, and continuously improve product engagement. Today, we’re spotlighting how Amplitude’s Impact Analysis helps teams do just this.

Untangling behavior is hard

We don’t need to tell you that understanding how and why customers behave a certain way is hard, even with a behavioral analytics solution in place. Behavior is complex and causality is notoriously difficult to prove.

Every week, our customers in the B2B, B2C, and media industries came to us with questions like:

• “How does our product’s new, faster search capability affect how often users share content with their colleagues?”
• “How does surfacing a new FAQ page affect how often users contact customer support?”
• “How does viewing the season preview of a new show affect how often customers watch video content?”

These PMs and Analysts have hypotheses about about user interactions within their product, especially those related to critical conversion and monetization behaviors. But they struggle to validate and measure the extent of these relationships in order to take action on that information.

That’s why we built Impact Analysis.

Impact Analysis is a unique Amplitude chart that helps teams understand how users engaging with a feature for the first time impacts how they behave in the product over time.

Our hope is that Impact Analysis helps people better understand the relationships between different user experiences by providing them with the means to: 1) identify opportunities to boost engagement and 2) evaluate and broadcast the impact of new or changed features on critical product behaviors.

How to create a new Impact Analysis chart in Amplitude.

Identifying opportunities for impact

Good PMs usually have some intuition about how behaviors are related in their product. They might want to test out their hypotheses immediately by jumping into a series of experiments—but this isn’t always easy or fast. Experiments still require planning, resources, and buy-in. Product teams can build conviction for their experiments more easily if they have data to back their hypotheses.

Impact Analysis gives PMs the ability to directionally validate hypotheses they have about how two user behaviors are related, before launching experiments to rigorously test their ideas.

Do it in Amplitude

Let’s say you are a product manager at AmpliTunes, a music and video streaming app. Since 80% of AmpliTunes’s revenue comes from in-content ads, your team’s current North Star is “number of songs or videos played.” This key engagement behavior is both a leading indicator of revenue and the value that users get from the product.

After conducting some user interviews, you notice that some people “favorite” content in order to revisit it at a later time. This makes you wonder if there might be a relationship between users favoriting content and playing content. Before setting up an experiment to test this, you use Impact Analysis to validate your hypothesis.

The chart below shows the average number of times users play content per week, before and after they favorite content for the first time.

Impact Analysis chart showing how users performing the Favorite Song or Video action for the first time in the last 12 weeks impacts the average number of times they perform the Play Song or Video action.

“Week 0” indicates the week users did the Favorite Song or Video action for the first time. Before Week 0, users play content an average of only three times a week. But after Week 0—that is, after users first favorite content—this jumps to an average of about eleven times a week.

This analysis backs your initial hypothesis with data—there could, in fact, be relationship between “favoriting” and playing content. This report could help you rally your team around your hypothesis and motivate them to experiment with new engagement strategies involving the “favoriting” feature.

(For full details on how to set up this report in Amplitude, see our documentation.)

Broadcasting product impact

Companies like Airbnb, Uber, and Netflix don’t have a magic silver bullet for growth—they build systematically, looping continuously through four building blocks of growth: generating hypotheses, experimentation, acting on their learnings, and broadcasting impact.

Impact Analysis helps teams validate hypotheses for improving user engagement and build a case for experimentation. Just as importantly, it helps close the learning loop by giving teams a powerful way to visualize the impact that the features they’ve rolled out has on other engagement behavior.

Do it in Amplitude

As an AmpliTunes PM, last quarter, you rolled out a feature that allows users to follow automatically generated playlists or those created by friends. You can broadcast the impact of this new feature on user engagement by using Impact Analysis.

The chart below shows the impact that the feature rollout had on two important behaviors.

Impact Analysis chart showing how users performing the Follow Playlist action for the first time in the last 12 weeks impacts the average number of times they perform the Play Song or Video action (blue) and the Search Song or Video action (pink).

The blue curve shows the average number of times users play a song or video per week before and after they follow a playlist for the first time. The pink curve shows the average number of times users search for a song or video before and after they follow a playlist for the first time.

The average number of times that users listened to content jumped from 7-8 times per week to around 12 times per week after they followed a playlist for the first time. Similarly, the average number of times that users searched for content jumped from around 6 times a week, to more than twice that number after they followed a playlist for the first time.

This analysis shows that the new Playlist feature impacts not only how often users play content (AmpliTunes’s North Star) but also how often they search for content.

Next steps

For teams focused on iterative product development, Impact Analysis is a valuable tool to use at both the start and end of the learning loop. Teams can validate hypotheses to better understand the effects between user behaviors; they can also see the impact of product updates on engagement behavior.

While Impact Analysis does not replace experimentation for determining causality, it’s a useful tool to get a quick pulse on whether or not you are building something that your customers value.

Want to see it in action? Here’s a quick demo that shows you how to build an Impact Analysis report in Amplitude. (And for more details, take a look at our documentation.)

Interested in trying out Impact Analysis for yourself? Explore our demo.