Uncover Opportunities for Product Improvement Through Self-serve Data Analytics

At the heart of product management is an inherent curiosity and drive to answer questions. It’s not enough to see how a product performs and wonder why. A good product manager follows their curiosity, slicing and dicing the data in as many ways as possible to diagnose what’s happening.

Some PMs have that curiosity but don’t have the tools to take it further. Instead, they likely have to rely on someone like a data scientist or dedicated analytics team. It takes much longer to formulate hypotheses and answer questions this way, slowing down the product development process. Having the right tool to answer questions independently can make all the difference for conversion.

Long lead times and data dead ends

I know this from personal experience as a PM at used-car marketplace Shift. The used-car market is complex. It’s cyclical, so while the wind has largely been at our back during the pandemic, we face significant headwinds at other times. Buying any car is also a lengthy process. For many people, their car is the second largest purchase in their life, after their home. Buyers want to be confident they’re picking the right car for them, which takes time and consideration. The average car-buying journey takes three months, and there are many steps within that journey, from research and budgeting to zeroing in on specific models and the merits of individual cars.

Having the right tool to answer product data questions independently can make all the difference for conversion.

I stepped into a senior product manager role at Shift in 2020, where I’m responsible for growth. “Growth” can cover a lot. I focus on SEO and work closely with our marketing department to optimize advertising, which is important given our large inventory and advertising budget. We also run promotions at certain times, such as July 4th or during end-of-year sales, and I help ensure those promotions are successful.

When I arrived at Shift, the team was using Segment as our Customer Data Platform (CDP). We were also running Periscope Data, a business intelligence (BI) tool that runs on top of SQL queries. I could make it work because I know a bit of SQL, but it took a ton of time, and many of the people in the product organization didn’t have that same capability. Instead, they’d have to submit a ticket to create a chart and then submit subsequent tickets if they didn’t get what they needed. There was a long lead time to get those data answers, and even then, we hit a lot of dead ends because the data was incomplete.

Better and faster decisions across the organization

The lack of access to real-time data forced the product team to move slower, which is why Shift turned to Amplitude Analytics shortly before I arrived. Few people used the platform yet, but I could see the value. Self-service data would empower PMs, designers, and anyone else in the company to answer their own questions.

To put this in context: When I onboarded, there were just four PMs at Shift. We now have 16. That growth alone would have been impossible if everyone continued to send data requests through the analytics team. We needed to enable individual PMs to make, edit, and share charts.

Empowering product managers and designers to answer their questions allows them to quickly make critical decisions.

We started an ongoing initiative to grow Analytics usage at Shift. Every two weeks, I hold an open-invite walkthrough of the platform’s easy-to-share and easy-to-understand dashboards. I talk to anybody about the platform, whether they’re from product, design, user research, or anywhere in between. I have an agenda for those sessions, but often I find the most engaging sessions are where people arrive with a specific question, such as, “How many people make it through steps three, four, and five of our loan application?” When I show people how to identify the relevant analytics events to address their needs, they immediately see the platform’s value.

Over time, I’ve seen people increasingly use and trust Analytics, and an improvement in data confidence in the larger team. Empowering PMs and designers to answer their questions allows them to make critical decisions for their areas much faster, building data-driven products and strategic road maps. We can see this in the solid YoY improvements in one of our key metrics, Visitor to Lead: the user journey from visiting our site to being actively interested in a car. Since popularizing Analytics at Shift, we’ve made many improvements to our shopping process, including creating hundreds of articles to help educate people as they buy. All of this resulted in a higher Visitor to Lead metric.

How we use Amplitude features to improve our product

Our PMs leverage a lot of features within the platform to improve our product, including:

Amplitude Experiment: I think of this as the second level of being data-driven. The first level is simply understanding what is going on. This second level we’ve unlocked with Experiment is seeing the impact of changes based on A/B testing. We previously ran A/B tests using our homegrown solution, but it still required our data scientists to spend time writing code to create the test dashboard in Periscope Data.

Moving testing to Experiment means we can build dashboards, start and stop an experiment, and answer our questions ourselves. One example is a test we ran on a potential new feature called Car Comparison. Car Comparison allows users to select multiple cars and compare their attributes, price, and details like how many accidents each has had. It also shows photos of these cars side by side. Comparing apples to apples helps users make purchasing decisions quicker and more confidently. This feature was launched with the help of Experiment. Finally, the Product Manager could control the rollout and split testing themselves (without needing Engineering or Data Science). This dramatically improved the cycle time from launching to learning. After a few weeks, we saw a statistically significant improvement in key metrics and immediately updated Experiment to launch the feature to 100% of car shoppers. Car Comparison was a great success in A/B testing, and now that we’ve rolled it out, it’s been a major contributor to our Visitor to Lead metric.

Segmentation and Cohorts: We often use segmentation to create cohorts of users and diagnose issues beneath the surface of our data. For example: As more people become comfortable shopping on a mobile device, we see many users visit our site for the first time on a mobile device, then switch to a desktop later to complete financial forms. So we will create a cohort to track that two-device behavior.

Another thing we have seen with inflation and supply chain issues, there has been an increased interest in the used car market. As one of the leading used-car marketplaces, this has meant a huge increase in bots crawling our site and gleaning data on our vehicles for market research. The bots initially caused a lot of concern because they created huge spikes on specific page types, like our vehicle detail pages. But now, we have created a cohort that allows us to identify and filter out these bots from our user data. We also use cohorts to segment users by marketing channel.

User Lookup: User Lookup is a fantastic diagnostic tool. We might have a question like, “Does an event exist for when a user clicks next on the image carousel?” In that case, I find my anonymous user ID in Amplitude, click through a carousel in my browser, and then see which events fired.

User Lookup can help understand how to build a funnel, too. If I want to understand the flow of a certain user behavior, I will dive into the event stream to see the important events leading up to the conversion in question. Using this tool shows us what the journey looks like for an individual customer and the path they took to purchase a car from us.

Becoming data-driven empowers everyone to uncover the opportunities within a product.

Funnels: Funnels are critical for us because the car-buying process is long and involves so many steps. We attract users at various stages of the car-buying process—some people come to us at the very beginning of their journey when they’re still figuring out which type of car is right for them. They might be in conversations with their partner about the purchase or determining whether they need a car at all. Other customers arrive on our site having already done their research, knowing precisely the year, make, and model they want.

Funnels help the product team break the complex car-buying user journey into stages, with specific goals indicating the customer is getting closer to purchasing. So our first funnel might be signing up to our website. The next might involve favoriting multiple cars or adding a saved search. Through Analytics, we’ve identified critical events in the buyer journey. For example, we have seen that when customers click to look at a CARFAX report, they’re much likelier to purchase a car.

Identifying opportunities worth pursuing

In some organizations, the highest-paid person in the room decides where the product goes next. They have a hunch the homepage should be blue, so the homepage becomes blue. No one asks concrete questions because it’s hard to get concrete answers. But you shouldn’t make decisions based on anecdotal evidence alone.

Becoming more data-driven empowers everyone at Shift to uncover the opportunities that exist within our product. When we see something unexpected in Analytics, we can all dig deeper to see if that moment is an opportunity in the making and even test to find out the implications of pursuing it. That makes for better decision-making for everyone. It’s easy to get lost in big numbers and metrics, but Analytics provides great quantitative data that we can pair with qualitative data and work with the user research team to decide which opportunities are worth pursuing—and which ones are false starts or not worth our while.

Amplitude helps Shift PMs focus our efforts on the pieces that matter, that will drive business impact, and help customers get more comfortable with buying used cars.