What Happens When You Can’t Trust Your Data?

Editor’s note: this article was originally published on the Iteratively blog on March 4, 2019.

There is a massive trend towards companies investing in improving how they make data-informed decisions. Businesses spend hundreds of thousands of dollars on tools to enable their organization to self-service, yet sometimes the data that flows into these tools is not trustworthy. Similar to an aircraft, these tools provide you gauges to course correct your business, but if they are showing you the wrong information you’ll end up in a death spiral. This is a massive problem for companies that rely on understanding customer behavior for their long-term success.

Trust erodes over time

Every time consumers of this data encounter an integrity issue it erodes their trust and makes them less likely to use data to make decisions in the future. After a while, they eventually give up altogether and rely only on their intuition, which more often than not is wrong. Worse is when they use the data to make a business decision only to find out in retrospect that the data was inaccurate.

Companies sometimes try to solve this by spending analyst time cleaning up their data and normalizing it instead of empowering the analysts to do what they were hired for, which is to help generate business insights that lead to growth. Retroactively cleaning up your data only works when you know that you have a specific data integrity issue; your analysts can’t fix a problem if they don’t know about it. It’s better to clean up the data at the source and avoid unclean data from flowing into your data warehouse altogether.

The reason this problem exists is that the teams who are dependent upon this data and the ones responsible for capturing it operate in separate worlds. For some product teams, analytics can be an afterthought; it’s something that they know they should be doing but don’t commit the time required to make it part of their DNA. This is primarily because most organizations reward shipping over measuring what is shipped. High-performing organizations don’t hide behind output but instead focus on the outcomes that they are striving to achieve. The only way to do this is for teams to determine what metrics they want to improve, identify the events that are needed to measure that metric, and align their business to improve those metrics. For your organization to really embrace data, product analytics requires dedicated resources and needs to be thought of as a feature of your product, not something that is one and done.

The workflow for determining what events to capture, instrumenting them, and verifying that they are correct can be fraught with human error. For product analytics to be a P1 feature, there should be a well-defined process that removes the potential for human error and enables teams to define, track and verify their product analytics as part of the software development life cycle. For some teams there is no single source of truth for this information; it’s often spread across Confluence pages or Google Sheets and quickly becomes out of date. Worse: developers have to copy and paste this information or interpret what should be captured from a Jira ticket.

So, what can I do?

Luckily, Amplitude offers advanced data governance features to ensure that you can trust the data sent to your analytics platform. In addition to these features (or if you’re not using Amplitude yet) you can take these actions to help build confidence in your companies product analytics:

1. Tie incentives to hard metrics

• Assign metrics to teams and reward them for hitting them
• Give teams ownership on how to achieve results
• Make the metric visible to the organization.

2. Change the definition of done

• Don’t ship new features without a clear tracking plan
• Verify that the events are being tracked correctly
• Measure the outcome of work that is shipped

3. More data ≠ better data

• Data quality is more important than data volume
• Structure your events to answer business questions
• Establish a standard naming convention & company-wide taxonomy

We’re keen to hear any other tips you have to help teams build confidence in their product analytics. If you’re actively working on improving your product analytics, we hope you’ll join the Amplitude Community and share what you’ve learned. And sign up for a custom demo to discover Amplitude’s data governance features.