Mastering Identity Resolution in Experimentation

Many product and data pros understand that having deep insight into every user’s journey is a critical reason to invest in analytics. Creating this holistic customer profile requires effective identity resolution—which is also vital to experimentation.

Identity resolution (ID resolution) is the process of identifying and linking multiple pieces of information to a single individual, ensuring that different data points—like email addresses, phone numbers, or device IDs—are tied to the same person so you can understand their behavior.

In experimentation, identity resolution requires two key components: gathering accurate and reliable data and ensuring users have smooth and consistent experiments. In essence, identity resolution is the backbone of reliable experimentation.

The following explores ID resolution in experimentation so you can understand its impact on data integrity and user interaction quality—and why a single source of truth for analytics and experimentation helps you ensure you’re accurately and effectively executing this process.

Importance of identity resolution in experimentation

Identity resolution is essential in experimentation for two primary reasons: it empowers you with reliable user data and helps you ensure a consistent user experience—both key to making confident decisions.

Accurate and reliable data collection and analysis

In experimentation, your primary goal is to gather data that accurately reflects how each test variant influences user behavior and impacts your KPIs. However, suppose a user interacts with your product across multiple devices or under different identifiers. In that case, their data can be fragmented, and you might misinterpret their behaviors as belonging to different users rather than the same person.

This fragmentation leads to inaccurate data because it falsely represents the user’s complete journey or reaction to the variables in experiments like an A/B test. For instance, a user might see a new feature on their mobile device but make a purchase using their desktop. Without identity resolution, you might mistakenly attribute these actions to two separate users, skewing your experiment results.

Consistent user experience

Users visiting your website or app expect a seamless interaction regardless of how or where they access it. Without effective identity resolution, you could expose the same user to different variants, creating a disjointed and potentially confusing experience. This inconsistency can skew your data (as users react to different experiences) and negatively impact user satisfaction and trust in your product.

By effectively linking disparate IDs, software like Amplitude ensures that the data collected reflects each user’s interaction with the experiment, regardless of their device or if they’re engaging anonymously or logged into your product.

ID resolution challenges in traditional experimentation

Identity resolution can be challenging using traditional experimentation methods and disparate point solutions.

Inadequate linking of multiple IDs

Teams using disparate tools often struggle to accurately link a user’s multiple IDs across different devices and sessions. Stitching these IDs together using kludgy integrations and data pipelines is too challenging and expensive.

For example, a user might interact with your product using a smartphone, tablet, or laptop, potentially under different IDs, such as an email address, a device ID, or an anonymous user ID. Even when integrated, these disparate tools might treat these IDs as three separate entities, leading to fragmented data and a disjointed understanding of your user’s behavior. Moreover, you’d likely need engineering resources to build, maintain, and monitor these integrations on an ongoing basis.

Difficulty in maintaining consistent user experiences

Traditional experimentation methods like deterministic bucketing, which assigns a specific variant to a user ID, can be problematic if a user has multiple IDs.

For example, when a user browses a site without logging in, they might see one version of a page. Later, after logging in, they might see a different version of the same page. The user might not notice a minor change, like different colors. But drastic differences between the two variations—like a completely different navigation menu or promotional offer—could be jarring and confusing.

Complexity in handling alias IDs

Some experimentation tools provide ways for alias IDs to attribute conversions from one ID to a variant seen by a different ID. However, implementing these alias solutions often requires significant engineering effort, creating a barrier for teams looking to scale experimentation. This is especially true for product teams with limited resources or those seeking more streamlined, user-friendly experimentation processes.

Challenges in experiment scope

Some traditional tools suggest limiting the scope of experiments to circumvent the issue of multiple IDs. For instance, they might recommend running separate A/B tests for different user states, such as testing landing pages separately from in-app features.

While this approach can simplify identity resolution challenges, it restricts your ability to run comprehensive experiments and might not accurately capture your complete user journey.

Amplitude’s innovative approach to ID resolution

Most experimentation tools are not natively integrated with an analytics platform, so they can’t provide automatic user ID merging. Amplitude Experiment, however, ensures properly attributed results and a consistent user experience—even if they use multiple devices—with no extra engineering lift.

That’s because Amplitude uses remote evaluation. Instead of deciding which variant a user will see within the app (e.g., variant A vs. B), the request is sent to Amplitude’s servers to make the evaluation decision which means you can tap into all of your user data in Amplitude.

For example, Amplitude recognizes that both User123 and User567 are the same person because it tracks them before and after logging in. As soon as the user does something involving both IDs (i.e., browsing as User123 and logging in under User567), Amplitude instantly merges these two IDs with identity resolution.

But what if a user’s device IDs can't be linked together immediately? For example, if the user gets ID’d on two different devices and identified on each before logging in. Though no existing software solution can avoid this issue, Amplitude automatically detects and reports these variant jumping scenarios, enabling you to filter these users from your experiment analysis.

Amplitude Experiment addresses the issue of variant jumping in identity resolution using different techniques:

• Automatically detecting variant jumping: Amplitude Experiment automatically detects cases of variant jumping, enabling experimenters to identify when a single user has been exposed to multiple variants due to having multiple IDs.
• Managing user experiences across IDs: Once the platform detects variant jumping, it can better manage the user experience. If it identifies that another ID associated with the same user has already been exposed to a different variant, it will override deterministic bucketing. This approach ensures that the user receives a consistent experience regardless of their device or ID.
• Filtering users in experiment analysis: Amplitude Experiment enables you to filter users who have experienced variant jumping out of your experiment results. This feature maintains the integrity of your data. By excluding these users, experimenters can ensure that their analysis is based only on users who had a consistent experience.

Use Amplitude Experiment for easy and reliable ID resolution

Amplitude Experiment excels in solving complex identity resolution challenges. Automatically linking multiple devices or user IDs ensures user consistency and enables product teams to track user behavior accurately across different devices and log-in states.

Amplitude Experiment enables quicker, more accurate decision-making to help businesses find opportunities to drive growth and enhance customer experiences. Product teams are empowered to drive innovation, giving your business a competitive edge in creating user-centric products that improve user acquisition, monetization, and retention.