How Amplitude Uses Amplitude for No-Code Web Experimentation

Those of us in marketing and growth roles always try to maximize conversion rates across our websites, often using A/B testing. While A/B testing has huge potential to improve conversion rates, historically, growth teams get stuck with hard-to-use technology that is often disconnected from the tools we use every day. I know it’s crucial to figure out how to speak to audiences effectively, but legacy tools create too much guesswork, slow website performance, and a lot of engineering work. This makes earning higher conversion rates and personalizing our web experiences challenging.

When I got early access to Amplitude Web Experimentation, I was thrilled by how easy it was to set up and launch tests. I’m not an engineer, but with self-service experimentation, I didn’t need to be. Here are my takeaways from my initial experiments and what I’m planning for the future of our website’s A/B testing program.

Trying Web Experimentation

As a Senior Website Specialist on the Growth marketing team, I always look for data on how visitors use our website. I want to know what drives traffic, what convinces visitors to sign up for our free trial, and how we turn that conversion into a long-term relationship.

I’m no stranger to running A/B tests using various tools. We use Amplitude here, obviously, but I’ve also worked with point solutions like Optimizely. It was a given that I’d try out Web Experimentation when it launched to see what it could do. I always have so many tests I want to run but lack the resources. This seemed like the perfect opportunity to try one and see what I could learn.

Starting fast with Web Experimentation

Web Experimentation enables the Growth team to get more web testing without using engineering resources. Once the script tag was implemented on our website with only one line of code, I could do everything else on my own.

That means I don’t need to send specifications for each A/B test to engineering and wait for them to make the changes. Web Experimentation simplifies the process with a visual experiment editor that enables me to adjust text, colors, or images on individual pages.

Then, it works like every other A/B test run through Amplitude, where each variant is served to a random number of visitors. I get the results in the Amplitude tool and can draw conclusions based on the data and visualizations since it is natively connected to our powerful web analytics capabilities.

Testing the Amplitude “Butter Bar”

I knew I wanted my first test to focus on the “Butter Bar” on our website. If you scroll up, it’s that banner you see at the top of your screen, and it’s hot real estate. We on the web team are constantly getting requests to use it for various promotions.

The Amplitude website “Butter Bar,” highlighted by the red arrow—the focus of my first A/B test with Web Experimentation.

The problem is that no one knew exactly what performed best in that space. Since I joined Amplitude, I’ve wanted to run tests on the Butter Bar. But that was a challenging ask: Our CMS only allows for one Butter Bar to be applied at any time. Testing it would have required a significant outlay of engineering resources, and there were no guarantees we could have gotten it to work.

With Web Experimentation, I could open up that visual editor and build those variations myself. I hypothesized that we could utilize the Butter Bar across the website to promote more demo sign-ups vs. having it generalized for product and/or content updates. This is still an ongoing experiment, but we’ve seen a 12% lift in CTA clicks and a 30% lift in demo sign-ups with our demo sign-up variant. In short, the leading variant was hugely effective, and we might never have figured that out without Web Experimentation.

What I learned about running an A/B test with Web Experimentation

The speed and simplicity of Web Experimentation sets it apart from other A/B testing tools. It’s hugely amplified our ability to get things done on the growth marketing team. Here’s what we’ve learned about when—and how—to use it most effectively.

It’s best for smaller changes

Web Experimentation shines when it’s used for smaller changes on our website. I don’t think it will replace running bigger scale A/B tests using Feature Experimentation and delivering experiments via feature flags, and I don’t want or need it to. It’s perfect for things like the Butter Bar test or trying things out on less popular pages.

For instance, our homepage, blog, and pricing page get huge amounts of traffic, so it's always been worth expending engineering effort to run tests through Feature Experimentation. But if I want to look at solution or product pages, Web Experimentation enables me to run those tests without feeling like I could be better off using my resources elsewhere.

It’s also a tool I plan to reserve for more minor changes. If we look at overhauling our homepage entirely, that would be a lot of work in Web Experimentation. Figuring out how to implement an entirely new layout is a big challenge, even for someone like me with front-end experience! On the other hand, maybe I want to test some new messaging or imagery on the homepage—the visual experiment editor is great for making those adjustments on the fly.

The great thing about these smaller tests is that they can feed into larger ones. Following our Butter Bar experiment, we are considering updating the functionality to give us more personalized content relevant to our website visitors based on characteristics we know about them and what we think they’ll get the most value from. I can see Web Experimentation returning other results that tell us whether it’s worth investing in an experiment idea. These little, self-serve experiments can serve as a signal that drives our larger strategy.

It helps return quick results

The turnaround on an A/B test can be long. I usually run tests for 2-4 weeks, and then on top of that, if you’re running a feature experiment, you have the concept and design stages and then work with engineering to get everything set up. That can mean months of waiting, and sometimes you want results more quickly than that.

With Web Experimentation, the setup is cut down to just a few days. It’s still important to go through the process of setting a hypothesis and carefully planning your changes, but if you are using the tool for those smaller-scale tests, this is usually much easier. That means you can reasonably get results within the month.

The depth of data helps you draw actionable insights

Seeing beyond the first-level effects of your experiment can help you learn more about your audience and make smarter decisions. We saw this in our Butter Bar experiment: Many A/B testing tools would have only shown us whether our CTA was getting more clicks. Because Web Experimentation is part of Amplitude, we could watch the progression of those new converts as they moved down the product funnel.

We were lucky: Our lift in initial clicks also led to a lift in demo sign-ups. We’re continuing to track those users to see if they become active with our product and eventually convert to our Plus plan, which would be our best-case scenario.

Web Experimentation enables that depth of analysis and provides some excellent segmentation capabilities, too. We’re looking at whether people are on desktop or mobile and what country they’re coming from to see whether that affects the numbers.

If you’re just looking at that first-level effect, with how many people click your new CTA, you’re learning that one specific phrase or sentence activates your audience to take immediate action. What you’re not learning is whether that activation bears out and leads to increased revenue. It’s essential to get a tool like Web Experimentation that helps you understand the full impact of each test.

Getting started with self-serve testing

We see Web Experimentation as a natural extension of our philosophy that self-serve tools help your employees work more effectively and achieve more for your company. Growth and marketing teams can return better results with this sort of autonomy.

Here’s what I would advise teams wanting to try out Web Experimentation.

Don’t ditch the scientific method

The increased velocity enabled by Web Experimentation takes some pressure off your A/B test planning, but it’s essential to stick with the principles that enable data-driven growth. It's still important to start with a clear idea of what you want to test and a solid hypothesis.

Making changes “just to see what happens” is as likely to mislead you as it is to return a helpful insight. Start each A/B test by knowing what you’re looking for and why you’re looking for it. Otherwise, you’ll waste valuable time chasing “improvements” that don’t tie into your larger business objectives.

Don’t rush it

One of the benefits of Web Experimentation is a faster turnaround, but A/B testing will always require time. As I mentioned, I run every test for 2-4 weeks, even if I see something interesting after the first 1,000 data points come in.

Your testing velocity will increase with this tool because it enables you to simplify the setup and implementation process. But don’t rush the crucial part, which is the data-gathering phase. It’s hard to be confident in incomplete results, and that’s all you get if you don’t give your test time to breathe.

Don’t be afraid to fail

A “test-and-learn” mindset is one of the best things you can do to improve in a data-driven way. Web Experimentation lends itself to this because it makes setting up and launching smaller experiments simple. Don’t hesitate to use it to gather data to help you make decisions in ongoing discussions!

So often, it seems like a question has no clear answer, and then you get the data and realize you were just looking in the wrong place. When you have the data in hand, you can more easily determine the impact of your actions and prioritize the right efforts.

Amplitude Web Experimentation has done everything possible to help you run more experiments and improve your data-driven growth efforts. Learn how to get started with this new tool and try it out for free today!