Amplitude AI Builders: Leo Jiang Discusses AI Visibility

This post is part of our Amplitude AI Builder series. Each one will feature an Amplitude engineer discussing an AI product that they are building.

Marketing means measuring. In the last couple of years, as customers stopped using search engines and started talking to AI chatbots, marketers have had difficult meetings about how to quantify the way their brand shows up in LLM conversations. Those meetings have a lot more questions than answers. I know, because I’ve been in them. Every company wants to be the one that ChatGPT or Claude recommends to interested customers, but that goal is much easier said than done. Even if a team proactively committed to marketing to those LLMs, there was no definite way to measure success or track progress over time.

Luckily for me (and other marketers), Leo Jiang, a staff AI engineer at Amplitude, saw the same problem. Only he had the expertise to build something to solve it. For this post, Leo and I talked about how AI Visibility moved from an idea to a hackathon project to a customer-ready feature at light speed.

What does AI Visbility do?

It helps marketers understand how AI chats are talking about their brand. It does for AI search what SEO tools have done for traditional search engines. Right now, marketers can go to an LLM and ask questions about their products or their competitors, but they won’t get the whole story. They have no way of understanding their brand in the context of a larger LLM aggregate. AI Visibility can identify where brands are relatively weak in AI search. Marketing teams can use that information as a guide to create more content and close those gaps.

With AI Visibility, marketing teams get very clear data about how often their brand is showing up in AI searches. They can track their own company’s performance over time or rank how they compare against specific competitors. It’s got a bunch of tools to help people improve their content to make their company more noticeable in LLM searches.

Our customers can even add Amplitude data on top of that—for instance, to track the number of visitors who came to their websites from AI chats.

How does a brand actually use it?

When you load the page, AI Visibility fetches your data. That data is essentially already computed from an auto-generated list of prompts relevant to each company. For Amplitude, I think there are 400 or 500 prompts. When a company starts using AI Visibility, we ask an LLM to generate hundreds of prompts about that company. Then we run those prompts across a range of LLMs and track the responses. We extract data from those answers, we identify all the other brands that are mentioned, and we collect citations.

When people view the page, we load all this data, then do some aggregations and computations to format it into views that are useful. People can edit their topics, modify their prompts, and rerun the LLMs to try to get more accurate results.

Do you have to be a long-time customer of Amplitude to use AI Visibility?

It’s not connected to your Amplitude data. Amplitude customers can add in traffic data on top of the AI search reporting, but the data AI Visibility uses is essentially public data, so anyone can use it. Teams that don’t use Amplitude can still view their AI search visibility metrics.

How did you decide to build AI Visibility?

I’ve always wanted to build my own startup. I’d talked about that goal with my director, and he suggested that I could do an internal incubator-style startup here at Amplitude. I would need to do the research and propose my own project. That’s how it got started.

I spent a month and a half looking into the right thing to build. I talked to people internally, externally, and I even connected with some of the Amplitude founders to see what they would support.

I’ve been doing SEO for a long time—probably 15 years—so an SEO project was one of the first ideas I had. But I realized that companies are increasingly thinking about how they can show up more in AI chat. Consumers are shifting away from traditional search, and there aren’t many good tools out there that track similar metrics for AI chats. It just made sense for me to work on something that met that new need.

So right now, this is like an SEO engine for LLM searches. What’s next?

We’ll start with collecting data and case studies. We’ll look at what companies are doing with the information they get and what strategies work best for brands to improve their LLM performance. Then we’ll build new tools to help other teams do the same things.

Right now, for example, the common AIO (AI optimization) recommendation is to add FAQs to each page. I’m not sure where that came from or how proven it is, but it came from somewhere, and it’s tribal knowledge. Are FAQs actually improving LLM marketing performance? It’s hard to tell.

We want to solve that. We want AI Visibility to actually give recommendations about what type of language to add to pages—not just generic recommendations, but tactics customized to each brand based on their specific performance. It’s one of our top-requested features. That’s definitely on our roadmap.

What stands out to you as you look back on the AI Visibility project?

I think the key part is how quickly this was built. We built the first iteration in about a week during our AI Week hackathon, and it won the People’s Choice Award for best overall product. About a month after that, we had an Amplitude Wizard Bar where we got it in front of real customers. To go from the first build to showing customers that fast is almost unheard of.

Everyone seems impressed that we built it in a week. It isn’t perfect, and it might be missing features that other products have, but if we did all that in just a week, imagine how quickly we can improve. If we can keep this rate up, we’ll have the best tool on the market very soon.

That kind of speed is only possible because we used Cursor, an AI code generator. I just needed to think of what features to include, the product design, and the system architecture. Cursor handled most of the actual coding. It still made mistakes, so I had to guide it along the way, but it still saved me months of coding time. Even though AI code generators are powerful time savers, having extensive software engineering experience is still invaluable to prevent mistakes from accumulating.

So this is your startup. You took it from nothing to a feature that actual customers are using today. What do you see as your personal signature on AI Visibility?

I don’t just write code. For a true startup feel, I had to do everything—back-end, front-end, marketing, design, PM, customer interviews, sales, hiring, etc. It was a great learning experience to take on a variety of roles and work with a variety of cross-functional teams. Ultimately, my expertise is frontend products, so I’d say having a great UX is my personal signature.