How To Structure and Manage Your Product Analytics Team

Over the last fifteen years, I have noticed a pattern. As I have worked with companies, departments, and teams to help them make more data-informed decisions, I have noticed that they usually fall into one of three categories:

• They fully integrate evidence-based decision-making into how they work
• They aspire to be in the first group, but have yet to experience that enlightened and elusive state of full integration (they are still building the muscle).
• They are struggling to take their first steps.

No one wants to be in the third group. Someone, or something, is blocking their first steps. The first group is intimidatingly small, and you can be sure that anyone bragging about being there probably isn’t as evolved as they think they are. For the rest of us, and as an analytics practitioner I do mean “us,” there’s the second group.

If you’re going to be data informed, you have to start somewhere. And if you’re in the thick of working your way up that curve, you probably have lots of questions. With that in mind, I’d like to share some answers to questions I’ve asked (and been asked) along the way, as well as some hints and gotchas.

Do we really need people specifically dedicated to product analytics?

Answer:

Sort of. The goal is better decisions. To make better organization-wide decisions using data, you’ll need some level of data literacy. To spread data literacy in an organization you’ll need:

1. People who are data literate and eager to teach others
2. Teammates who are eager and incentivized to absorb that teaching
3. Effective ways to teach people how to fish, and 4) effective ways to let them fish

That’s a daunting to-do list to navigate for a full-timer, but especially for someone tackling this in their spare time. Sometimes, you’ll have product managers, UX researchers, engineers, or business analysts with decent chops, but they’re lucky if they can focus those skills on their particular team, current project, or area, let alone on all teams, projects, and areas.

So yes, I’d say you need at least one person dedicated to analytics. But hiring this “product analytics manager” without the express goal of raising the data literacy bar across the company—including letting them own or influence the selection of tools and clearing organizational roadblocks—will end in a downward spiral of busyness.

There’s another area where having a dedicated product analytics manager helps, and that is making sure you have a lightweight system in place to maintain your data taxonomy, even in the face of new product introductions, frequent product updates, and lots of ongoing experiments. But again, it is unlikely they’ll “control” this in a vacuum. If they try to, people will just work around them.

So the short answer here is “yes, but consider making their focus raising overall data literacy.”

Potential Pitfall:

Don’t forget about the qual! The oft used saying “Quant tells us what. Qual often tells us why,” has held true in my experience. Data literacy extends to how you elicit and interpret qualitative data. You can apply the same four point thought process above to qualitative research.

Having people focused on and dedicated to raising overall literacy in this area is extremely helpful. Some of my most rewarding professional experiences have involved partnering with experienced UX researchers.

Ok, we need a product analytics “manager”, but how should we deploy them?

Answer:

Rather than being prescriptive about a particular org structure, it’ll be more useful to talk about how you can arrive at your answer. It is impossible to be “fit-for-purpose” for everything. Instead, consider what your current structure is really optimized for, and the answer to that question can yield some interesting insights.

A product analytics team should be structured based on a number of factors including the nature of your product, current stage in the product life cycle, product strategy, where domain expertise is located in your organization, and who is making product decisions. Helpful prompts include:

• Where are decisions actually happening? Who is making those decisions? What types of decisions are we making?
• What information is needed to make those decisions?
• Where is that information coming from?
• Where are our questions similar, and where are they variable?
• Which aspects of our business and product strategy are stable, and which aspects are emergent?
• Where is consistency and repeatability important? Where is flexibility important?
• Where is extremely high confidence necessary? Conversely, where is a marginal reduction in uncertainty acceptable?

What becomes very clear when pondering these questions is that most organizations have a great deal of variety in terms of context. Few operate as a monolith. You’ll have highly exploratory new product efforts, coupled with “stable” KPIs serving as the heartbeat for the whole business.

A helpful way to make sense of the reality of concurrent organizational demands is Simon Wardley’s concept of Pioneer-Settler-Townplanner (which, according to Wardley is “a derivative of Robert X. Cringely’s description of companies as Commandos, Infantry, and Police as expressed in the delightful 1993 book, Accidental Empires”).

Here’s a great graphic from Wardley explaining the concept. For the purpose of this discussion, pay close attention to the “Deals with…” and “Happy with…” sections.

Credit: Simon Wardley, On Pioneers, Settlers, Town Planners, and Theft

Let’s consider how we’d address the varying needs of Pioneers, Settlers, and Town Planners.

Pioneers

How might your analytics team meet the needs of a pioneering effort that is poorly understood, novel, a gamble, requires experimentation, and will ignore the customer initially?

To reference Pedro Uria-Recio’s wonderful post Organizing Analytics like the Human Brain, you might choose embedding in a functional area or a fully decentralized approach. Why?

Or in the case of true Pioneers, one of the “founders” might be a data polyglot themselves. Whichever you choose, you’ll need to use exploratory techniques, be unconcerned about scalability/consistency, and above all seek to navigate the novelty and inherent risks.

Settlers

Now for supporting Settlers. This challenge might be a better fit for an analytics team that is decentralized, but it also belongs to a center of excellence. Again, from Uria-Recio:

With Settlers, you may be more interested in repeatability, known frameworks for acquisition and retention, the ability to scale your taxonomy and data trust, and understanding the impact of incremental improvements. “Listening to your customers” in the form of understanding how they interact with your product, combined with qualitative data, is important for this group. With Settlers we are squarely in the realm of product analytics and products like Amplitude.

Town Planners

Finally, you have supporting Town Planners, which could be a good fit for a centralized and/or consultative structure, with a focus on standardization and efficiency. This might be a better fit for more traditional BI tools.

That’s a lot to process, but the basic idea is to structure your analytics team according to the various needs of your organization. Also critical is the idea that Pioneer efforts are often enabled (but not obstructed and slowed down) by Town Planner services.

You can’t cookie-cutter your approach; you have to be a thoughtful systems thinker.

Hint:

Whatever model you choose, there’s one very important thing to keep in mind: your structure has to provide all team members the opportunity to build/create user empathy. It’s virtually impossible for an analyst to provide real insight when they have no connection to the customer (whether internal or external). Speaking as a former analyst, the human connection is critical for job satisfaction and doing your best work.

How many product analytics specialists do we need?

Answer:

Predictably, it depends on the size of your team and the org model you choose. Here are a couple of personal anecdotes to illustrate the extremes.

Limitless. First, confession time. I once accepted a phone interview with a SaaS giant where my primary objective was to learn how they organized their analysts. As it turned out, this particular company has not only an analyst, but also a Data Scientist on each product development team. Every team. Mind, blown. So, yes, there are companies who do that, and it works for them. This particular company is definitely in group one.

Team of One. On the other hand, there’s a different personal experience. For a couple years I was a one-person product analytics operation in a not-so-small SaaS company. One person. A couple dozen (maybe 30) dev teams. Zero self-service tools. Suffice it to say, this was not a super-effective analytics delivery method. And telling 95+ percent of your constituency “no” on a weekly basis is very demoralizing.

But that doesn’t mean we didn’t make progress. As the old saying goes, “You have to say no to say yes,” and I was able to push things forward with two teams by focusing on them. Yes, it was to the exclusion of others, but by focusing on small, defined use-cases, we were able to further the department’s understanding of the way our product was being used.

Numbers, but with challenges. Yet another experience was one where it seemed like all of the necessary pieces were in place. Dedicated analysts, engineering support, an appetite for analytic insights, the best tools and infrastructure money could buy—it was all there—but there was one critical element missing: leadership alignment.

Those analysts, engineers, and insight hungry consumers? They all answered to different masters, and those masters couldn’t (and never actually did) agree on what was important: how goals should be defined and achieved or who should be responsible for executing what. Even the person who was supposedly “leading” had no authority to execute a plan (to the extent that one even existed in the first place).

So how about headcount?
What did these disparate experiences teach me about “numbers”? That structure and really establishing your core purpose is more important than numbers. You can do a great deal with a small team if you are free to embed, partner, teach, and centralize the services that make sense, as well as offering self-service tools to the frontlines. With valuable outcomes comes support, and with a strong foundation comes scaling what works instead of scaling for scaling’s sake.

Simply adding numbers to a centralized team that is straining under high work in progress, top-down requests, and fragile tooling will probably make the situation worse, not better. You’ll be number crunchers at best, rather than decision supporters and data literacy multipliers.

Hint:

There are force multipliers you can put in place to help democratize the use of data in decision making. Here are two that are among the most powerful:

• Education: Give analysts the opportunity to share and teach data literacy skills, both formally and informally. They’ll appreciate the opportunity to be noticed and appreciated. Even better, they’ll love it when other team members are able to pitch in.
• Self-service: Your stack will undoubtedly have components that require users to possess some technical (coding) skills. But if it’s going to act as a force multiplier, there need to be tools that don’t have that barrier to entry. Introducing a self-service component to your ecosystem will help your entire team become more capable with data.

Most importantly, why are we doing this?

Answer:

Better decisions, better outcomes, happier customers, and happier teammates.

We want to up our analytics game to better understand the value customers get out of using our product/service. When we deepen our understanding of the value customers get from using our product/service, we can make it easier to unlock that value and deliver more of it, faster.

At the end of the day, analytics is tough. It’s science…but it’s also art. If it was easy, everyone would be in that first group. It’s important to hold on to the perspective that competently leveraging your data is a journey. There is no finish line. No bar to get over. No list of boxes to check. Just keep taking that next step, and don’t forget to celebrate your wins along the way.