Analytics has changed. Here's what to do about itEverything builders need to know about analytics nowIn the last few years, analytics has changed as much as coding has, but with nearly none of the fanfare. Products now take in more data, scale faster, and have higher costs. Doing analytics the old way will leave you overwhelmed and underinformed. To help you update your knowledge, here are the things you need to know about how analytics has changed, and the best practices for doing analytics now. 1. Let AI make your analytics proactiveHumans can’t handle all the analytics data coming their way. For example, the median company records 9 hours of session replays per month, which leads to 99.8% of session recordings going unopened. AI and automations might do a worse job than humans, but they can do analytics more proactively. This can squeeze more value out of your analytics data that currently goes unused and wasted. Getting AI to do proactive analytics requires giving it direction. It’s not enough to just let AI loose on your analytics data and say “find some fixes.” When given free rein over all your data, agents often get lost or confused. Our proactive analytics agents (we call them Scouts) always have skills to guide them and a clear output format. Examples include:
2. Connect your product and business analyticsPre-AI, you could get away with focusing on product analytics. More users and usage was good because it didn’t cost you anything (or cost very little). Just making your product good was good enough, worrying about the business could come later. This isn’t the case anymore. The primary reason for this is that AI costs a lot of money. Devs pay hundreds of dollars for coding agent subscriptions. AI-powered features can cost a lot. Building AI into your product means it’s no longer possible to scale to hundreds of thousands of users with your only expenses being a $15/m server and Stripe fees. Software doesn’t have zero marginal costs anymore. You’ll actually need to figure out what customers will pay for and how to charge them from the start. You can vibe code an app, but you can’t vibe code a business. AI helps here (again) by:
With data like revenue, subscriptions, account status, ad spend, and AI observability in one place, you can figure out things like:
These are all things that product engineers at PostHog look at when deciding what to build, and it’s only possible because the business analytics data is connected to the product data. 3. Treat agents like a new set of usersAnalytics has not only expanded into new data sources, but whole new types of users: agents. Just like humans, being able to understand what agents are doing is critical to providing them a good experience. Agent analytics is a nascent field, but here’s some tools at your disposal:
This matters because agent usage is different from humans and is growing. A third of organizations that query PostHog do it through agents. Agents create about half of objects in PostHog like insights, dashboards, and feature flags. 1 in 5 new orgs signs up are through an agent. It’s not implausible to think that agents might be the primary user of PostHog in the near future. The same might be true for you. 4. Create shared, verified context (before it’s made up)Analytics is mostly self-serve now. Anyone can ask anything and get plausible and confident answers. The problem with these answers is that they’re not always correct. Even Anthropic found that 1/20 internal analytics queries are wrong. This is because agents often…
You, your agents, your colleagues, and their agents will assume they are right and there’s no gatekeepers (like a data team) to say they’re wrong. This is why shared, verified context matters. It’s how you prevent these inaccuracies from happening. What does this look like?
A lot of this isn’t new. Data teams have always maintained definitions. It’s just more important now that anyone can “do analytics” and that leads them to do it more. AI might make analytics easier, but there’s still work to do to set context so you, your agents, your colleagues, and their agents can “do analytics” well in ways that are specific to your team, needs, and goals. Words by Ian Vanagas, who has also changed. 💼 Help us change analytics more📖 More good reads
|