PostHog
Day 45 after signup
Not everything needs an LLM
How we use decision models
Why it works
- Open with a relatable frustration, then introduce the alternative so readers quickly understand why the topic matters.
- Organize technical advice around distinct use cases, pairing each with a concrete example and a caution.
- End each section with a concise takeaway that turns the example into guidance readers can apply.
How it is built
- Publication header
- Article headline and byline
- Problem framing
- Use-case sections
Email copy
Forwarded this email? Subscribe here for more Not everything needs an LLM How we use decision models Jina Yoon Nov 9 READ IN APP LLMs love to yap. Ask a simple question and you get a few hundred words of "it depends." It's annoying as hell. This is why we love decision models like Jev for simpler tasks. You ask a structured question and get a typed answer with a probability, which looks something like this: Noul. Are hedgehogs mammals? → Yes, 0.99 Choice. Which animal family do hedgehogs belong to? → Mammal, 0.99 Score. What is the current conservation status of hedgehogs? → Least Concern, 0.77 Whereas LLMs return free-form and overly confident prose, decision models make model uncertainty explicit. In a calibrated model, answers that score 0.8 are right about 80% of the time. That means a score of 0.5 is a coin flip. We've been experimenting with them at PostHog and several use cases have made it to production. This is everything we've learned about using decision models so far: when to use them, how to use them, and mistakes to avoid. 1. Use them to gate expensive LLM calls Decision models are up to 444.6x cheaper and 193.6x faster than frontier models,¹ so you can use them to check whether you even need to make an expensive LLM call at all. We do this in a pipeline that scrubs personal info in Replay Vision, our product that watches and summarizes user session recordings for you. Here's how it works: Each session summary goes through 2 yes/no questions from a decision model. Using narrow questions like this cuts false flags to about a tenth of what a single question produced. 1. “Does this name a real person?” 2. “Does this show contact details or an ID?” If either scores 0.4 or above, Gemini rewrites the summary without the personal info and it gets checked again. Otherwise, it's clean and passes through the gate. In production, 98.7% skip the LLM call entirely. Without the decision model, we'd have to keep fussing with prompts and regexes that miss things, or sending every summary to Gemini to catch the 1% that need it. But remember they make mistakes A probability score of 0.85 still means ~150 wrong answers per 1,000 checks, so plan ahead for false positives and false negatives. In the PII example, a false positive only costs us an extra LLM call, whereas a false negative stores personal data as clean. That's why we set the threshold low at 0.4. When false positives are the costlier mistake, use higher thresholds. For example, we recently updated the PostHog in Slack bot, so it can answer questions without being tagged. To avoid being annoying, it checks every message in a channel to see (1) whether it's asking for information, and (2) whether PostHog could answer it. Only messages that pass both at 0.85+ go through. Takeaway: Use decision models to filter jobs before they hit an expensive LLM call, and set the threshold based on which mistake costs you more. 2. Use them as model routers Our PostHog in Slack bot also uses a decision model as a model router. When the bot gets tagged, it first checks whether the message asked for a specific model. If not, one choice question picks from the list: “Pick the cheapest option that will most likely finish the task correctly on the first try. A failed run costs more than a smarter option.” This works because decision models are calibrated to work out of the box. Before, you'd have to use a cheap LLM, or go through all the effort of training your own classifier. But don't assume they're interchangeable Different tasks, models, and providers all have different thresholds. We were experimenting with using them to auto-merge similar logs in Error Tracking and found that thresholds changed between v1 and v2 of the same model. We had to make its thresholds configurable per PostHog project and keep the judge in shadow mode while recalibrating based on hand-labeled data. This is one of the reasons why Fernando on our growth team pushed for a model-agno