Inworld Inference product launch email

Hi there, Today, Inworld Inference is open to every developer. We rebuilt how open models are served for consumer apps from attention kernels to the router that keeps conversation context cached. Why rebuild the serving stack? Across the apps we serve, median prompts run 13–36× longer than replies. Long conversation histories go in; short responses come out. Optimizing token generation alone only solves part of the problem. Here’s what we built differently: Reuse the conversation, not just the model. Our KV-aware router matches requests to replicas that already hold their cached context, balancing cache reuse against load. Up to 90% of a repeated prompt can be served from cache, with cached input priced up to 99% lower than uncached input. The model processes less history before starting the next reply. Route each turn toward its cached context, while balancing load. Conceptual diagram; launch post, Figure 9. Make every decoding step do more. Our realtime decoding uses a lightweight draft to propose several tokens for the full model to verify in one pass. But the verification step can become its own bottleneck. On Gemma 4 26B, optimizing that path helped our full build cut median time to first token from 154 ms to 117 ms on replayed production traffic—with unchanged math and reasoning eval scores. Draft several tokens, then verify them in one pass. Conceptual diagram; launch post, Figure 12. Give each call its own speed and cost budget.* A live reply and a background summary shouldn’t need the same serving configuration. Set service_tier per request, with Fast for latency-sensitive generation and Flex for lower-cost background work. Configure first-token timeouts and compatible fallback models so a stalled model doesn’t leave your app waiting indefinitely. * Per-request service tiers are currently in beta. General availability is coming soon. This stack already serves tens of trillions of tokens a month, almost entirely for consumer apps. Gemma 4 26B in production: about 1.8x requests per serving instance, with faster first tokens. September-October 2026; launch post, Figure 16. With Inworld, Latitude migrated its first production task in just one day, while Status by Wishroll cut AI costs by more than 95% without sacrificing content quality. Why isn’t turning on prefix caching enough? What makes draft-and-verify decoding faster under production load? Our launch post goes inside the architecture, measurements, and trade-offs behind the results. Read the full launch post → Join the conversation on X and LinkedIn. The Inworld Team Inworld AI, 1975 W El Camino Real, Suite 300, Mountain View, CA 94040, USA Manage preferences