Hi there,

Today, Inworld Inference is open to every developer. We rebuilt how open models are served for consumer apps from attention kernels to the router that keeps conversation context cached.

Why rebuild the serving stack? Across the apps we serve, median prompts run 13–36× longer than replies. Long conversation histories go in; short responses come out. Optimizing token generation alone only solves part of the problem.

Here’s what we built differently:

Reuse the conversation, not just the model.

Our KV-aware router matches requests to replicas that already hold their cached context, balancing cache reuse against load. Up to 90% of a repeated prompt can be served from cache, with cached input priced up to 99% lower than uncached input. The model processes less history before starting the next reply.

Route each turn toward its cached context, while balancing load. Conceptual diagram; launch post, Figure 9.

Make every decoding step do more.

Our realtime decoding uses a lightweight draft to propose several tokens for the full model to verify in one pass. But the verification step can become its own bottleneck. On Gemma 4 26B, optimizing that path helped our full build cut median time to first token from 154 ms to 117 ms on replayed production traffic—with unchanged math and reasoning eval scores.

Draft several tokens, then verify them in one pass. Conceptual diagram; launch post, Figure 12.

Give each call its own speed and cost budget.^*

A live reply and a background summary shouldn’t need the same serving configuration. Set service_tier per request, with Fast for latency-sensitive generation and Flex for lower-cost background work. Configure first-token timeouts and compatible fallback models so a stalled model doesn’t leave your app waiting indefinitely.

* Per-request service tiers are currently in beta. General availability is coming soon.

This stack already serves tens of trillions of tokens a month, almost entirely for consumer apps.