Nexus: KV-Cache Routing to Slash LLM Inference Costs

As large language models move from research curiosities to production workloads, the infrastructure needed to serve them is becoming an increasingly bitter pill. One of the least glamorous but most stubborn bottlenecks is the key-value (KV) cache, the memory structure that stores attention state across generation steps. It grows with context length and batch size, and for long-context applications it can dwarf the model weights themselves, forcing clusters to add more GPU memory just to keep up.

A new preprint, "Nexus: Efficient KV Cache Routing for Large Language Model Serving," proposes a system-level fix: instead of pinning the KV cache to a single GPU or node, route it across a pool of machines and pull it back only when the next token needs it. The goal is to decouple the cache footprint from the memory ceiling of any one device, allowing a cluster to serve far more concurrent requests — and much longer contexts — with the same hardware.

The Problem with Monolithic Caching

In a typical LLM serving setup, each request holds its KV cache on the GPU that runs its forward pass. That's fast — no network hops — but it creates a hard coupling between the size of the context you can serve and the capacity of one accelerator. The result is that long-context workloads either require expensive H100-class hardware or force requests to queue, wasting idle capacity elsewhere in the cluster.

Several existing systems try to mitigate this with caching eviction or compression, but they trade away quality by dropping information. Nexus takes a different approach: keep the full cache, but treat it as a distributed resource rather than a per-GPU one.

How Nexus Works

The core idea is a router that sits between the request scheduler and the cluster. When a request's context grows past a threshold, Nexus splits its KV cache across a configurable set of nodes. Each generation step, the router determines which nodes hold the relevant cache slices and fetches them just-in-time, overlapping that transfer with compute on the current device.

Key components include:

• Cache-aware routing: Nexus tracks the location and size of every cache slice, using a global view to balance load across nodes.
• Tiered placement: Frequently accessed slices stay on fast local memory while cold slices are pushed to slower but cheaper remote storage.
• Latency hiding: The system prefetches the next slice while computing the current token, so the network I/O is hidden behind the model's maths.

The authors report that in trace-driven simulations with production workload patterns, Nexus reduces end-to-end latency by up to 45% and increases sustained throughput by roughly 2.5× compared to a baseline that keeps caches local, while also shrinking memory waste from underutilized reserved caches.

The Catch: It's Still a Research Paper

The results are promising, but the paper is primarily simulation-based. Real-world serving systems have to contend with network jitter, heterogeneous hardware, and the sheer complexity of scheduling thousands of concurrent requests. The authors acknowledge that their router adds a few milliseconds of decision overhead, which could be nontrivial for ultra-low-latency interactive use cases.

Still, the direction is right. As model sizes plateau but context windows balloon — 200K and 1M token contexts are already shipping — the KV cache is becoming the dominant cost in inference. Any scheme that lets you amortize that cost across a cluster rather than paying it per GPU is worth watching. Nexus is one of the more practical proposals to date, because it doesn't require changes to the model or the serving framework's core logic; it just slots in as a routing layer.

For developers running private models or building inference platforms, this is the kind of systems research that turns into a library you can pull in six months, not a breakthrough that rewrites your stack. If you're currently bumping your head against out-of-memory errors on long prompts, keep an eye on this one.