Inference Disaggregation: A Practical Path to Per-Request GPU Cost Attribution

LLM serving is notoriously hard to cost-account. A single request occupies a GPU for a variable amount of time, with wildly different compute profiles depending on prompt length and generation length. Most teams end up estimating costs with back-of-the-envelope averages, which works fine until someone asks "why is my batch so expensive?" A new technique called inference disaggregation aims to change that by separating the two dominant phases of inference—prefill and decode—and treating them as distinct, individually billable units of work.

The core insight is simple: prefill (processing the input prompt) and decode (generating tokens one at a time) have very different compute and memory characteristics. Prefill is compute-bound and highly parallel; decode is memory-bandwidth-bound and sequential. When you run them together on the same GPU, you are forced to provision for the worst case of both, and you can't tell which phase actually consumed the time on a per-request basis. Disaggregation separates the two phases across different GPU pools or time slices, which enables precise measurement and—by extension—fair, per-request cost attribution.

What the framework Actually Does

The arXiv paper outlines a concrete framework for implementing disaggregated inference. It introduces a scheduling layer that routes each request through a prefill worker and then a decode worker, with queues and callbacks to manage the transition. The authors demonstrate that this separation doesn't just add observability—it also opens the door to phase-specific optimizations, such as using different GPU types for each phase or batching strategies tailored to each workload.

The practical upshot for developers is twofold. First, you can finally get a per-request cost breakdown: prefill tokens cost X, decode tokens cost Y, and the total is the sum. Second, you can optimize each phase independently, which can lead to better GPU utilization and lower overall cost. The paper reports measurable improvements in both throughput and cost accuracy in their experiments, though the exact numbers depend heavily on workload characteristics.

Why This Matters for DevTools

Inference disaggregation is not a new idea conceptually—vLLM and other frameworks have played with separating prefill and decode—but this paper makes it a first-class, generalizable framework. For anyone building on top of hosted models (like via an API) or running their own serving stack, this is a step toward more transparent billing and more efficient resource allocation. It also sets the stage for better "pay-per-token" pricing models where users are charged for what they actually consume, not a blended average.

For tool builders, disaggregation could become a key feature in the next generation of model routers: instead of "max throughput" or "lowest latency," you could choose between "cost-optimized" or "prefill-priority" modes. The framework also hints at future research around phase-aware autoscaling and heterogeneous GPU fleets, which would be a significant boon for anyone trying to run LLMs on a budget.

If you're running your own inference stack, it might be worth experimenting with a disaggregated setup—even a simple two-pool version—to see if it gives you better cost visibility and utilization. The paper's approach is practical enough to implement with open-source tools like vLLM or TensorRT-LLM, and the observability alone is a win.