[tag]

#llm-serving

2 stories

[research]Aug 24, 2026By ByteBulletin Editor

Nexus: KV-Cache Routing to Slash LLM Inference Costs

New research introduces a routing mechanism that distributes key-value cache storage across machines to cut memory overhead and latency in large-scale LLM serving.

[research]Aug 3, 2026By ByteBulletin Editor

Inference Disaggregation: A Practical Path to Per-Request GPU Cost Attribution

A new framework separates the 'prefill' and 'decode' phases of LLM inference to give teams granular, fair cost accounting per request.