Latency-Aware Routing Dynamically Balances Speed and Quality Across LLMs

A new paper on arXiv introduces a latency-aware routing framework for large language models that dynamically selects between multiple LLMs based on predicted response times and task requirements. The approach aims to solve a common trade-off in deployed LLM systems: fast, smaller models often sacrifice quality, while larger models provide better responses but introduce higher latency.

The proposed system monitors real-time queue lengths, model throughput, and network conditions to estimate the end-to-end latency for each available model. When a user request arrives, the router predicts how long each model would take and selects the one that meets user-specified latency constraints while maximizing output quality. Early experiments show that latency-aware routing can reduce p99 response times by up to 40% compared to naive load-balancing, while maintaining comparable or better quality scores across benchmarks.

How It Works

The router maintains a lightweight prediction model per LLM, updated continuously based on recent observations. For each incoming request, it estimates completion time for every candidate model, then applies a cost function that penalizes both quality degradation and latency violations. The model with the best trade-off is selected.

This is particularly useful for applications with varying latency budgets, such as chatbots in customer service (where some queries can wait for a high-quality answer) versus real-time code completion (where sub-second responses are critical). The framework is model-agnostic and can work with any LLM serving infrastructure.

Implications for Developers

For teams deploying multiple LLMs behind a single API, latency-aware routing offers a way to automatically optimize the speed–quality frontier without manual tuning. It also opens up possibilities for cost savings by routing simple queries to cheaper, faster models while reserving expensive models for complex tasks.

The paper's authors plan to open-source the routing library in the coming months. Until then, developers can implement their own version using basic queuing theory and online learning techniques described in the paper.