Journal of Digital Health Records, Medical AI and Informatics Systems
Reducing Tail Latency in LLM-Backed Microservices Through Asynchronous Decomposition and Request Hedging
Abstract
Chandrashekhar Singh
Large language model inference services can exhibit substantial latency variability, particularly for long-context workloads. In fan-out applications, the completion time of an application-level request is determined by the slowest downstream call. Consequently, a small number of delayed inference calls can dominate end-to-end response time.
This paper presents an anonymized production-motivated case study of a document-processing microservice that invokes a managed, OpenAI-compatible inference endpoint. We evaluate three designs: a synchronous monolithic request, asynchronous decomposition into independently processed batches, and asynchronous batching combined with delayed request hedging. In the hedged design, a duplicate call is launched only when an original batch exceeds an empirically selected latency threshold, and the first successfully completed response is retained.
Across 100 production-shaped replay requests, asynchronous decomposition reduced typical latency by processing independent batches concurrently, while request hedging reduced the remaining slow tail. In the observed workload, P95 end-to-end latency decreased from approximately 60 seconds to approximately 13 seconds. The improvement was achieved without changing the model family, prompt semantics, or inference infrastructure, although it introduced approximately 10% additional inference-token consumption under the measured latency distribution.
The paper contributes an implementation pattern, an empirical procedure for selecting a hedge threshold, and operational safeguards covering idempotency, cancellation, bounded concurrency, and cost containment. The technique is applicable to read-only, latency-sensitive distributed operations when duplicate execution is safe and the latencies of duplicate calls are sufficiently independent.