PERFORMANCE ENGINEERING FOR LLM INTEGRATED APPLICATIONS.

Abstract

The integration of large language model (LLM) application programming interfaces into the critical request path of production software introduces a class of performance-engineering problems that classical deterministic system design does not anticipate. Unlike conventional remote procedure calls, an LLM completion call is stochastic in output, autoregressive in structure, and highly variable in latency, since response time scales with generated token count, provider queueing conditions, and model routing decisions rather than fixed computation. This paper characterizes the sources of non-determinism and latency variability in LLM-integrated systems and proposes a structured performance-engineering and testing framework organized around four pillars: token-streaming instrumentation, adaptive timeout strategy, retry-storm-resistant backoff design, and fault-injection-based resilience testing. The framework treats time-to-first-token (TTFT) and inter-token latency (ITL) as first-class service-level indicators, applies deadline propagation across multi-hop agentic call chains, and formalizes retry budgets, jittered exponential backoff, and circuit breakers as the minimum viable resilience surface for AI-dependent services. A comparative synthesis of resilience patterns and a latency-budget taxonomy for four common LLM interaction classes are presented. The discussion argues that performance testing for LLM-integrated applications must shift from throughput-oriented load testing toward probabilistic, chaos-driven validation that treats variable latency and partial failure as expected operating conditions rather than exceptional ones.