In this insight
To understand LLM latency, measure the journey around the model as well as the model itself. A slow AI response can come from waiting for a worker, gathering context, generating an answer, saving it or getting it back to the screen.
Adding a faster model may help one stage and leave the user waiting just as long. Adding workers may clear a queue while overwhelming a database or an outside service. The useful first question is not which component looks slow. It is where this request spent its time.
Draw the journey the user experiences
Start when the person presses the button. The browser sends a request. The service checks it and records the work. A worker may pick it up later, retrieve relevant information, call a model, save the result and notify the browser.
Each boundary can introduce waiting or failure. A single timer labelled AI duration hides those differences. Record enough timestamps to separate acceptance, queue waiting, processing and delivery without recording private prompts or source content.
Also define the end. Is the result finished when generation stops, when it is saved, or when the person can read it? For a user-facing measurement, those are not interchangeable. A result sitting in storage while the screen spins is still an unfinished experience.
Use a small timing example
Imagine a fictional request takes twenty seconds. It waits twelve seconds for a worker, spends three seconds gathering context, uses four seconds at the model and takes one second to save and display the result. These numbers are invented to explain the method.
Making the model twice as fast would save only two seconds in this example. Understanding why the queue waits twelve seconds is the more promising investigation. Perhaps the relevant worker is busy, unavailable or responsible for jobs with very different runtimes.
Now imagine a different request with almost no queue delay but a long generation stage. It needs a different investigation. The lesson is not that queues are always the culprit. It is that a visible symptom does not identify the component responsible for it.
Choose measurements that answer different questions
Time to first token measures how long generation takes to begin producing visible text. Total response time includes the rest of the answer. Neither necessarily includes waiting before the model call. Keep the measurement boundaries explicit.
A short answer can finish quickly because it says less, not because the system became more efficient. Compare similar work and record output size, concurrency and whether relevant caches were already warm. Otherwise, a pleasing average may describe a different task.
Look at slower requests as well as the typical one. A median can look healthy while some people repeatedly wait much longer. Separate successful requests, timeouts and failures so that dropping difficult work does not masquerade as an improvement in speed.
Change the bottleneck without moving the failure
If workers are saturated, inspect their workload and capacity. If context retrieval dominates, inspect what is being fetched and why. If model time dominates, consider task size, output requirements and suitable model choices. Change one meaningful variable and compare again.
More concurrency has costs. It may increase contention, memory pressure or provider throttling. A retry policy can worsen an overloaded service if every client retries immediately. Faster submission is not useful if it produces a larger pile of unfinished work.
Cancellation also deserves attention. When a person cancels, does the job stop, or does it keep consuming capacity? The answer can depend on the stage and provider. Make the visible status honest and avoid promising that cancellation always reverses work already performed.
Make waiting understandable while improving it
A truthful progress display helps the person decide what to do. Waiting to start, gathering information and generating a result are more informative than a percentage that has no measured basis. Do not imply the answer is ready before it is saved.
If a page reconnects or a user refreshes, it should find the existing job rather than blindly submit another. Otherwise, a slow response invites repeated clicks, and those clicks create the very queue the team is trying to reduce.
Validate changes through the whole journey. The success condition is not merely a shorter provider timer. It is a correct result reaching the person more reliably, at an acceptable cost, with clear behaviour when it cannot finish.
Model conversion exercise
Use this exercise with a real workflow and the people accountable for its outcome. Record disagreements as modeling questions instead of silently choosing an answer.
Draw a timing line for one test request using fictional input. Mark acceptance, worker start, context ready, model start, first output, saved result and visible result. Identify the longest interval. Repeat under a small, controlled increase in concurrent requests and compare like-for-like tasks. Record failures separately, and do not send customer content into a profiling tool without authorization.