A request to an assistant spends time waiting in line, reading the prompt, and generating one token at a time. This topic names those parts, models the total with example parameters, and shows how streaming, batching and retries fit around it.
Term | Meaning |
|---|---|
Time to first token | Wait from sending a request to seeing the first token. |
Time per output token | Steady pace of each further token once generation starts. |
Prefill | Work that reads the prompt before the first token. |
Queue | Wait for a free slot when the service is busy. |
Throughput | Tokens finished per second across requests. |
Lesson | What it covers |
|---|---|
Two clocks, what sets each, and a worked short-versus-long comparison. | |
Total as queue plus prefill plus tokens times pace, with a runnable model. | |
Streaming for visibility, batching for throughput, and which use case wants which. | |
Request and token budgets, doubling backoff, and retries that stop in time. |
0.40 + 100 x 0.020 = 2.40 s. Streaming shows tokens as they arrive but leaves the total unchanged.