Time to first token is the wait from sending a request to seeing the first token. It covers queueing, reading the prompt, and producing that first token.
Time per output token is the steady pace after that: how long each further token takes. Short replies are set by the first clock; long replies are set by the second.
Measure | What it is | Felt when |
|---|---|---|
Time to first token | Queue plus prompt read plus first decode step | Every reply, and dominates short ones. |
Time per output token | Cost of each further token | Long replies, where it multiplies by count. |
Total | First-token wait plus tokens times pace | Planning capacity and deadlines. |
Queue delay comes from load: requests ahead of this one. It changes with the moment, so treat it as a measured range rather than a constant.
Prompt reading grows with input length, and decode pace varies by model and serving setup. Anything that differs between setups varies by model: measure it.
Assume the stated example parameters below. The calc block compares a short reply with a long one.
example parameters (stated, not measured)
time to first token = 0.80 s
time per output token = 0.040 s
short reply, 5 output tokens: 0.80 + 5 x 0.040 = 0.80 + 0.20 = 1.00 s
long reply, 50 output tokens: 0.80 + 50 x 0.040 = 0.80 + 2.00 = 2.80 s
first-token share: 0.80 / 1.00 = 80% for the short reply
first-token share: 0.80 / 2.80 = 2/7, about 28.6%, for the long reply0.80 s of 1.00 s, so 80%. The reply is dominated by time to first token.
Time to first token. The decode pace matches, so the gap sits in queue or prompt handling before the first token.