Total latency is the sum of three parts: queue wait, prompt read (prefill), and one decode step per output token. Each term takes an example parameter, stated wherever it appears.
The form is total = queue + prefill + n_output x time_per_token. It is a planning aid, not a performance guarantee.
Term | Unit | Grows with |
|---|---|---|
Queue | Seconds | Load at the moment of the request. |
Prefill | Seconds | Input length and prompt processing. |
n_output x time_per_token | Tokens x seconds per token | Reply length at the measured decode pace. |
The calc block evaluates the model for three cases. Each line shows the substitution before the result, so it checks by hand.
total = queue + prefill + n_output x time_per_token (example model)
case A: 0.00 + 0.40 + 100 x 0.020 = 0.40 + 2.00 = 2.40 s
case B: 0.50 + 0.60 + 200 x 0.015 = 1.10 + 3.00 = 4.10 s
case C: 1.20 + 0.30 + 40 x 0.025 = 1.50 + 1.00 = 2.50 s
case A is decode-bound; case C is queue-boundThe program below implements the same sum and rechecks the three worked totals. Two generators with seed 7 must agree, which shows that fixed seeds repeat.
import random
import sys
def total_latency(queue_s, prefill_s, n_out, tpot_s):
return queue_s + prefill_s + n_out * tpot_s
cases = [
(0.00, 0.40, 100, 0.020, 2.40),
(0.50, 0.60, 200, 0.015, 4.10),
(1.20, 0.30, 40, 0.025, 2.50),
]
bad = 0
for q, p, n, t, exp in cases:
if abs(total_latency(q, p, n, t) - exp) > 1e-9:
bad += 1
print("FAIL: case %d total differs from its worked value" % bad)
sys.exit(1)
r1 = random.Random(7)
r2 = random.Random(7)
if [r1.random() for _ in range(4)] != [r2.random() for _ in range(4)]:
print("FAIL: same seed produced different draws")
sys.exit(1)
print("PASS: 3 latency totals and repeated draws from seed 7 agree")1.20 + 0.30 + 1.00 = 2.50 s. The queue term dominates.
100 x 0.020 = 2.00 s extra. Queue and prefill do not move with output length.