Streaming sends each token as it is generated instead of holding the whole reply. The first token arrives after time to first token, and the rest follow at decode pace.
The total time is unchanged. What changes is when partial output becomes visible.
Style | First visible | Full reply done |
|---|---|---|
Buffered | When the last token finishes | Queue plus prefill plus all decode steps. |
Streamed | After time to first token | Same total as buffered. |
A server can decode several requests in one batch, sharing each step across them. Each request may run slower per token while the server finishes more tokens per second.
The example below assumes batching slows each request from 0.020 s to 0.050 s per token. That assumption is stated because the real trade varies by setup: measure it.
example parameters (stated, not measured)
alone: 0.020 s per token; batched: 0.050 s per token (assumed)
one 100-token reply alone: 100 x 0.020 = 2.00 s
eight 100-token replies in one batch: 100 x 0.050 = 5.00 s each
batch tokens per second: 800 / 5.00 = 160 against 1 / 0.020 = 50 aloneInteractive use values the first token: short replies, streaming on, small batches. Bulk use values tokens per second: long replies, large batches, no need to watch output arrive.
Pick the serving shape from the use case. One shape rarely serves both well.
Use | Goal | Serving shape |
|---|---|---|
Interactive | First token fast | Streaming on, small batches. |
Bulk | Tokens per second high | Large batches, streaming optional. |
First token at 0.80 s; full reply at 2.80 s. Streaming moves visibility, not the total.
5.00 s each. 800 tokens in 5.00 s is 160 tokens per second.