The prediction step

Each step takes the text so far and returns a score for every vocabulary item. Normalizing the scores gives a probability for each possible next token.

The model never writes a sentence at once. It only answers which token could come next and how likely each candidate is.

The generation loop

Generation repeats the prediction step. Each sampled token is appended and the longer text is scored again.

A short reply of 20 tokens takes 20 such steps. The early choices shape every later distribution.

1
2
3
4
5

joint probability of a 3-token continuation, exact
p = 0.8 x 0.7 x 0.9
  = 0.56 x 0.9
  = 0.504
result: 0.504, dimensionless probability

1
2
3
4
5

schematic illustration, not real model output, no specific system implied
prompt tokens: [write, a, reset]
step 1 candidates: reset 0.5, clock 0.3, enable 0.2 -> sampled: reset
step 2 candidates: block 0.6, logic 0.3, wire 0.1 -> sampled: block
final text: write a reset block

Tokens in and out

Model input and output are subword units, and the common approach is BPE-like segmentation. Boundaries rarely match whole words.

As an approximate rule of thumb, English prose averages about 3 to 4 characters per token with common tokenizers. Code, numbers, hex and non-Latin text use more tokens per character, and the exact ratio is tokenizer-dependent.

Common mistakes

A frequent mistake is treating one reply as the only possible reply. The distribution admits many continuations and sampling picks one path.

Related topics

• Training and inference

• Tokens and tokenization