Training adjusts the model numbers so that real text from the training collection scores higher. It processes many sequences in batches and updates the numbers after each batch.
Training is done before use and is far more compute-intensive than answering one prompt. Details vary by model, so treat any specific schedule as an illustration only.
EXAMPLE parameters, stated illustration only, checkable by hand
batch = 4 sequences x 512 tokens per sequence
= 2048 tokens per update step
unit: tokens per update stepInference freezes the trained numbers and answers new prompts. Each step reads the numbers to score the next token without changing them.
One prompt produces one sampled path through the loop. Answering again re-runs the same fixed numbers with fresh sampling draws.
Phase | Numbers | Input | Output |
|---|---|---|---|
Training | Updated after each batch | Collection text with known continuations | Adjusted numbers that predict the collection better |
Inference | Fixed, only read | A new prompt | Sampled continuation tokens |
Cost, hardware, and controls differ between the phases. Training throughput is measured over batches, while inference latency is measured per prompt and per token.
Behavior controls such as sampling settings apply at inference. Retraining or further tuning is the route when the numbers themselves must change, and the exact route varies by model.
Aspect | Training | Inference |
|---|---|---|
Numbers change | Yes, every update step | No, numbers are read-only |
Typical input size | Batches of many sequences | One prompt plus its growing continuation |
Repeated runs differ | Runs differ by data order and setup | Runs differ by sampling draws |
A frequent mistake is asking inference to learn a correction permanently. A correction inside one session changes the text in context, not the stored numbers.
2048 tokens. Four times 512 equals 2048 tokens per update step.
The correction changed the session text, not the stored numbers. A new session reloads the same numbers, so the old prediction can recur.