Sampling draws one token according to the step probabilities. High-probability tokens appear most often, but lower-probability tokens remain possible.
Greedy decoding instead always picks the top token. Sampling adds variation, while greedy selection removes it at the cost of a single fixed path.
Strategy | Rule | Outcome |
|---|---|---|
Greedy | Always take the highest-probability token | One fixed continuation per prompt |
Sampling | Draw one token with the stated probabilities | Different runs can take different paths |
Temperature rescales the scores before normalization. Lower values sharpen the distribution toward the top token, and higher values flatten it across alternatives.
Temperature 0 is approximately greedy selection. Even then, bit-identical output across runs or hardware is not guaranteed, so compare behavior rather than assuming it.
Setting | Distribution shape | Typical effect |
|---|---|---|
Lower temperature | Sharper, top token dominates | Replies look more repeatable |
Temperature 1 | Unchanged baseline shape | Reference sampling behavior |
Higher temperature | Flatter, alternatives more likely | Replies vary more between runs |
Each step draws from its distribution, and one different token redirects the rest of the reply. Two runs of the same prompt therefore diverge whenever any step draws differently.
The program below uses a FAKE scripted probability table in place of a model, not a real system. It shows sharpening and flattening over 2000 draws from a fixed 3-token table with a fixed seed.
import math
import random
import sys
VOCAB = ["reset", "clock", "enable"]
LOGITS = [2.0, 1.0, 0.0]
TEMPS = [0.5, 1.0, 2.0]
SEED = 7
DRAWS = 2000
def softmax(logits, temp):
scaled = [x / temp for x in logits]
m = max(scaled)
exps = [math.exp(s - m) for s in scaled]
total = sum(exps)
return [e / total for e in exps]
def sample_index(probs, rng):
r = rng.random()
acc = 0.0
for i, p in enumerate(probs):
acc += p
if r < acc:
return i
return len(probs) - 1
def entropy(probs):
return -sum(p * math.log(p) for p in probs if p > 0.0)
def bad(msg):
print("FAIL: " + msg)
sys.exit(1)
dists = {t: softmax(LOGITS, t) for t in TEMPS}
for t in TEMPS:
if abs(sum(dists[t]) - 1.0) > 1e-9:
bad("probability total not one")
for t in TEMPS:
top = max(range(len(VOCAB)), key=lambda i: dists[t][i])
if top != 0:
bad("top token moved")
p_top = {t: max(dists[t]) for t in TEMPS}
if not (p_top[0.5] > p_top[1.0] > p_top[2.0]):
bad("sharpen order wrong")
ent = {t: entropy(dists[t]) for t in TEMPS}
if not (ent[0.5] < ent[1.0] < ent[2.0]):
bad("spread order wrong")
rng = random.Random(SEED)
counts = {t: [0, 0, 0] for t in TEMPS}
for t in TEMPS:
for _ in range(DRAWS):
counts[t][sample_index(dists[t], rng)] += 1
for t in TEMPS:
if sum(counts[t]) != DRAWS:
bad("draw total wrong")
if min(dists[t]) <= 0.0:
bad("alternative token impossible")
print("PASS: cooler sharpens and warmer flattens fixed 3-token table over 2000 draws")A frequent mistake is reading temperature as a quality dial. It controls variation, while verification decides quality.
Variation shrinks. The top token wins more draws, so reruns follow the same path more often.
No. Fixed scores plus different sampling draws produce exactly this pattern, starting at the first draw that lands elsewhere.