Picking examples

Coverage beats count. A small set that spans the classes teaches more than a large set from one class.

Include one boundary case per class, such as an asserted-low reset near an asserted-high one. Effects vary by model, so candidate sets are compared by measurement.

The program below uses fixed lists, not a model. Spread picks covering three classes beat the first three pool entries covering two.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import sys

POOL = [
    ("a", "reset"),
    ("b", "reset"),
    ("c", "enable"),
    ("d", "enable"),
    ("e", "handshake"),
    ("f", "handshake"),
]

def covers(picks):
    seen = []
    for i in picks:
        cls = POOL[i][1]
        if cls in seen:
            pass
        else:
            seen.append(cls)
    return seen

first_three = covers([0, 1, 2])
spread = covers([0, 2, 4])

if len(first_three) == 2:
    pass
else:
    print("FAIL: first-three cover changed")
    sys.exit(1)
if len(spread) == 3:
    pass
else:
    print("FAIL: spread cover changed")
    sys.exit(1)
print("PASS: spread picks cover 3 classes, first three cover 2")

Ordering and format

Order effects are reported for some models and absent for others. The safe rule is a fixed order with a fixed format.

Tests reshuffle the same pairs to check stability. A set that survives reshuffling is kept; a fragile one is rewritten.

Budgeting context

Long inputs cost more and are used less reliably than short ones. Common tokenizers split English prose into subword units, about 3 to 4 characters per token as an approximate tokenizer-dependent rule.

Code, numbers, hex, and non-Latin text use more tokens per character. In the basic attention form, cost grows with the square of the sequence length, while efficient variants exist.

Models tend to use the start and end of long inputs better than the middle, a tendency reported in studies rather than a law. The worked budget below keeps shares exact and summing to 100 percent.

1
2
3
4
5

EXAMPLE context budget = 4000 tokens
instruction 300, examples 2400, input 1000, reserve 300
total = 300 + 2400 + 1000 + 300 = 4000 tokens
shares: 300/4000 = 7.5%, 2400/4000 = 60%, 1000/4000 = 25%, 300/4000 = 7.5%
sum = 7.5 + 60 + 25 + 7.5 = 100

Common mistakes

A prompt that grows without a budget eventually squeezes out the new input. Reserves protect the answer space.

Related topics

• Zero-shot and few-shot

• Prompt debugging

• Caches