A planner lists every budget item with an EXAMPLE chars-per-token rate and adds them up. It compares the total with the budget and proposes a trim when the plan is over.
The rates below are EXAMPLE parameters, not facts about any real tokenizer. Shuffling the sections never changes the total, which the program checks.
The program below is runnable and uses only the standard library. It estimates four sections, checks the arithmetic against hand-computed constants, and prints one PASS line when every check holds.
import random
import sys
# EXAMPLE estimator parameters, not facts about any real tokenizer.
CHARS_PER_TOKEN_PROSE = 4
CHARS_PER_TOKEN_CODE = 3
BUDGET_TOKENS = 2000
rng = random.Random(11)
SECTIONS = [
("system-prompt", 400, CHARS_PER_TOKEN_PROSE),
("recent-chat", 3000, CHARS_PER_TOKEN_PROSE),
("spec-extract", 2400, CHARS_PER_TOKEN_CODE),
("answer-reserve", 2000, CHARS_PER_TOKEN_PROSE),
]
def estimate_tokens(n_chars, chars_per_token):
return (n_chars + chars_per_token - 1) // chars_per_token
def plan_total(sections):
return sum(estimate_tokens(n, c) for _, n, c in sections)
bad = []
estimates = [estimate_tokens(n, c) for _, n, c in SECTIONS]
if estimates != [100, 750, 800, 500]:
bad.append("section-estimates")
total = plan_total(SECTIONS)
if total != 2150:
bad.append("total")
remaining = BUDGET_TOKENS - total
if remaining != -150:
bad.append("remaining")
if (remaining >= 0) != (total <= BUDGET_TOKENS):
bad.append("fits-flag")
shuffled = list(SECTIONS)
rng.shuffle(shuffled)
if plan_total(shuffled) != 2150:
bad.append("order-check")
trimmed = [s for s in SECTIONS if s[0] != "recent-chat"]
trimmed_total = plan_total(trimmed)
trimmed_remaining = BUDGET_TOKENS - trimmed_total
if trimmed_total != 1400:
bad.append("trimmed-total")
if trimmed_remaining != 600:
bad.append("trimmed-remaining")
if bad:
print("FAIL: " + str(len(bad)) + " budget checks wrong: " + ",".join(bad))
sys.exit(1)
print("PASS: 4 sections need 2150 tokens of a 2000-token budget; trimmed plan needs 1400")The full plan needs 2150 tokens against a 2000-token budget, so it is 150 over. Dropping the recent chat leaves 1400 tokens with 600 to spare.
The order check proves the total is order-independent. Budgeting is commutative even though attention quality is not.
The plan exceeds the budget by 150 tokens. Trim a section or summarize history until the remainder is not negative.
To prove the total does not depend on order. The shuffled total must still equal 2150.