Subword units (BPE-like) are the common approach to tokenization. A working rule is about 3 to 4 characters per token for English prose with common tokenizers, and fewer for code, numbers, hex, and non-Latin text.
All such rules are approximate and tokenizer-dependent. In the basic form, attention cost grows with the square of the sequence length, though efficient variants exist.
Long-context use is less reliable than short-context use. Models tend to use the start and end of long inputs better than the middle, a tendency reported in studies rather than a law.
An index covers the whole tree while each question reads only a few files. The working set stays small and the full text stays on disk until needed.
Option | Cost per question | Weakness |
|---|---|---|
Paste everything | 300000 tokens each read. | Costly, slow, and the middle goes unreliable. |
Targeted retrieval | About 8000 tokens of index hits. | Needs a usable index of files and symbols. |
Module summaries | One page per block, refreshed rarely. | Summaries drift when code changes. |
Agent with file tools | Reads on demand within a step budget. | Must be capped or it reads the whole tree anyway. |
Counts and quality vary by model: measure both on this repository. Retrieval plus on-demand reads is the recommended default.
Assume 30000 lines at 35 characters per line and 3.5 characters per token as an example rule. Assume the 3-units-per-million example price from the previous lesson.
size = 30000 lines x 35 chars = 1050000 chars
tokens = 1050000 / 3.5 = 300000 (example rule, tokenizer-dependent)
full read = 0.3 million x 3 = 0.9 units; 100 reads = 90 units/month
targeted = 100 questions x 8000 tokens = 800000 = 0.8 million; 0.8 x 3 = 2.4 units/month
ratio = 90 / 2.4 = 37.5Result: targeted reads cost about 2.4 units against 90 for full reads. Recommendation: index once, retrieve per question, and paste only the subsystem under review.
The index below is a FAKE scripted map in place of a real search index. It returns the files tagged with the query term.
The caller checks the returned total against the budget.
import sys
def retrieve(query, tags, sizes):
hits = []
for name in tags:
if query in tags[name]:
hits.append(name)
total = 0
for name in hits:
total = total + sizes[name]
return hits, total
def main():
sizes = {"top.v": 4200, "reset_sync.v": 5200, "fifo.v": 6100, "arb.v": 4300, "tb_top.v": 7400, "regs.md": 2100}
tags = {"top.v": ["reset_sync", "clk"], "reset_sync.v": ["reset_sync"], "fifo.v": ["fifo"], "arb.v": ["arb"], "tb_top.v": ["tb"], "regs.md": ["regs"]}
budget = 12000
hits, total = retrieve("reset_sync", tags, sizes)
if hits != ["top.v", "reset_sync.v"]:
print("FAIL: hit list is off")
sys.exit(1)
if total != 9400:
print("FAIL: retrieved tokens off")
sys.exit(1)
if total > budget:
print("FAIL: budget check is off")
sys.exit(1)
chars = 30000 * 35
if chars != 1050000:
print("FAIL: char count is off")
sys.exit(1)
tokens = chars * 2 // 7
if tokens != 300000:
print("FAIL: token estimate is off")
sys.exit(1)
full_mu = 100 * 300000 * 3 // 1000
tgt_mu = 100 * 8000 * 3 // 1000
if full_mu != 90000:
print("FAIL: full reads cost is off")
sys.exit(1)
if tgt_mu != 2400:
print("FAIL: targeted cost is off")
sys.exit(1)
if full_mu * 10 // tgt_mu != 375:
print("FAIL: cost ratio is off")
sys.exit(1)
print("PASS: 300000 token estimate with 9400 retrieved under a 12000 budget and 90 against 2.4 for 100 reads")
main()1,050,000 characters divided by 3.5 is 300,000 tokens under the stated example rule.
100 reads cost 90 units against 2.4 for the targeted flow. The ratio is 90 / 2.4 = 37.5.