Tokens and Tokenization · ai-skills

Just Enough AI · Beginner to Intermediate

5topics
5lessons

A model reads text as tokens, which are subword pieces cut by a fixed splitting rule. Token counts govern cost and fit, so this topic teaches counting, splitting and estimation.

Unit	Meaning
Character	One text character. It is stable across tools, but a model does not consume it directly.
Word	A whitespace-separated span. It is a rough human unit, not a model unit.
Token	One subword piece the model consumes. Its boundaries vary by model, so measure them.

Text to tokens to model.

Text is cut into tokens by a fixed rule before a model reads it (schematic).

Lesson	What it covers
Tokens, words and characters	Three units compared, with counts on short fixed examples.
Subword tokens and byte-pair encoding	The merge idea plus a small runnable trainer.
Why some text costs more tokens	Code, identifiers, numbers and other scripts split shorter.
Approximate rules of thumb	Chars-per-token ranges, stated as approximate and variable.
Estimating tokens before sending	Count characters, divide by a stated rate, add margin.

Check yourself A prose file holds 12000 characters. Why is its token count still only approximate before measuring?

Chars per token is approximate and tokenizer-dependent. A quotient is a range until measured on the actual splitter, so the count varies by model.

Up nextTokens, Words and Characters

Practice this lesson Report an error