What reuse means

A hosted service can reuse work for a prompt opening it has seen before. The opening must stay identical across calls, since reuse is keyed on exact repetition.

Whether and how much this saves varies by model and service, so measure it. The concept below holds wherever repeat billings exist: identical input need not be processed twice.

A worked saving

Assume a service counts repeated prefix tokens at 1 unit per million and new tokens at 3 units per million. These are stated EXAMPLE rates, not any real price.

1
2
3
4
5
6
7
8
9

prefix 6000 tokens, fresh suffix 500 tokens, 10 calls, identical prefix
without reuse: 10 x (6000 + 500) = 65000 tokens
cost = 65000 x 3 / 1000000 = 195000 / 1000000 = 0.195 units
with reuse: first call 6500 new tokens + 9 suffixes of 500
new tokens = 6500 + 9 x 500 = 6500 + 4500 = 11000
new cost = 11000 x 3 / 1000000 = 33000 / 1000000 = 0.033 units
repeated cost = 9 x 6000 x 1 / 1000000 = 54000 / 1000000 = 0.054 units
total = 0.033 + 0.054 = 0.087 units
saving = (0.195 - 0.087) / 0.195 = 0.108 / 0.195 = 0.5538... = about 55%

What breaks reuse

Reuse breaks at the first changed token of the opening. Edits, reordering and even whitespace changes can move everything after them out of the reusable region.

• Appending new material at the end preserves the longest reusable opening.

• Editing the middle restarts reuse from the edit point onward.

• Timestamps and counters in the opening defeat reuse on every call.

Common mistakes

Related topics

• Summarisation and information loss

• Cost arithmetic

• Latency and throughput