Every token looks back

At an intuitive level, each token can look at earlier tokens when the model picks the next one. A pronoun can check the noun it refers to; a signal name can check its declaration.

Each such look is one pairwise comparison. The total number of comparisons is what makes long inputs costly.

Counting the looks

Adding 1 + 2 + ... + n gives n(n + 1) / 2 comparisons. That closed form is exact for the basic causal form of attention.

1
2
3
4
5

causal pairs for n tokens, self included: n(n + 1) / 2
n = 8: 8 x 9 / 2 = 72 / 2 = 36
n = 1024: 1024 x 1025 / 2 = 1049600 / 2 = 524800
n = 2048: 2048 x 2049 / 2 = 4196352 / 2 = 2098176
double the length: 2098176 / 524800 = 3.998, just under 4

Why users feel the square

Doubling the input roughly quadruples the basic attention work. Longer prompts therefore cost more compute and more waiting, with exact figures varying by model and service.

Efficient attention variants exist that scale better than the basic form. They change the constant factors and sometimes the scaling, not the advice to keep inputs lean.

Common mistakes

Related topics

• When the window fills

• Long inputs and the lost middle

• Latency and throughput