What attention heads actually attend to

I sorted every head in tern-370m by the pattern it produces on held-out text. Most fall into a handful of simple types.

Findings

About a third of heads attend mostly to the previous token. A smaller group copies earlier tokens, and roughly one in ten could be removed with no measurable change in loss.