Scaling laws below 100M parameters

Scaling laws are usually fit on large models. I trained forty small ones to see whether the same curves hold at the low end.

Where it bends

Below roughly 5M parameters the fit drifts from a clean power law. Embedding size starts to dominate, and the usual compute-optimal ratios stop applying.