Attention Sparsity Patterns in Small Transformers Trained on Low-Resource Languages

Abstract

We study how attention heads specialise when transformer models under 50M parameters are trained on corpora of fewer than 10 million tokens. Across five low-resource languages we find that induction heads emerge later and remain measurably sparser than in high-resource baselines, and that this sparsity predicts downstream transfer accuracy. We release our training code, checkpoints and evaluation harness.