2026-11-04T23:32:12Z https://edapress.org/oai

oai:edapress.org:eda-2026-0142 2026-06-24 cs

Attention Sparsity Patterns in Small Transformers Trained on Low-Resource Languages A. Mekonnen J. Park Computer Science transformers interpretability low-resource NLP We study how attention heads specialise when transformer models under 50M parameters are trained on corpora of fewer than 10 million tokens. Across five low-resource languages we find that induction heads emerge later and remain measurably sparser than in high-resource baselines, and that this sparsity predicts downstream transfer accuracy. We release our training code, checkpoints and evaluation harness. EDA 2026-06-24 Text info:eu-repo/semantics/article application/pdf https://doi.org/10.0000/eda.2026.0142 https://edapress.org/paper/eda-2026-0142 en CC BY 4.0