Attention Sparsity Patterns in Small Transformers Trained on Low-Resource Languages
A. Mekonnen (High school), J. Park (Undergraduate)
We study how attention heads specialise when transformer models under 50M parameters are trained on corpora of fewer than 10 million tokens. Across five low-resource languages we find that induction heads emerge later an…