B aby LM s for isi X hosa: Data-Efficient Language Modelling in a Low-Resource Context (2025)

Abstract: The BabyLM challenge called on participants to develop sample-efficient language models. Submissions were pretrained on a fixed English corpus, limited to the amount of words children are exposed to in development (<100m). The challenge produced new architectures for data-efficient language modelling, outperforming models trained on trillions of words. This is promising for low-resource languages, where available corpora are limited to much less than 100m words. In this paper, we explore the potential of BabyLMs for low-resource languages, using the isiXhosa language as a case study. We pretrain two BabyLM architectures, ELC-BERT and MLSM, on an isiXhosa corpus. They outperform a vanilla pretrained model on POS tagging and NER, achieving notable gains (+3.2 F1) for the latter. In some instances, the BabyLMs even outperform XLM-R. Our findings show that data-efficient models are viable for low-resource languages, but highlight the continued importance, and lack of, high-quality pretraining data. Finally, we visually analyse how BabyLM architectures encode isiXhosa.

## introduction large language models (llms) are trained on trillions of words @xcite . humans are much more efficient language learners -children are exposed to less than 100 million words of speech/text by age 13 @xcite . this mismatch motivated the establishment of the babylm challenge @xcite , a shared task in which participants were invited to propose data-efficient language modelling techniques. submissions were pretrained on a fixed corpus of developmentally plausible english (e.g. child-directed speech, educational content) and ranked according to performance on natural language understanding (nlu) benchmarks. ## experimental setup pretraining our babylms and baseline are pretrained on the wura isixhosa corpus @xcite , which is a compiled by filtering mc4 @xcite to remove noise. the isixhosa dataset contains 13m words, similar in size to babylm strict-small. our models are trained for 200 epochs on a tesla v100 gpu. we detail our training process in appendix a. evaluation we evaluate on three isixhosa datasets -masakhapos (dione et al., 2023) for pos tagging, masakhaner @xcite for ner, and masakhanews @xcite for news topic classification (ntc). test set results are averaged across 5 finetuning runs. ## results elc-bert outperforms one skyline, xlm-r, on two tasks. unlike in the shared task, our models do not outperform the top skylines. we attribute this to an important difference between our setup and the shared task -the quality of pretraining data. the wura corpus does not match the quality of the babylm data, which was curated to include developmentally plausible text (e.g. child-directed speech, educational content). the previous success of these models in english is due to a combination of modelling innovations and extremely high-quality pretraining data, which is lacking for low-resource languages like isixhosa. ## analysis the babylms studied in this paper achieve dataefficiency by augmenting the standard mlm architecture. we now analyse how their unique architectural innovations encode the isixhosa language. elc-bert the residual connections of elc-bert learn to selectively weigh the output of previous layers. we visualise learned weights in figure 2 , comparing early pretraining to complete pretraining (intermediary stages are visualised in figure 4 in the appendix). the weighting exhibits significant deviations from a standard transformer layer (which assigns equal weight to all preceding outputs). in early stages of pretraining, the model is biased to the embedding layer and immediately preceding layers. as pretraining progresses, the model reduces its reliance on embeddings in favour of immediately preceding layers, but still assigns more weight to the embedding layer than the babylm elc-bert submission. we posit that this emphasis on the embedding layer underlies elc-bert's performance gains on pos tagging and ner, since embeddings encode information about word-level syntactic roles @xcite . mlsm during pretraining, mlsm predicts the latent semantic categories of masked tokens. to inspect the semantic distribution learned by the model, we extract the predictions for masked named entities in sentences sampled from masakhaner. figure 3 shows the top 10 semantic categories (each corresponding to an index) assigned to four named entities. for each target word, we also plot the probabilities produced for the other words to compare distributions. in our sampled sentences, two of the words (justin and augustine) are names of persons, while the other two (morocco and soweto) are names of locations. the plots demonstrate more semantic overlap between the same types of named entities. the names of persons have seven overlapping semantic categories, while the names of locations have nine overlapping categories. between the two named entity types, only two semantic categories overlap. this pattern indicates that mlsm effectively encodes the semantic properties of these named entities, to which we attribute its ner performance gains. we present a similar analysis for target words across pos tags in appendix b. ## conclusion this study explored the potential of two architectures from the babylm challenge, elc-bert and mlsm, to benefit low-resource languages. comparing our findings to those of the babylm challenge, we draw three main conclusions. firstly, the gains obtained by isixhosa babylms show that the sample-efficiency sought by the babylm challenge can prove effective in real low-resource settings. secondly, elc-bert once again emerges as the most data-efficient solution, even outperforming massively multilingual plms. lastly, the fact that our babylms do not outperform all skylines shows that the absence of high-quality corpora for isixhosa poses a barrier to further gains. the findings of the babylm challenge can be attributed to both architectural innovations and specifically curated pretraining data. the babylm pretraining corpus includes child-directed speech, educational video subtitles, and articles from simple wikipedia (an edition of wikipedia written in simplified english, using shorter sentences and common words). such high-quality, developmentally plausible data is not publicly available for isixhosa. our results show that this limits the potential of babylms for low-resource languages. more generally, this work unites two directions of research -cognitively plausible modelling and nlp for low-resource languages. we hope more researchers pursue work at the intersection of these two subfields, since they share the goal of improving data-efficiency in the era of scaling.