L ong T ail-Swap: benchmarking language models’ abilities on rare words (2025)

Abstract: Children learn to speak with a low amount of data and can be taught new words on a few-shot basis, making them particularly data-efficient learners. The BabyLM challenge aims at exploring language model (LM) training in the low-data regime but uses metrics that concentrate on the head of the word distribution. Here, we introduce LongTail-Swap (LT-Swap), a benchmark that focuses on the tail of the distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. LT-Swap is a pretraining corpus-specific test set of acceptable versus unacceptable sentence pairs that isolate semantic and syntactic usage of rare words. Models are evaluated in a zero-shot fashion by computing the average log probabilities over the two members of each pair.We built two such test sets associated with the 10M words and 100M words BabyLM training sets, respectively, and evaluated 16 models from the BabyLM leaderboard. Our results not only highlight the poor performance of language models on rare words but also reveal that performance differences across LM architectures are much more pronounced in the long tail than in the head. This offers new insights into which architectures are better at handling rare word generalization. We’ve also made the code publicly available on GitHub, enabling the generation of LT-Swap benchmarks based on any English text corpus.

## introduction the most recent efforts in nlp have focused on large language models (llms) trained on internet-scale datasets @xcite @xcite . these datasets are composed of trillions of words, which is orders of magnitude above what humans hear and read in their lifetime @xcite yet llm intelligence still has significant 1 https://github.com/facebookresearch/lt-swap blind spots compared to humans @xcite . as an answer to this problem, a recent research effort, called the babylm challenge @xcite @xcite , has stimulated interest in the building of data-efficient language models (lm) pre-trained in human-scale datasets. in this work, we brought our attention to the ability of lms to learn words in a few shots, like infants when they learn to speak @xcite . this is a notoriously difficult task because of the skewed distribution of words which features a long-tail of very rare word types, a phenomenon known as zipf's law @xcite . those long-tail words are ubiquitous in language @xcite and are known to pose problems for language models training at all scales. for example, @xcite shows a large performance decline when lms are asked questions about rare named entities. @xcite has unveiled a mode collapse when lms are recursively trained on their own output. more specific to our problem, @xcite has pointed out that lm generations trim the long-tail distribution of words by over-representing frequent words. however, even though long-tail words are notoriously hard to learn, the benchmarks used in the babylm challenge (blimp @xcite , ewok @xcite and superglue @xcite ) focus on the other end of the distribution, avoiding most of the long-tail words (see appendix e for more details). here, we wish to correct this blind spot and build a metric that specifically targets long-tail words. addressing the difficulty of learning words from the long-tail, we introduce a framework to automatically create a dataset-dependent evaluation task that we called longtail-swap (or lt-swap). lt-swap measures the syntactic and semantic abilities of pretrained lms when probed on long-tail words extracted from their pretraining dataset. our framework is applicable to any text corpus used for lm pretraining and is scalable to llm corpus. inspired by the blimp @xcite benchmark, lms' performance on lt-swap is measured by their ability to discriminate between two sentences, one of which is making an incorrect use of a longtail word. we leverage the instruction-following capabilities of recent llms @xcite to generate sentence pairs containing the words of interest. these words range from frequent words, seen more than 512 times, down to words seen only once or even harder: never seen inflections of words present in the pretraining set. lt-swap can be divided into three subtasks: wordswap, a semantic metric, inflectionswap, a pos tagging metric, and agreementswap , a syntactic metric. using the babylm datasets, containing, respectively, 10m and 100m words, we automatically create two lt-swap tasks: lt-swap10m and lt-swap100m and use those two tasks to evaluate 16 different lms pretrained on the babylm datasets. the main takeaways of our analysis are the following: the frequency bin 0 serves for inflections never seen in the pretraining set. from typical work on word counts from psycholinguistics @xcite @xcite , long tail word is a loose term that refers to words below 10ppm 5 that represent an absolute frequency count up to 100 for a corpus of 10m words. our choice of frequency bins figure 1 : overview of our framework to create the dataset-dependant lt-swap tasks. the candidate lm is evaluated on sentences, generated by llm, that contain long-tail words from its pretraining set. s1* and s2* are obtained by swapping the long-tail words from s1 and s2, thereby creating incorrect sentences. a last llm filtering step assert that s1* and s2* are indeed less likely than s1 and s2. after this preprocessing step, we ask an llm to generate sentences that are example usages for each long-tail word or inflection. the generated sentences are paired together. for each pair of sentences, for example, s1 and s2, we create s1* and s2* by swapping long-tail words. s1 and s2 are meant to be two correct usages of two long-tail words, whereas s1* and s2* are two incorrect sentences. inspired by @xcite , pretrained lms are evaluated using their own confidence to discriminate the correct from the incorrect sentences. we make sure that the task is feasible with a last llm filtering step that rejects pairs of sentences that cannot be discriminated from their incorrect versions. the llm used in this work for both the generation and filtering steps is llama3.1-405b. the way sentences are generated, paired together, and filtered depends on the lt-swap subtasks: wordswap, inflectionswap, agreementswap. the rest of this section explains how those three subtasks are created (see example generations in the appendix the final lt-swap score is the accuracy score on the number of times the candidate lm correctly discriminated a correct sentence from an incorrect one. more specifically, we compute an accuracy for each frequency bin and each subtask (wordswap, inflectionswap and agreementswap) and then average all those accuracies together to form the final lt-swap score. ## results in this section, we present the scores obtained by the selected range of language models on our tasks, along with a detailed analysis of the results. ## conclusion in this paper, we introduced a framework for generating lt-swap, a set of tasks based on long-tail words from a pretraining dataset. our benchmark highlights the need to study the performances of lms on the long-tail as it unveils sharper differences across architectures. in addition, our benchmark shows that increasing the number of words during improve lms encoding of long-tail words. finally, our prefix method hints at a potential direction to improve lms on long-tail generalizations.