C oncrete GPT : A Baby GPT -2 Based on Lexical Concreteness and Curriculum Learning (2024)

Abstract: We present a model for the Strict-Small track of the BabyLM Challenge 2024 (Choshen et al. 2024). We introduce a Curriculum Learning approach for training a specialized version of GPT-2 (Radford et al. 2019), that we name ConcreteGPT. We utilize the norms from (Brysbaert et al. 2014) which provide concreteness ratings for 40,000 English lexical items based on human subjects. Using these norms, we assign a concreteness score to each sentence in the training dataset and develop two curriculum strategies that progressively introduce more complex and abstract language patterns in the training data. Compared to the baselines, our best model shows lower performance on zero-shot tasks but demonstrates superior performance in fine-tuning tasks. Notably, our curriculum-trained models exhibit significant improvements over a non-curriculum based training of the same model.

## introduction optimising language model training to enhance efficiency without compromising performance presents a significant challenge, especially in the era of large language models (llms) which require trillions of input tokens and millions of petaflops for training @xcite . a promising approach lies in exploring training strategies that streamline the learning process and maximise resource utilisation. initiatives like the babylm challenge @xcite aim to find strategies to train effective llms under specific data constraints, that naturally reflect also on model sizes constraints following known scaling laws. one possible area of interest in this context is the use of training strategies related to curriculum learning, which refers to the idea of training machine learning models on meaningfully ordered data, for instance from easier to harder samples @xcite . this approach has yielded beneficial results on many tasks @xcite , but has not been widely adopted in the context of language modelling. typically, llms are trained on data scraped from the web, for which it is difficult to obtain a meaningful ordering. in the present work, we evaluate the hypothesis that a curriculum learning strategy informed by evidence from human language acquisition can enhance model performance in data-and/or computeconstrained settings. specifically, we attempt to understand the impact of considering word concreteness for ordering training data. in this context, concreteness refers to how tangible or perceptible the referent of a word is, with more concrete words being those that refer to physical objects or sensory experiences, while abstract words relate to concepts and ideas @xcite . word concreteness is often considered a proxy for the natural order in which children acquire language, beginning with words that represent familiar objects and situations @xcite . as language development progresses, children gradually learn terms that describe more complex concepts or relationships, which typically rely on the prior acquisition of simpler linguistic elements. understanding the impact of word concreteness on language model training could potentially lead to models that better grasp and generate language in a more nuanced manner, and more importantly, that learn faster and more efficiently. while some studies have explored different language complexity metrics for curriculum learning @xcite @xcite , none of the methods proposed in the 2023 babylm challange employed curriculum criteria related to lexical concreteness @xcite . in this work, we introduce a curriculum learning approach to train a specialised version of gpt2 @xcite , that we call concretegpt, which leverages word concreteness ratings. we exploit the concreteness norms from @xcite , which include concreteness ratings obtained from human subjects for 40,000 lexical items in english. using the norms, we compute a concreteness score for each sentence in the training dataset and create a curriculum that progressively emphasises more complex and abstract language patterns. we evaluate our approach on the strict-small track for the 2024 babylm challenge. for the track, participants are provided with a dataset of 10m tokens for pre-training their model. then, the model is evaluated in two ways: first, on a set of tasks in zero-shot settings, using perplexity (ppl) or pseudo-log-likelihood (pll) metrics as a proxy of model understanding; second, the model is fine-tuned using standard fine-tuning or lora on the glue benchmark tasks. we evaluate two different models that employ a slightly different approach to building the curricula for the training, and compare them with a baseline model trained with the same amount of flops without curriculum learning. we show that the curriculum-based models tend to outperform the non-curriculum model, while generally matching or slightly underperforming compared to the strong baselines provided by the task organizers (i.e., the winning models from the 2023 edition), despite a possibly lower computational cost. the paper is organised as follows. first, we outline the motivations behind our curriculum learning approach in section 2. section 3 details the methodology used to create the datasets and describe the curriculum design. sections 4 and 5 provide an in-depth discussion of the model, covering training specifics and results, and discuss the impact of two variations of the curriculum learning strategy. finally, section 6 draws some conclusions and highlights possible future directions. ## motivation behind curriculum design the motivation behind this approach stems from the hypothesis that a curriculum guided by word concreteness can enhance the model's learning trajectory by starting with more concrete, easily grasped examples and gradually advancing to more sophisticated verbal items. this method aims to improve the model's ability to handle a broader range of linguistic phenomena, potentially leading to more robust and contextually aware text generation. given that the model is not multimodal, one might initially question the value of using a curriculum learning approach based on lexical concreteness, since the representations learned by the model are not grounded in perceptual experiences. it is useful to start from the assumption that, in principle, all meanings can be considered as abstract, referring to general classes capable of subsuming heterogeneous and always particular phenomena @xcite . from this perspective, the value of an approach based on lexical concreteness does not lie in grounding meanings in perceptual experience @xcite . despite the abstract character of meaning, it is widely accepted that the first words children learn tend to have a tight connection to their experience with referents @xcite . early language acquisition typically involves words related to the child's surroundings, such as parents, pets, and daily routines objects. from these familiar meanings, children gradually expand their vocabulary to include words with more complex meanings that concern more abstract situations, require greater linguistic competence and larger cultural experience. thus, concreteness rating can be understood as an index of the difficulty in acquiring a word. for instance, learning a term like "dog" requires less linguistic knowledge and semantic structuring than understanding a more abstract concept like "justice". by initially exposing the model to sentences containing words with an high concreteness rating, we attempt to simulate this learning trajectory, providing the model with simpler, more fundamental contexts before progressing to the acquisition of more complex meanings and linguistic situations. the method proposed in this paper is consistent with the findings of @xcite and @xcite which suggest that lm embeddings encode perceptual structures (e.g, meaningful spatial relations and colors) without requiring perceptual grounding. the hypothesis is that a curriculum based on lexical concreteness can facilitate the acquisition of these meaningful structures. a potential objection to our method is why age of acquisition is not used directly as a feature for sorting the curriculum. the core principle of the curriculum learning approach is to establish a criterion that accurately assesses the difficulty of language items, thereby grouping items of similar difficulty together. in this context, a criterion such as age of acquisition alone does not serve this purpose effectively. linguistically similar itemsthose that function similarly in speech and possess comparable levels of difficulty -can exhibit different ages of acquisition. in fact, consulting data from wordbank @xcite , specifically the british oxford communicative development inventory (cdi), we observe that linguistically similar items are acquired at different ages by children. for example, the proportion of children understanding the word dog at 12 months is slightly over 0.6, whereas the corresponding value for the word lamb is just under 0.1. this disparity remains relatively constant until 25 months, even though lamb does not appear to present any particular challenges compared to dog. consequently, two words of similar difficulty may be sorted differently within the curriculum based solely on age of acquisition. in contrast, this issue is mitigated by the use of concreteness ratings, with lamb being rated at 4.97 and dog at 4.85 in @xcite . based on the dataset slicing, we devise two dif-ferent curriculum strategies: sequential -this strategy considers the slicing as-is, and the curriculum is based on their sequential ordering, from the most concrete to the most abstract. mixed -this strategy is more nuanced, and accounts for the fact that while sentence-level concreteness can be used as proxy for the natural order in which children acquire language, it is also likely that childrens will be exposed to more complex words as well. thus, starting from the original slices, we redistribute part of each slice into the other ones. specifically, each slice contains 50% of the data from the original slice, and 50% from the other three slices, in different proportions, to simulate an increasing percentage of progressively more abstract sentences in each slice. the exact proportions of sentences from each mixed slice are reported in figure 2 . ## model and training in our experiments, we use the gpt2 implementation from huggingface foot_1 as our base architecture, with its standard pretrained tokenizer. the model has 124m trainable parameters. to further limit the computational cost of training, we restrict the context length of the model to 128 tokens. this change is driven not only by concerns regarding computational resources, but also by theoretical considerations related to the development of working memory in humans, which appears to be limited during the early years of life @xcite . a reduced context length (though still larger than the number of tokens a child can process) better aligns with the cognitive plausibility criteria required by the challenge. we experiment with a hybrid training procedure where the model is sequentially trained on each slice of the dataset for three epochs with the same hyperparameters. note that we restart the training procedure each time, and that we randomized the batch sampling within the training slice. this means that data from each batch is randomly sampled (as customary for training lms) from the training slice. then, the resulting model is further trained on the entire dataset with a lower learning rate for an additional two epochs, again with random batch sampling. we follow this procedure for ## results and discussion the models are evaluated on two distinct sets of tasks: one requiring fine-tuning and the other performed in a zero-shot setting. fine-tuning was conducted using the script provided by the organisers (see section 4). the baselines are two models trained by the organizers, and inspired by the 2023 edition winning systems: ltg-bert (charpentier and samuel, 2023) and babyllama @xcite . for the fine-tuning task, models are fine-tuned on the glue benchmark tasks @xcite . for the zero-shot tasks, results are reported in on average, across all tasks, the three models consistently outperform the worse baseline (ltg-bert), but do not exceed the performance of the best baseline (babyllama) except for glue. of the three, the mixed model performs the closest to babyllama overall. however, this result is largely influenced by the poorer performance in blimp, especially in blimp supplement. in this case the curriculum learning strategy appears to negatively affect performance on acceptability tasks, as indicated by the weaker results observed in cola and blimp. a potential explanation is that curriculum learning based on lexical concreteness enhances performance in tasks with a stronger semantic component, such as mrpc, sst-2, and mnli, where the curriculum-trained models demonstrate superior performances. however, these findings appear to corroborate the idea that, given the same (and limited) amount of data and training compute, employing a cognitively plausible training strategy that leverages lexical concreteness as a proxy for a plausible or ## conclusion and future work in this paper we propose two models for the strict-small track of the babylm challenge 2024 @xcite . the models were trained using a curriculum learning strategy designed to optimise performance. the dataset provided by the organisers was divided into four slices based on increasing levels of lexical concreteness. from this division, two models were trained: the sequen-tial was trained on the slices in order of decreasing concreteness, while the mixed incorporated a progressively higher percentage of abstract and complex sentences at each epoch. for comparison, the same architecture was trained using a standard training procedure on the entire dataset with the same amount of compute (shuffle model). the shuffle model outperforms the curriculum-trained models only in the blimp supplement (for zero-shot) and in cola (for fine-tuned) tasks. in all other tasks however curriculum learning based on lexical concreteness, particularly the mixed model, demonstrates improved performance. compared to the baselines provided by the organisers, the mixed model exhibits comparable or lower performance on zero-shot tasks but performs well in fine-tuning tasks. these results are notable, especially given the relatively small amount of training compute provided to the model. our findings suggest that in low resources and/or low compute scenarios, cognitively plausible training strategies, specifically using concreteness, may help the model learn effective representation faster than with traditional training methods. nevertheless, we must point out that the proposed approach does not systematically outperform the strong baselines provided by the challenge organisers, especially in zero-shot tasks. possible explanations are that i.) our concreteness-based approach still requires some refinement, and that ii.) our models may be undertrained with respect to the baselines. based on these findings, we propose several directions for future work. first, training the model on a larger dataset and for more epochs would allow us to test whether the performance gap scales with additional data, potentially by further refining the progression of the slices in the mixed strategy. second, applying this curriculum learning approach to a multimodal model would help assess whether it also facilitates mapping between language and images. finally, it would be valuable to further investigate the differences in performance on acceptability tasks (which are more syntactic in nature) versus tasks focused on semantics and inference, to better understand the robustness of this trend.