A surprisal oracle for when every layer counts (2024)

Abstract: Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner-directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively-and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https://github.com/asayeed/ActiveBaby.

## introduction in this work, we describe our contribution to the "strict-small" task of the babylm challenge of 2024 @xcite which follows up our contribution to babylm 2023 @xcite . our effort this year focused on two activities: (1) testing the most successful contribution to babylm 2023, every layer counts bert (elc-bert; charpentier and @xcite under additional conditions and (2) implementing our training protocol, which we called active curriculum language modeling (aclm) over our attempt at replicating elc-bert. we test elc-bert under more constrained conditions and explore whether the result is stable under other hyperparameter settings. under very similar settings, we test our aclm approach to see whether it exceeds the performance of our baselines on the babylm evaluation tasks. our intuition is that a human learner is an active participant in the environment of language acquisition @xcite . that children are not passive participants in l1 acquistion goes essentially without saying in contemporary developmental psycholinguistics-and with anyone who has interacted with a small child for any length of time-but the best artificial learners under data-constrained conditions (such as elc-bert) are trained in an entirely passive way. their higher performance stems entirely from technical adjustments to the "training math". while this results in impressive performance, the insights it can give to the "whole picture" of how children acquire language from small data is limited, given what we know already about human development. instead, our over-arching hypothesis is that for every successful "passive" language modeling training algorithm, there is a way of scheduling the learning process that is more cognitively plausible or better-performing or both. this is not straightforwardly "classical" curriculum learning with the curriculum calculated or set in advance. rather, it takes its inspiration from active learning @xcite , where the learner (usually in a classification task) assesses its uncertainty on hitherto unseen items, and then asks for a human label, in a process that reduces the burden of labelling more training data than there are resources to label. aclm instead uses a cycle in which the learner trains an initial model from a small subset of the examples, and then iteratively adds to its dataset by using an uncertainty criterion over the items automatically, essentially creating a "dynamic" curriculum during the learning process @xcite . the outcome of the overall babylm 2023 task participation @xcite suggested that curriculum learning was not fruitful in exceeding the original baselines or in overall competitiveness on the babylm task as compared to model archi- initialization examples training set trigram surprisal similarity model initial roberta model pool (corpus) examples examples iteration candidate examples pool (corpus) trigram surprisal similarity model roberta model examples training set selected examples (removed from pool) least certain example surprisal vectors nearest-neighbour vectors tecture "tweaks" such as elc-bert. the present study suggests a more mixed picture: that the advantages from architectural modifications are highly sensitive to peturbations from hyperparameters, while a dynamically updated curriculum such as aclm still may have the potential to augment a high-performing model architecture while retaining some connection to the interactive nature of human language acquisition. ## background the babylm task necessarily involves the exploration of a very large solution space. in the previous year's challenge, we proposed an initial system, which we depict in fig. 1 . because we had to start somewhere, this involved design decisions based on educated guesses as well as a focus on efficiency and "getting it off the ground". the result of that effort was that our system ended in the "middle of the pack" and behind a baseline bert model in the actual competition, but nevertheless resulted in insights that led us to consider how we can continue to explore this part of the babylm solution space, considering our expectations from a cognitive perspective. our previous system started with a blank roberta model @xcite which was initialized by training on a small subset of the training corpus. the remainder of the training corpus (the "pool") was processed via a trigram model into sequences of per-token trigram surprisal values, where suprisal is defined as the negative log-probability of the trigram ending in the given token. these sequences were resampled into an arbitrarily-chosen seven dimensions (which we later found was the average token length of the samples in the corpus), which we call the "surprisal signature". at each iteration of training (several epochs), the roberta model is queried about every sentence in the current training set, starting from the initial subset: which previously-seen utterance had the highest average surprisal. then a k-nearestneighbours process was used to sample the most similar surprisal signatures to the signature of the least certain sentence. these are added to the active training set, and the next iteration commences. this process is different from the active curriculum learning process of @xcite . jafarpour et al. develop a way to combine a human expert-designed curriculum with an activelearning informativeness criterion in order to select instances for humans to label. since our training task is a language modeling task rather than a labeling or classification task, we can eliminate the human in the loop (effectively, the label is a word we already have in the text) and use the informativeness criterion to structure an automatic curriculum. one obvious weakness of this process is that the surprisal space is static and does not reflect changes in the learner's estimation of what is surprising with respect to what is being learned. there are other weaknesses of this process, such as the seeming arbitrariness of the seven-dimensional vectors or even the use of surprisal as the criterion itself. however, in this update of our previous work, besides replacing roberta with elc-bert, we focus algorithm 1 initialization phase of this year's aclm process. model ← new(elc-bert) activeset ← select_random(pool, n_initial) train(model, activeset, n_epochs) surprisalset ← [] for all instances i in pool do surprisals ← model.surprisals(i) surprisalset.append(surprisals) end for on the first weakness and implement a dynamic process as described below. ## learner-directed active curriculum language modeling in the original formulation of aclm, the "surprisal space" (the collection of "surprisal signature" vectors), was static throughout training, leading to a curriculum directed entirely by the model's uncertainty over each active learning iteration. from a cognitive perspective, this would be equivalent to a human learner whose expectations about the most educational thing in the environment never change from birth. we now propose an update to aclm: we generate a new suprisal space at every iteration, which more closely matches the idea that a learner changes its view of the learning environment as it learns. in practical terms, this means that our aclm model now re-evaluates the suprisal space using the elc-bert model itself, producing a new surprisal space reflecting the model's current "knowledge state". we view this as increasing the cognitive realism of aclm and making it reflect a more learner-directed approach to acquisition. we describe the aclm procedure in this year's submission at a high level. we refer to "iterations" of the training procedure to be periods between updates of the active training set from the pool (corpus of as-yet-unseen training items). multiple training epochs can take place during an iteration, meaning that the model may see the same training set without an update multiple times. algorithms 1 and 2 provide an overview of the process, with the former describing the initialization process and the latter the iterative curriculum adaptation. the biggest procedural difference between this and @xcite is the use of the model itself to update the surprisals at every iteration. in our previous submission, we split the corpus by utterance. this year, we follow the practice algorithm 2 iterations of the aclm process. the knn procedure also removes the instances from the pool. for iter ← 0 to n_iterations do max_surprised ← trainingset[0] for all instances i in trainingset do orig_surprisal ← model.surprisals(max_surprised) new_surprisal ← model.surprisals(i) if @xmath0.update(surprisalset.knn( max_surprised, k, pool)) train(model, activeset, n_epochs) surprisalset ← [] for all instances i in pool do surprisals ← model.surprisals(i) surprisalset.append(surprisals) end for end for ## analysis we list our results in the original elc-bert still vastly outperforms both the babylm 2024 strict-small baselines as well as all of our models on blimp and glue. we will not attempt to explain ourselves why the 2024 baselines underperform the 2023 elc-bert submission 2 and focus our discussion on this year's baselines. on blimp @xcite , which contains inferences over very fine-grained grammatical details (e.g., anaphor agreement, island phenomena), our non-aclm models do relatively poorly compared to ltg-bert and babyllama baselines. as the main difference is batch size, it is hard to speculate on any deeper reason for the lower performance. this is essentially a candidate for an "unprincipled" hyperparameter search, as it is hard to imagine what batch size specifically has to do with grammatical phenomena. our aclm models outperform our non-aclm models slightly, but which aclm models do best is not consistent over the supplement or the filtered portion of blimp. however, the overall consistency of outperformance of aclm on the filtered blimp suggests that aclm is having an effect. on ewok @xcite , which is a dataset of inferences over world knowledge, we have a completely different story. our small batchsize non-aclm elc-bert does far better than either babyllama or ltg-bert. our aclm runs do even better than our non-aclm elc-bert runs. there is no strong difference between any degree of dimensionality reduction for the surprisal space. on glue @xcite , our non-aclm elc-bert models are in the same range as the ltg-bert and babyllama baselines. however, our aclm runs are all superior to any model but the original elc-bert. we do not see any potential for speculation on the performance differences for the surprisal space dimension. ## conclusions and future work for grammatically fined-grained inference tasks, our blimp results show that we underperform all models including the baseline, even without aclm, which we would expect to be similar to the baselines or the original elc-bert. we can 2 one reviewer suggests that this may partly be the result of a switch in averaging procedure in the evaluation pipeline provided by the task organizers. the batch size difference seemed to have a major effect on ewok and no effect on glue for non-aclm models. the simplest explanation is that elc-bert is simply very sensitive to hyperparameters. to investigate this further, we plan to conduct a hyperparameter study, in particular considering that some of the differences between models are rather small. for example, tuning the learning rate to batch size could be an avenue for optimization, though this has yet to be explored. however, we can contextualize the batch size effect in terms of the performance of our aclm training regimen. our aclm models were trained under conditions similar to our elc-bert runs. consequently, we did not expect them to actually exceed the ltg-bert baseline on blimp. we found this to be true, again possibly reflecting the batch size dependence of the task. but we saw consistent improvements on ewok and glue over both our elc-bert-only runs and the ltg-bert baseline. these improvement were independent of the dimensionality of the surprisal space, but, in hindsight, this is unsurprising because the input length was already uniform. in our entry from last year, we found that reversing the surprisal criterion (effectively choosing the least surprising candidates from the pool) caused a significant delay in result convergence, suggesting that this criterion has an effect, even if we did not have the right conditions to cause it to exceed a baseline bert model in performance. we find yet again tantalizing evidence that there are conditions under which controlling the order of learning matters. ewok is a world-knowledge-oriented dataset. we speculate that our learner-directed process may approximate an order that reflects cognitive dependencies among the human tasks-that is, the learner "fine-tunes" successively on increasingly "complex" tasks. exploring this requires direct inspection of what learning order is actually chosen by aclm and empirical investigation in to whether these orders might reflect developmental needs. a similar explanation may apply to the consistently higher performance of our aclm runs on glue. glue contains common-sense reasoning entailments, and this may reflect an implicitly preferable learning order that our surprisal criterion is finding. we emerge from this task optimistic about aclm as a way of exploring learner-directed strategies for simulating language acquisition through training large language models. there is still a huge methodological space to explore as well as many potentially relevant hyperparameters. for efficiency and comparability reasons, we adopted elc-bert's sentence-independent uniform input length, which likely nullified the effect of varying surprisal space dimensions. however, we believe that sentence length ought to have an effect on the learner's choices in what to focus on next. in the case of varying sentence length, the method of reduction to a uniform space would likely therefore matter and be an appropriate target of future work. we have also focused on surprisal as the measure that steers the interactive learner, but we find it unlikely that a single measure would represent the totality of optimal behaviours. therefore, another direction for future work would be testing other measures or combinations thereof.