ELC - P arser BERT : Low-Resource Language Modeling Utilizing a Parser Network With ELC - BERT (2024)

Abstract: This paper investigates the effect of including a parser network, which produces syntactic heights and distances to perform unsupervised parsing, in the Every Layer Counts BERT (ELC-BERT) architecture trained on 10M tokens for the 2024 BabyLM challenge. The parser network’s inclusion in this setup shows little or no improvement over the ELC-BERT baseline for the BLiMP and GLUE evaluation, but, in particular domains of the EWoK evaluation framework, its inclusion shows promise for improvement and raises interesting questions about its effect on learning different concepts.

## introduction recent advancements in transformer-based language models, in particular large language models (llms), have largely been achieved by scaling the parameter count as well as the size of the dataset @xcite . whilst there is ongoing research in identifying efficient training and sampling methods for llm pre-training, @xcite project that between the year 2026 and 2032 the datasets for training llms will be equivalent to all extant human text data. in response to the staggering amount of data upon which llms are trained, the babylm challenge aims to incentive research in the development and pre-training of language models by setting realistic human-developmental limitations on the training data @xcite . in particular, the challenge has three data-limited tracks: two texts only tracks that restrict the data corpora sizes to 10m and 100m (strict-small and strict, respectively), the latter of which is inspired by approximately the amount of data a 13 year old child will have seen, and a vision-language track, combining text and images. the 2024 babylm challenge is the second iteration of this challenge. the overall best system from the first challenge was the every layer counts bert model (elc-bert) @xcite , which showed effective results by changing the residual connection between the transformer layers. although, in the first babylm challenge, systems with architectural modifications produced the best results, a plurality of submitted systems used curriculum learning, of which only one found significant gain from this approach @xcite . this paper introduces elc-parserbert, a model submitted for the strict-small track, which incorporates the parser network proposed in @xcite into elc-bert. the parser network is able to induce both dependency and constituency syntactic structures, an example of which can be seen in figure 1 , and the aim of its inclusion is to investigate whether this structural bias aids the baseline elc-bert model. this paper also investigates whether using a curriculum learning based approach with this model architecture yields any improvement. ## results the results of the first experiment can be seen in these domains, however, can then be further broken down to see which categories within the domains had significant differences by running mcnemar tests on the predictions for each group within a domain. in the "material properties" domain, ecl-parserbert predicted significantly better for the context type "direct" rather than "indirect," but it cannot be said that this is directly due to the inclusion of the structural bias in the model. it is more likely, however, that there are particular concepts that elc-parserbert understands better or worse than ltg-bert, although it may be possible that this is indirectly caused by the inclusion of the parser network in the model during pre-training. for instance, elc-parserbert gets all 20 of the instances correct where the context is "cold" or "warm," whereas ltg-bert only gets 4 of them right. the fourteen more that elc-parserbert predicted correctly all had a "direct" context type, and there is a similar case for the "fragile" and "sturdy" contexts. moreover, in the "quantitative properties" domain, elc-parserbert actually performs worse in the "direct" context type questions, but this similarly follows from poor performance in particular concepts such as "a lot of" versus "a little." the categories (concepts, context types, etc.) with significant differences in prediction between elc-parserbert and ltg-bert within the domains found to have significant differences, as seen domain name p-val accuracy ewok_physical-dynamics_filtered_results 0.028784 -0.12 ewok_social-properties_filtered_results 0.0049673 0.10 ewok_agent-properties_filtered_results 0.0044958 -0.02 ewok_material-properties_filtered_results 0.0011381 0.18 in ## conclusions and future work in the context of the babylm challenge 2024, this paper experimented with the elc-parserbert architecture, which is formed by adding the parser network from the structformer @xcite to the elc-bert architecture @xcite . there was no significant improvement found in the blimp, blimp supplemental, and (super)glue evaluation tasks through the inclusion of the parser network with the training as described. in the ewok evaluation framework, however, the elc-parserbert architecture showed comparable results to the ltg-bert-b model and improvement over the other baselines. this paper also examined the effectiveness of using the flesch reading ease @xcite metric to determine an ordering of the training data for curriculum learning for training the elc-parserbert architecture. the use of this particular learning curriculum on this training data with this architecture did not show any significant improvement generally, but the inclusion or exclusion of this learning curriculum did significantly alter the quality of predictions for certain concepts. investigating the cause of these particular concept affinities might be the focus for future work. future work may also seek to improve upon the elc-parserbert model by ensuring sentences are producing parse trees separately for each sentence in a context window, as is done in gpst @xcite . additionally, the elc-parserbert's largest shortcoming was in the (super)glue evaluation tasks, which employed the default hyperparameters for finetuning set by the organizers, so searching for more optimal hyperparameters may yield overall model improvement.