Critical Size Hypothesis: How Model Hyperparameters Correlate with Its Linguistic Abilities (2024)

Abstract: In recent years, the models were tested on different probing tasks to examine their language knowledge. However, few researchers explored the very process of models’ language acquisition. Nevertheless, the analysis of language acquisition during training could shed light on the model parameters that help to acquire the language faster. In this work, we experiment with model hyperparameters and reveal that the hidden size is the most essential factor for model language acquisition.

## introduction modern deep learning models have achieved significant results in the field of language modeling and text generation @xcite . therefore, language models (lms) are often used in linguistic research to find systematic similarities in the language data. performance of the state-of-the-art models, such as transformer-based ones @xcite , on linguistic tasks show that they have learned measurable language structures during the training process @xcite . consequently, it is interesting to explore how the lms acquire the language during their training process and what part of their architecture helps to acquire a language better. in this work, we study the correlation between the acquisition process in the bert model and different model sizes. linguistic tasks are meant to represent three levels of language grammar structure: morphology, syntax, and discourse. in other words, we pose the following questions: which parameters of models influence the language acquisition process? ## related work the first work on probing of neural networks across time was carried by @xcite . the authors showed that first, a lstm model @xcite acquires syntactic and semantic features and later information structure. @xcite looked at the training process of albert @xcite and concluded that semantic and syntactic information is acquired during the early steps while accuracy on world knowledge fluctuates during the training. @xcite showed similar results on roberta @xcite : the model shows good results on linguistic probing tasks starting from early stages, and later it learns factual and common sense knowledge. @xcite studied training dynamics of multilingual models, they reveal that while linguistic information is acquired early, transfer learning abilities are evolving during the entire training process. @xcite examined the trajectories of models' language acquisition, and they find no impact of architecture or a model size on training trajectories. @xcite provides survey and theoretical discussions on how neural networks can help us learn more about language acquisition. following one of the ideas we conduct an ablation study of model's hyperparameters. ## results and discussion first, we conducted the same experiments on four small models described in section 3.1. as figure 1 shows, compared to multibert, models show worse accuracy. however, among small models, the one with the increased hidden size shows the best results in all cases, except for penn discourse treebank and tree depth, where the model with the increased number of layers shows the best results. this model shows the second best results on other tasks. the behaviour of the model with the increased number of attention heads is inconsistent compared to the base model (hidden size of 128, 2 attention heads, and 2 layers). on some tasks, such as penn discourse treebank and discourse coherence, it shows worse accuracy than the base model. on other tasks, it shows better quality than the base model but worse than other models. nevertheless, these observations are not applicable to the tasks from blimp. as charts show, on tasks, such as passive and principle a, the base model shows better quality than any other models, including the multibert model. at the same time we see that small models encounter difficulties with the acceptability of sentences with transitive verbs and with islands. the described above leads to the conclusion that bigger models are more successful in language acquisition. different parameters of model size give different level of improvement. thus, the most important parameter for language acquisition is level task model size morphology subject number 768/8 morphology person 128/2 morphology passive 512/4 morphology transitive 512/8 syntax top constituents 768/8 syntax tree depth 768/8 syntax adjunct island 768/8 syntax principle a 512/4 discourse discourse coherence 128/2 discourse connectors 768/8 discourse sentence position 512/4 discourse penn treebank 128/4 the results of our experiments with model sizes show that the increase of hidden size has the biggest impact on the quality of models. the number of layers was the second important parameter and improved quality better than the number of attention heads. our results are similar to the results reported in @xcite : they show that larger hidden size tend to improve quality. the hidden size might be important for smaller models because different layers code different information. for example, @xcite summarise that the first layers are task-invariant and contain general linguistic information while the latest layers are usually task-specific. on the contrary, attention heads are usually more detailed, for example, they are known to remember specific syntactic patterns @xcite . @xcite reveal that attention heads learn the same patterns. therefore, when the resources to encode information are limited, attention heads do not add much new information. regarding the hidden size, our results are different from the results in @xcite . while they postulate that number of layers is the most essential parameter, our results show that hidden size is better for performance improvement. since on some tasks small models did not reach the level of a base model, we train more models but following the results we achieved in our experiments with model parameters, we limit our experiments to hidden size and number of layers leaving behind the number of attention heads as an insignificant factor. these experiments summarised in for most blimp tasks the level of the base models is achieved by models of hidden size of 518 with 4 or 8 layers, which is a smaller size than for other 'morphosyntactic' probing tasks. the results of the experiments prove that increasing hidden size shows better results than increasing number of layers. moreover, models with the hidden size of 768 and 8 layers show results close to the model with the same hidden size and 12 layers. therefore, we conclude that hidden size is the crucial parameter for language acquisition. ## conclusion this works addresses the problem of language acquisition in state-of-the-art models and answers which factors influence the language acquisition process. to display correlation between language acquisition and different model parameters, we trained four models: one with the minimal hidden size and minimal number of layers and attention heads and three models with one parameter increased and others frozen. these experiments reveal that hidden size appears to be the most essential parameter for language acquisition, whereas attention heads do not significantly increase a model's performance. finally, we compared all tasks with the size of a model that shows the quality comparable with the base model used before. the idea behind this comparison is to find any correlation between different language levels and probing measures. as a result, models distinguish discourse from morphology and syntax but there is almost no difference between 'morphological' and 'syntactic' tasks. yian zhang, alex warstadt, haau-sing li, and samuel r bowman. 2020. when do you need billions of words of pretraining data? arxiv preprint arxiv:2011.04946.