How Much Do Modifications to Transformer Language Models Affect Their Ability to Learn Linguistic Knowledge? (2022)

Abstract: Recent progress in large pretrained language models (LMs) has led to a growth of analyses examining what kinds of linguistic knowledge are encoded by these models. Due to computational constraints, existing analyses are mostly conducted on publicly-released LM checkpoints, which makes it difficult to study how various factors during training affect the models’ acquisition of linguistic knowledge. In this paper, we train a suite of small-scale Transformer LMs that differ from each other with respect to architectural decisions (e.g., self-attention configuration) or training objectives (e.g., multi-tasking, focal loss). We evaluate these LMs on BLiMP, a targeted evaluation benchmark of multiple English linguistic phenomena. Our experiments show that while none of these modifications yields significant improvements on aggregate, changes to the loss function result in promising improvements on several subcategories (e.g., detecting adjunct islands, correctly scoping negative polarity items). We hope our work offers useful insights for future research into designing Transformer LMs that more effectively learn linguistic knowledge.

## introduction at the core of many natural language processing tasks are language models (lms), which compute the probability distribution of the next token that follows a given input context. the transformer @xcite , as one of the most popular architectures for language modeling, has been widely adopted for large-scale pre-training, such as in bert @xcite , gpt-2 @xcite and gpt-3 @xcite . the success of large-scale lm pretraining has propelled a surge of analysis on the linguistic knowledge encoded by language models. while prior works have uncovered many exciting facts regarding the linguistic capability of those pretrained lms @xcite @xcite , most of these analyses are conducted on publicly-released model checkpoints, and thus the impact of various lm training configurations remains relatively unexplored, limited to lstm lm configurations @xcite or varying training data size @xcite . in this work, we focus on transformer lms @xcite instead of lstms, and we investigate two aspects of lm training distinct from previous works -(1) the lm training objective, for which we experiment with the focal loss and multi-task training; and (2) the transformer's self-attention mechanism, which we restrict to a local window of tokens. we train a suite of transformer lms that minimally differ from each other in one of these two aspects, and evaluate the effect of these changes via non-parametric probing on blimp @xcite , a targeted evaluation benchmark of multiple english linguistic phenomena (e.g., island effects, anaphor agreement). experimental results demonstrate that none of these modifications yields significant gains on blimp in aggregate. however, we do observe that modified training objectives (e.g, using focal loss instead of standard cross entropy loss) result in improvements to specific subtypes of linguistic phenomena. overall, our experiments suggest that it could be promising to scale up transformer lms with modified training objectives, as they may help improve syntactic generalization. ## method language models compute p(w i | @xmath0, the probability distribution of the next token w i given the preceding context @xmath1. the conventional training objective of an lm is to minimize the surprisal of tokens in a training set. the surprisal of a single token can be expressed as the negative log probability of that token given the preceding context (prefix): while many models were proposed to compute p(w i | @xmath2, we focus on the transformer architecture @xcite , which consists of a stack of alternated self-attention and feed-forward blocks and has become the mainstream architecture for large-scale lm pretraining. unlike prior work, which has focused on fixed transformer language model checkpoints, we are curious to see how intervening in the training process would impact the resulting models. specifically, we ask: are there any training objectives or model design choices that would improve the models' acquisition of linguistic knowledge? ## results & analysis overall, we did not find a significant improvement on blimp after applying the aforementioned modifications. determiner noun agreement another notable change is within determiner noun agreement. this phenomenon tests whether a model recognizes in-correct noun after a determiner (e.g., "that tables" is unacceptable). the model trained with focal loss is better than the baseline model on multiple paradigms by large margins, especially on cases where adjective is inserted between the determiner and the noun. the accuracy of baseline model is improved from 81% to 88%. the second best modification is when the transformer is trained with local attention, which consistently outperforms the baseline for all but two paradigms. ## conclusion to complement recent analyses on the linguistic knowledge encoded by released transformer lm checkpoints, we investigate four transformer language models, each trained with slightly different settings. we evaluate these variants on blimp, a targeted evaluation set to probe the language models' capability of various linguistic phenomena. our results show that although the averaged performance is similar after applying those changes, there are promising gains on local paradigms. we hope our work could shed light on future research into more effective learning of syntactic knowledge by transformer language models.