A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in A merican E nglish (2020)

Abstract: Transformer-based language models achieve high performance on various tasks, but we still lack understanding of the kind of linguistic knowledge they learn and rely on. We evaluate three models (BERT, RoBERTa, and ALBERT), testing their grammatical and semantic knowledge by sentence-level probing, diagnostic cases, and masked prediction tasks. We focus on relative clauses (in American English) as a complex phenomenon needing contextual information and antecedent identification to be resolved. Based on a naturalistic dataset, probing shows that all three models indeed capture linguistic knowledge about grammaticality, achieving high performance. Evaluation on diagnostic cases and masked prediction tasks considering fine-grained linguistic knowledge, however, shows pronounced model-specific weaknesses especially on semantic knowledge, strongly impacting models’ performance. Our results highlight the importance of (a)model comparison in evaluation task and (b) building up claims of model performance and the linguistic knowledge they capture beyond purely probing-based evaluations.

## introduction endeavors to better understand transformer-based masked language models (mlms), such as bert, are ever growing since their introduction in 2017 (cf. @xcite for an overview). while the bertology movement has enhanced our knowledge on the reasons behind bert's performance in various ways, still plenty remains unanswered. less well studied and challenging are linguistic phenomena, where, besides contextual information, identification of an antecedent is needed, such as relative clauses (rcs). @xcite , e.g., analyzed bert's comprehension of function words, showing how relativizers and prepositions are quite challenging for bert. similarly, @xcite find rcs to be difficult for bert in the cola acceptability tasks. in this paper, we focus on rcs in american english to further enhance our understanding of the grammatical and semantic knowledge captured by pre-trained mlms, evaluating three models: bert, roberta, and albert. for our analysis, we train probing classifiers, consider each models' performance on diagnostic cases, and test predictions in a masked language modeling task on selected semantic and grammatical constraints of rcs. rcs are clausal post-modifiers specifying a preceding noun phrase (antecedent) and are introduced by a relativizer (e.g., which). extensive corpus research @xcite found that the overall most common relativizers are that, which, and who. the relativizer occupies the subject or object position in a sentence (see examples (1-a) and (1-b)). in subject rcs, the relativizer is obligatory @xcite , while in object position omission is licensed (e.g., zero in example (1-b)). our aims are (1) to better understand whether sentence representations of pre-trained mlms capture grammaticality in the context of rcs, (2) test the generalization abilities and weaknesses of probing classifiers with complex diagnostic cases, and (3) test prediction of antecedents and relativizers in a masked task considering also linguistic constraints. from a linguistic perspective, we ask whether mlms correctly predict (a) grammatically plausible relativizers given certain types of antecedents (animate, inanimate) and vice versa grammatically plausible antecedents given certain relativizers (who vs. which/that), and (2) semantically plausible antecedents given certain relativizers considering the degree of specificity of predicted antecedents in comparison to target antecedents (e.g. boys as a more specific option than children in example (1)). moreover, we are interested in how these findings agree with probing results and investigate model specific behavior, evaluating and comparing the recent pre-trained mlms: bert, roberta, and albert. this is to our knowledge the first attempt comparing and analyzing performance of different transformer-based mlms in such detail, investigating grammatical and semantic knowledge beyond probing. our main contributions are the following: (1) the creation of a naturalistic dataset for probing, (2) a detailed model comparison of three recent pre-trained mlms, and (3) fine-grained linguistic analysis on grammatical and semantic knowledge. overall, we find that all three mlms show good performance on the probing task. further evaluation, however, reveals model-specific issues with wrong agreement (where roberta is strongest) and distance between antecedent-relativizer and relativizer-rc verb (on which bert and albert are better). considering linguistic knowledge, all models perform better on grammatical rather than semantic knowledge. out of the relativizers, which is hardest to predict. considering model-specific differences, bert outperforms the others in predicting the actual targets, while roberta captures best grammatical and semantic knowledge. albert performs worst overall. ## probing mlms representations for grammatical knowledge of rcs we train supervised probing classifiers (here: acceptability classifiers) to assess the linguistic knowledge contained in a model's representations, focusing on grammaticality. a model's grammaticality awareness should be reflected in the hidden representations produced by the model for that particular sentence. hence, by training a classifier on top of these representations, we should be able to discriminate between grammatical and ungrammatical sentences based on their sentence embeddings. 2 moreover, besides capturing knowledge on grammaticality, we also examine whether this knowledge becomes more or less separable in the representation, considering representations produced by different layers of a model. ## discussion and conclusion with our work we moved towards tackling some issues in the evaluation of the linguistic capabilities of pre-trained transformer-based masked language models as, e.g., proposed by @xcite . most prominently, we try to better understand how a strong performance on supervised probing tasks is reflected in the predictions of the language models. to do so, we create a dataset based on naturalistic (not artificially generated) data and perform an extensive evaluation of masked language predictions in the context of rcs. moreover, rather than considering one model only, we compare three models to investigate the extend to which findings for bert can be generalized to other transformer-based models. our probing results show a significant improvement of all three transformer-based models over the baselines for almost all layers, suggesting indeed encoding of linguistic knowledge relevant for grammaticality classification, as shown in previous work (e.g. @xcite ). performance is similar across models, with bert and roberta performing slightly better than albert, and while contextualization improves overall, we have shown it helps immensely when considering less frequent rc modification types such as @xmath0, for which uncontextualized baselines learn simple heuristics. evaluation on a diagnostic set, however, clearly reveals weaknesses of probing classifiers and model-specific behavior. roberta is quite confident in wrong agreement cases between antecedent and relativizer, while this is problematic for bert and albert. considering distance between relativizer and antecedent/rc verb, they clearly outperform roberta. based on these insights, we conclude that viewing probing results in isolation can lead to overestimating the linguistic capabilities of a model. our masked language modeling evaluation provided deeper insights into model-specific differences. we evaluated relativizer as well as antecedent prediction. overall, all models show better performance on grammatical than semantic knowledge (animacy and plausibility). regarding relativizer prediction, all models perform worst on the target word which (plausible, as it is the most versatile of the relativizers). comparing models, bert is best in predicting the actual targets, roberta outperforms the others in capturing grammatical and semantic knowledge, while albert performs worst overall. evaluation on semantic types of antecedents shows prediction of unrelated or not directly related antecedents, especially for albert in which rcs. interestingly, both bert and albert predict hierarchically more general antecedents in who rcs, while roberta is able to better capture specificity. we believe that more work in this direction will lead to (a) awareness of the complexity of linguistic knowledge that such models might have to capture, considering e.g. generalization tasks, and (b) provide enhancement in evaluation strategies on how to best capture linguistic knowledge, such as a combination of probing, diagnostic, and cloze tests, but also in developing best practices of evaluation.