Probing for Referential Information in Language Models (2020)

Abstract: Language models keep track of complex information about the preceding context – including, e.g., syntactic relations in a sentence. We investigate whether they also capture information beneficial for resolving pronominal anaphora in English. We analyze two state of the art models with LSTM and Transformer architectures, via probe tasks and analysis on a coreference annotated corpus. The Transformer outperforms the LSTM in all analyses. Our results suggest that language models are more successful at learning grammatical constraints than they are at learning truly referential information, in the sense of capturing the fact that we use language to refer to entities in the world. However, we find traces of the latter aspect, too.

## introduction neural network-based language models (lms) have been shown to learn relevant properties of language without being explicitly trained for them. in particular, recent work suggests that they are able to capture syntactic relations to a large extent @xcite @xcite . in this paper, we extend this line of research to analyze whether they are able to capture referential aspects of language, focusing on anaphoric relations (pronoun-antecedent relations, as in she-yeping wang in figure 1 ). previous work, such as @xcite , @xcite and @xcite , showed that augmenting language models with a component that uses an objective based on entity or coreference information improves their performance at . . . he 1 was elected to be president of the people's republic of china, and chairman of the 2 central 2 military 2 commission 2 . yeping 3 wang 3 was born in shanghai in 1926. she 3 studied in shanghai foreign language college, and started working in 1949. for a long time, she 3 . . . language modeling. intuitively, in the example in figure 1 , understanding that the first she refers to yeping wang makes words related to studying or working more likely to follow than other kinds of words. that is, referential information helps language models do their task. the cited work includes explicit coreference guidance; however, since referential information is useful for language modeling, we expect language models to learn referential information even without explicit supervision. here we analyze to what extent this is the case. we carry out our analysis using probe tasks, or tasks that check whether certain information is encoded in a model @xcite @xcite . the reasoning is as follows: even if a linguistic property is encoded in the network, it is not necessarily directly accessible through the model output; therefore, we train a probe model to predict a feature of interest, in this case anaphoric coreference, given the model's hidden representations as input. we focus on the two main linguistic levels that are relevant for coreference: morphosyntax, with grammatical constraints such as the fact that pronouns agree in number and gender with their antecedents, and semantics -in particular reference, such as the fact that a pronoun refers to the same entity as its antecedent. our hypothesis is that language models will capture grammatical properties, but not semantic information. this hypothesis is based on the observation that morphosyntax is a formal property of language that is easier to induce from cooccurrence patterns. the fact that language refers to entities is not obvious from language alone @xcite , and lms use only textual input. instead, what we find is that, while it is true that language models are much better at grammar, they do show evidence of learning semanticoreferential information to some extent. our explanation for this unexpected, partially positive result is that, because the same entity underlies all its mentions, the contexts in which the mentions appear are coherent and distinct from those of mentions of other entities. for instance, in figure 1 , the second she mention gives additional information about yeping wang that is consistent with the information given in the previous sentence. this paper has two main contributions. the first is an analysis methodology to probe for referential information encoded in language models, on two linguistic levels (morphosyntax, semantics) and two kinds of context: local (around one paragraph of context), and global (document context). this methodology can be applied to any architecture. the second contribution is a deeper understanding of the referential capabilities of current language models, and of the differences between transformers and lstms. the transformer outperforms the lstm in all the analyses. for morphosyntax, the transformer and the lstm have the same behavior with a performance difference; instead, they show different behavior with regard to semantico-referential information. ## related work coreference and anaphora resolution @xcite are among the oldest topics in computational linguistics and have continued to receive a lot of attention in the last decade, as manifested by several shared tasks @xcite @xcite . in our analysis we use the ontonotes dataset @xcite , developed within the coreference resolution community. our probe tasks are related to coreference resolution; however, our goal is not to train a coreference system but to analyse whether language models extract features relevant for reference without explicit supervision. a recent line of work has focused on demonstrating that neural networks trained on language modeling, without any linguistic annotation, learn syntactic properties and relations such as agreement or filler-gap dependencies @xcite @xcite @xcite . this is typically done by analysing the predictions of lms on controlled sets of data. part of this research uses probe models (also known as diagnostic models) to analyse the information contained in their hidden representations @xcite @xcite @xcite , as we do here -applying it to referential information. there is less work on referential information than on syntactic properties such as subject-verb agreement. as for anaphoric reference, @xcite include a limited test using 904 sentences from ontonotes. their results suggest that lms are able to do unsupervised coreference resolution to a certain extent; our first probe task can be seen as an extended version of their task obtaining more specific insights. @xcite analyze the kind of information that lstm-based lms use to make decisions in within-sentence anaphora. they find a strong male bias encoded in the network's weights, while the information in the input word embeddings only plays a role in the case of feminine pronouns. we analyze anaphora in longer spans (60 tokens / whole document) and include also a transformer. the above work suggests that lms capture morphosyntactic facts about anaphora to a large extent. there is much less evidence that lms can capture a notion of entity, as that which nominal elements refer to, and that they are able to track entities across a discourse. @xcite show that lstm-based models have poor results on texts with a high presence of entities; paperno (2014) that they cannot predict the last word of text fragments that require a context of a whole passage (as opposed to the last sentence only), with data that mostly contain nominal elements. several models @xcite @xcite were developed as an augmentation of rnn lms to deal better with entities, with the implicit assumption that standard models do that poorly. @xcite achieved good results on an entity-linking task, but showed that the network was not acquiring entity representations. as for transformer-based architectures, recent research suggests that they give same or better contextualized representations in comparison with lstm language models, and that they better encapsulate syntactic information @xcite . on the other hand, @xcite show that big transformer model representations perform on par or even poorer than smaller lstms on tasks such as number agreement or coordination, and that, like lstms, they have the problem that agreement accuracy decreases as the subject becomes more distant from its verb. most recent work on analysis of linguistic phenomena in nns focuses on bert @xcite @xcite . in this paper we chose to use transformerxl @xcite as our transformer model, and not bert, for comparability: we wanted to compare the two most standard architectures for lms on as equal ground as possible, and the two chosen models, trans-formerxl and awd-lstm @xcite , share the same training objective and are trained on the same data, with comparable vocabularies. ## morphosyntactic factors to shed light into which morphosyntactic information lms encode that is useful for coreference, we train a simple anaphora resolution probe model using the hidden layers of lms as input. by the logic of probe tasks, if the probe model is successful then that means that the relevant information is encoded in the hidden states, and error analysis can provide insight into which kinds of information are available. ## semantic (referential) factors the language model clearly captures morphosyntactic (grammatical) properties that constrain anaphora resolution; in this section, we show that it struggles more with is the semantic (referential) aspect, but it still captures it to some extent. ## conclusion previous work has provided robust evidence that language models capture grammatical information without being explicitly trained to do so @xcite . in this paper, we have analyzed to what extent they learn referential aspects of language, focusing on anaphora. we have tested two models representative of the prevailing architectures (transformer, lstm), and our methodology can be extended to any other architecture. we find that the two models behave similarly, but the transformer performs consistently better (around 10% higher accuracy in the probe tasks). 8 future work should test other architectures, like cnn-based lms and lstms with attention, to provide additional insights into the linguistic capabilities of language models. as expected, our results show that language models capture morphosyntactic facts about anaphora: based on the information in the hidden layers, a simple linear transformation learns to link pronouns to other pronouns or noun phrases, and to do so largely respecting agreement constraints in gender and number. although it is much harder for models to induce a more global notion of entity (what we have called semantico-referential aspects), models seem to encode entity-specific information to some extent. models get confused when there are other mentions in the context, especially if they match in some morphosyntactic feature, but less than could be expected; and they show some limited ability to distinguish mentions that have the same form but are in different coreference chains, though hampered by their heavy recency bias. the recency bias affects lstms more, but is also found in transformers, consistent with previous work on syntax @xcite . our results thus suggest that language models are more successful at learning grammatical constraints than they are at learning truly referential information, in the sense of capturing the fact that we use language to refer to entities in the world; however, they still do surprisingly well at referential aspects, given that they are trained on text alone. future work should investigate where these primitive referential abilities stem from and how they can be fostered in future architectures and training setups for language modeling, and neural models more generally.