Challenges in including extra-linguistic context in pre-trained language models (2022)

Abstract: To successfully account for language, computational models need to take into account both the linguistic context (the content of the utterances) and the extra-linguistic context (for instance, the participants in a dialogue). We focus on a referential task that asks models to link entity mentions in a TV show to the corresponding characters, and design an architecture that attempts to account for both kinds of context. In particular, our architecture combines a previously proposed specialized module (an “entity library”) for character representation with transfer learning from a pre-trained language model. We find that, although the model does improve linguistic contextualization, it fails to successfully integrate extra-linguistic information about the participants in the dialogue. Our work shows that it is very challenging to incorporate extra-linguistic information into pre-trained language models.

## introduction identifying the real-world entity an expression refers to is crucial for natural language processing, since humans use language to talk about the world. this, however, requires models that represent the real world such that linguistic expressions can be mapped to them. for instance, in figure 1 , which is a snippet of a dialogue from the tv show friends, we need to know that it is joey tribbiani who is speaking to be able to interpret the pronoun "i". state-of-the-art nlp models typically focus on linguistic context, not on extra-linguistic context such as who is speaking to whom. we aim at integrating extra-linguistic context, in particular information about participants in a dialogue; also, we aim at combining it with information coming from the linguistic context. we focus on the character identification task of semeval 2018 @xcite ), aimed at classifying mentions from the dialogue scripts of the tv show friends (see figure 1 ). the model that joey tribbiani ( 183 won the semeval competition @xcite proposed an external module to encode entity information in a structured way (henceforth, "entity library"). this approach enabled the incorporation of extra-linguistic information, in particular speaker information, which allowed the model to learn patterns such as "i refers to the character that is speaking"; and, as a result, it worked comparatively well on rare entities. however, @xcite showed that the model's good performance was not correlated with meaningful entity representations. moreover, the model performed poorly in expressions that require a good grasp of the linguistic context, like 3rd person pronouns and common nouns. aina et al.'s base model was an lstm trained from scratch on the character identification task (with the exception of pre-trained noncontextualized word embeddings). we propose to instead add the entity library to a pre-trained language model: bert @xcite . pretrained language models @xcite have been shown to provide good contextual representations @xcite , and they have enabled advances also in referential tasks @xcite @xcite . we expected that combining bert with the entity library would synthesize the benefits of both, encoding and exploiting both the extra-linguistic and linguistic information in the context. we also expected that, as a result of these improvements, this model would yield better entity representations. contrary to expectation, however, we do not improve on the state-of-the-art model of @xcite . through analysis, we show that our model does improve the performance for contextdependent expressions, such as third-person pronouns, suggesting that it is better at handling the linguistic context; however, it performs worse on expressions that depend on the extra-linguistic context, such as first-and second-person pronouns, which are much more frequent in the data. moreover, the entity representations are only marginally improved. the problem, we argue, comes from the fact that integrating extra-linguistic information in pre-trained language models is far from trivial. task in order to have a comparable setup to previous studies, the dataset and the task are the same as the ones described in choi and chen (2018). the training and test data span the first two seasons of the sitcom friends, and the task is to predict which character is referred to by each referring expression (see figure 1 ). model in our model, the input tokens go through a pre-trained bert. then the speaker information (i.e., an embedding identifying the character who produced the utterance) is concatenated to the token representation. this representation is fed to a multi-layer perceptron (mlp). the output of this step is compared to the entity library (entlib) proposed in @xcite , via dot products with each character embedding in the entlib, in order to produce the final prediction (softmax over the dot products). the entity library is a learnable matrix where each row is associated with one of the 401 characters from the dataset. as in the version in @xcite , the parameters of the speaker embedding matrix and of the entity library are shared. the weights of bert are tuned to the character identification task. section a.2 in the appendix reports model details. the most notable differences of our architecture with that of @xcite and @xcite are the following: 1) we run the input text through a pre-trained language model; 2) our model processes the input token with its textual context before accessing the speaker information. by contrast, aina et al.'s architecture directly passes the input token to the lstm jointly with the speaker. this latter difference will be crucial in explaining the results, as we will see in the next section. we conduct ablation experiments to investigate the benefits of different components of our model: