C ore LM : Coreference-aware Language Model Fine-Tuning (2021)

Abstract: Language Models are the underpin of all modern Natural Language Processing (NLP) tasks. The introduction of the Transformers architecture has contributed significantly into making Language Modeling very effective across many NLP task, leading to significant advancements in the field. However, Transformers come with a big computational cost, which grows quadratically with respect to the input length. This presents a challenge as to understand long texts requires a lot of context. In this paper, we propose a Fine-Tuning framework, named CoreLM, that extends the architecture of current Pretrained Language Models so that they incorporate explicit entity information. By introducing entity representations, we make available information outside the contextual space of the model, which results in a better Language Model for a fraction of the computational cost. We implement our approach using GPT2 and compare the fine-tuned model to the original. Our proposed model achieves a lower Perplexity in GUMBY and LAMBDADA datasets when compared to GPT2 and a fine-tuned version of GPT2 without any changes. We also compare the models’ performance in terms of Accuracy in LAMBADA and Children’s Book Test, with and without the use of model-created coreference annotations.

## introduction language models (lms) have seen significant improvements in performance due to the transformers architecture @xcite . the resulting pretrained language models (plms) such as bert @xcite , gpt2 @xcite and xlnet @xcite have contributed to significant advancements in many natural language processing (nlp) tasks. plms, regardless of their training objective and methodology, aim to learn contextualized text representations. as such, the available context during training has a key role in the models performance. the quadratic computation complexity in the attention mechanism of the transformer architecture, in terms of input sequence length, has been a limiting factor to the amount of contextual information the models can have at each step. to make this architecture more efficient, a plethora of approaches have been introduced @xcite @xcite , aiming to lower the computational complexity without sacrificing performance. @xcite summarizes these approaches and categorizes them based on type of attention mechanism in their survey. yet, even with linear complexity, physical resources will always limit the amount of contextual information that can be handled simultaneously. in terms of language, entities represent a natural way to tie words together through large pieces of discource. in nlp, coreference resolution is the task that aims to identify and group these entity mentions together, when they refer to the same real world entity @xcite . therefore, coreference resolution presents a natural way to link the context that we are currently handling with distant information, far outside the capabilities model architectures and current hardware resources. however, while this information is present in the text, it is usually not annotated and when annotated are very sparse and in small quantities making it extremely difficult to train large lms @xcite . in this paper we present a framework to effectively use coreference annotations to further finetune large plms, increasing their performance far more than just by fine-tuning on the same data. by using large plms we take advantage of existing resources that are expensive to reproduce and come with a big environmental cost @xcite . what is more, fine-tuning takes advantage of the massive amount of data that essentially initialize the model, making it possible to introduce new capabilities to models with small amounts of annotated data. in our approach, we use gpt2 as our base model and extend its architecture with the addition of a new entity-gating layer that handles entity annotations along with a gating mechanism that handles information flow between the base model and the entity-gating layer. as such, our approach uses entity representations when they are available through both training and inference, without imposing any constraints to the model's functionality. for our experiments, we compare the performance of gpt2, post and pre fine-tuning, with and without our changes in a series of relative tasks. furthermore, since most coreference annotated datasets are either very small or hard to acquire, we use gumby @xcite as our fine-tuning dataset which is a model annotated corpus. in addition, by using noisy annotations we aim to show the resilience of our approach to noise and the universality of our framework. our results highlight the effects of our framework in language modeling, modeling long-range dependencies, and in specific word types where our fine-tuned model achieves better performance than gpt2. ## background this section provides a concise overview of the transformers architecture @xcite , which is the foundation of our approach, followed by a brief explanation of autoregressive language modeling used by our base model, gpt2. ## coreference-aware language modeling language models are limited to the amount of information they can process based on the available context window k (eq. ( 5 )). however, increasing k will lead to an exponential increase in computation complexity. as such, we implicitly increase the available information, without increase the context window, using coreference annotations. we do this by introducing entity-representations ( §3.2), in the form of vectors, that are utilized by the model to infuse the respective entity information to all the tokens in the sequence that are part of entities. the entity-representations are created from the whole discourse available, hence holding contextual information that are very distant to the context window. as such, we make no alterations to the language modeling objective. these vector representations are introduced to the model via an entity-gating layer ( §3.3) that is added to model architecture. ## experiments we fine-tune our entity-aware lm with the original training objective of maximizing the logprobability of u (eq. ( 5 )). language models are evaluated in terms of perplexity (ppl) which is the exponentiated average negative log probability per word prediction, as we are using a word-level base model. as such, ppl is a direct reflection of the model's loss. for fine-tuning we use gumby, a model annotated coreference corpus, containing 4960 documents. the entity-aware fine-tuned lm is evaluated on gumby, using the created entity representations during fine-tuning and in a zero-shot setting, without any further training, on lambada, wikitext2, wikitext103 in terms of ppl and on lambada and cbt in terms of accuracy. we also evaluate the effect of newly introduced entity annotations from a separate coreference resolution model on the zero-shot evaluated corpora, to investigate their effect in the model's performance. detail information about the model parameters and experimental setup are provided in appendix a. appendix b enlists our methodology to annotate the lambada and cbt corpora with coreference clusters, used in zero-shot evaluation only. ## error analysis in our experiments, we showcased the effects of corelm on the base model, in different scenarios. in this section we investigate the cases in which the base model performed better than the corelm model and vice-versa. for that reason, we manually compared the predictions made between the two models, using the gumby fine-tuned gpt2 model as the deciding factor in our observations between the effects of corelm fine-tuning and simple fine-tuning. we limit our analysis on the cbt corpus which provides a meaningful way to evaluate the changes due to its word category variants. in our analysis, we noticed that fine-tuning on gumby lead to wrong choices by the model in all categories, which were not made by the gpt2. however, when corelm was used with coreference annotations, 83% of those cases were corrected. the vast majority of the corrected cases were in the prepositions and named entities word categories, with only 19% of the corrected ones in verbs and common noun word categories. a positive correlation between corrected cases and correct coreference annotations was noticed, as cases in which corelm persisted on the wrong option also had different coreferent annotations than the correct sentence. in the cases where corelm performed better than both the fine-tuned and base model, we noticed the same correlation between coreference annotations that directly affected the available options. unavailingly, a small number of cases in which, the nouns following a preposition or the named entities themselves were annotated in a wrong coreference cluster, lead to different probability distributions for the available options and eventually to making the wrong selection. furthermore, comparing the predictions made by corelm with and without the use of coreference annotations, we notice that the majority of the errors persisted when the option was not directly annotated in a coreference cluster. ## discussion our approach takes advantage of plms and increases their performance by exploiting distant contextual information in the form of entity representations using coreference annotations. what is more, it can be used with and without the existence of such information, hence not hindering the flexibility of plms. our experiements when fine-tuning gpt2 and the gpt2 with corelm on the same data, without using coreference information show that even when the fine-tuning data are not best suited for the downstream tasks, corelm maintains more of the original model's performance, making it a more resilient fine-tuning methodology. in addition, as corelm is very modular, it can be applied to the majority of lms, including non-autoregressive approaches such as bert. in language modeling, our approach achieved significantly lower perplexity in all corpora compared to the gpt2 fine-tuned version. what is more, gumby proved to be an ill-suited corpus to fine-tune for both lambada and wikitext based on the post fine-tuning performance. regardless, fine-tuning with corelm demonstrated significant gains, even compared to the pre fine-tuned model. in both lambada and cbt, we show increase in accuracy compared to gpt2 pre and post fine-tuning. most notably, the named entity and prepositions word types showed the biggest increase in cbt, with common nouns suffering in comparison compared to the pre fine-tuned version and verbs attaining a slightly better accuracy. our error analysis highlights the effect of coreference annotations to these changes in performance. in all cases, fine-tuned gpt2 achieved lower scores in all word categories compared to corelm fine-tuned, which indicated that gumby is not a suitable fine-tuning corpus for these tasks. using model-created coreference annotations during zero-shot evaluation did increase the performance slightly. while the performance increase is minor, the entity representations used were only initialized from the scope of each example as there was not long contextual information to take advantage off. what is more, coreference annotations increased the performance regardless of the information being within the context window in all the examples, indicating that further gains can be achieved by using coreference annotations extensively over large pieces of discourse. unavailingly, our approach is based on the ability of other models to accurately predict coreference clusters so that corelm can exploit the coreferent mentions. the errors in the predicted clusters introduce noise, to both entity representations and the final model. what is more, maintaining a persistent set of entity representations, is computationally expensive and can be very burdening when considering a large collection of documents. as a result, an entity management mechanism, similar to the one used in @xcite , would be required for corelm to scale efficiently to bigger document collections. ## related work early entity-aware lms were trained from scratch with entity information available through the training process. specifically, @xcite and @xcite both introduced models that used reference information with attention-based mechanisms to incorporate them into the model. @xcite model made use of both intra-linguistic, coreferring mentions in text, and extra-linguistic, tables and lists, features through three different components, only creating learnable embeddings for intra-linguistic features. @xcite model was focused only on intra-linguistic mentions, i.e. corefererring mentions, and introduced additional control variables indicating if the next token is part of an entity as well as the number of remaining entity tokens. recently, @xcite extended @xcite approach by using learnable entity embeddings. these approaches also have the ability to autoregressively predict the entity of the following word and constrain the word generation to a specific entity. engen combines entitynlm @xcite with s2sa @xcite , to train a generative language model that uses both previous sentence representations and entity representations in order to generate coherent text @xcite . in comparison to their approach, our approach handles entities and information flow differently. @xcite also introduced a transformer-based approach towards incorporating learnable entity representations, using multi-head attentions inside all the transformer blocks of the model. in comparison to past approaches, this approach was only focused on the effective use of entity information in a lm and did not predict the following entity information. however, all of these models required high quality annotated data to be trained from scratch which were limited and of specific genre @xcite . other methods have focused on using only extra-linguistic information, ignoring pronouns and nominal mentions in text. ernie @xcite uses knowledge graphs (kg) to extract entities for identified named entities. however, ernie represents entity information using a pre-trained knowledge embedding model, trained on the used kg, and does not create dynamic entity representations. @xcite use knowledge bases to learn word type embeddings based on the learned type representations. similar to past approaches @xcite it can autoregressively constrain the prediction to a certain entity type. the type information is restricted using pre-defined vocabularies making the model less dynamic in its entity predictions. both approaches have been found to be very effective for the tasks that were respectively designed, however they lack in expandability of their domain of application without requiring complete retraining. ## conclusions and future work in this paper we presented corelm, a modular fine-tuning framework for plms to exploit modelcreated coreference annotations in order to create better mention representations and an overall better lm. in our experiments we showcased a performance increase when evaluating in a zero-shot setting, compared to the similarly fine-tuned model, even when the fine-tuning corpus did not generalize well to the end tasks. our analysis shows that coreference annotations play a significant role in both fine-tuning and in downstream task performance, with correct annotations leading to better performance when used. in addition, our work helps in adding a new frontier to coreference resolution through the effective use of coreference annotations in language modeling. in this paper we showcased the effects of coreference annotation even when the information is within the context window of the model. using coreference annotations can further lead to the decrease of the required context window and boost approaches like shortformer @xcite , leading to better and more efficient lms. in the future we aim to create a more efficient approach to lm through the use of both intralinguistic (coreference) and extra-linguistic (kg) features. undeniably, kgs provide a means for structured, high quality information that cannot be found in a single text. we believe that an information fusion from coreference annotation and graph nodes, along with short context window will not be computationally prohibitive and lead in better, information rich, lms.