E . T .: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks (2020)

Abstract: In the last decade, the field of Neural Language Modelling has witnessed enormous changes, with the development of novel models through the use of Transformer architectures. However, even these models struggle to model long sequences due to memory constraints and increasing computational complexity. Coreference annotations over the training data can provide context far beyond the modelling limitations of such language models. In this paper we present an extension over the Transformer-block architecture used in neural language models, specifically in GPT2, in order to incorporate entity annotations during training. Our model, GPT2E, extends the Transformer layers architecture of GPT2 to Entity-Transformers, an architecture designed to handle coreference information when present. To that end, we achieve richer representations for entity mentions, with insignificant training cost. We show the comparative model performance between GPT2 and GPT2E in terms of Perplexity on the CoNLL 2012 and LAMBADA datasets as well as the key differences in the entity representations and their effects in downstream tasks such as Named Entity Recognition. Furthermore, our approach can be adopted by the majority of Transformer-based language models.

## introduction language modelling is the task of transforming individual words into vector representations based on the context they appear in. hence, distant term dependencies are an inherited issue within the task. language models always seek for smart approaches towards incorporating context from longer distances as it allows for better representations compared to their limited context counterparts. intuitively, imagine attempting to start reading a novel series from the second book onward, with no information about the first. the amount of information previously missed is something that cannot be acquired. however, this is the case with most language models. while an understanding of the words is present due to the contextual information at each word's occurrence, entity information that are in distant text are lost or not transferred. until recently, recurrent neural networks (rnns), and specifically long short-term memory (lstm) networks, have been the core of all the state-of-the-art approaches @xcite . thanks to the transformers architecture @xcite , through the use of attention mechanisms, models such as xlnet @xcite , gpt @xcite and bert @xcite can account for even longer sequences. however, the computational limitations of the multihead attention in the architecture make it hard to increase the contextual information in such models @xcite . as a result, research has been focused on introducing variations to the transformer architecture, with focus on the multi-head attention mechanism, in order to alleviate part of the computational cost and increase the contextual information available to models. in this paper we present a novel approach, that makes use of coreference information during training a language model via our entity-transformer architecture, which extends the original transformer block in transformer-based language models. to that end, we incorporate the important entity information that would otherwise be unreachable for the model. as a result, we effectively boost the representations of the entity mentions, where entity information is present, without hindering the performance of the language model where entities are not present. in our experiments, we extend the gpt2 architecture to formulate our model, named gpt2e and train it on the conll-2012 dataset @xcite using the annotated coreference information. we evaluate the model's performance in terms of perplexity on the conll 2012 and the lambada @xcite datasets and showcase the effects of such training on the word representations as well as on the downstream task of named entity recognition (ner) using the conll 2012 dataset. to that end, we compare gpt2e's performance to a base model (gpt2) when trained on the same data, to highlight the effects of coreference information when paird with our entity-transformer architecture. ## related work in the last decade, the field of neural language modelling has witnessed enormous changes. with pretrained neural language models being the current go-to approach in all nlp reserach, a variety of methods models have been developed. we distinguish two major categories: general purpose language models. steady improvements have been achieved to this field with the use of deep rnns and pre-training on a large number of training data @xcite . with transformers, language models have been able to capture longer linguistic structures without the use of rnns and surpass their rnn counterparts by a big margin @xcite . recent research has focused on ways of taking advantage of more context @xcite and introducing effective methodologies to scale up the models and train them @xcite @xcite . language modelling with entity decisions. yanglm @xcite was the first to incorporate entity decisions to a language model by introducing learnable entity embeddings. alternative entity handling mechanisms are introduced in both entitynlm @xcite and setlm @xcite in addition to a length variable for entitynlm. all of the aforementioned approaches are rnnbased and hence their performance is expected to be sub-par to transformer based models. furthermore, @xcite concludes that language models handling entity decisions do not improve in performance with the addition of more hidden units and that the source data is of limited number and of specific genre which do not highlight the benefits of explicit entity information. @xcite , through attention head probing, experimentally proves that bert does model anaphoric phenomenon in the form of antecedent selection, with attention heads directly attending to the respective mention's antecedent. however, these information are not explicitly used to further enhance the model. furthermore, ernie @xcite , which uses knowledge graphs to infuse entity information to the model, only does so for named entities, completely ignoring pronouns and nominal mentions. ## our approach in order to incorporate coreference information to a language model, we require training and testing data with entity information present and a mechanism to handle existing and non-existing entities. to that end, our proposed model, gpt2e, is based on the gpt2 language model, with changes to the transformer block and an entity handling mechanism, which are described in the following subsections. as a result, gpt2e is a combination of multi-layer entity-transformer decoder blocks. the model applies multiheaded self-attention operations over the input tokens, position-wise feed-forward transformations, and entity-based attention operations. the model architecture can be described as follows: ## experiments our approach is evaluated in two steps. first we evaluate our gpt2e language model, in comparison with a gpt2 model, trained on conll 2012 and evaluated on both conll 2012 and lambada datasets. we then use the trained models to extract word representations for entity mentions based on the coreference annotations in text and measure the differences of such representations. for ner, we use the language models to extract word representatios and train the same baseline model on the conll 2012 dataset. ## results to evaluate the results of our entity-transformers architecture and the effects of corereference annotations to language modelling, we measure the change in performance of the language model using perplexity (ppl). furthermore, we compute the average difference of the representations between mentions of the same entity of the gpt2e model, between each entity mention between gpt2 and gpt2e and between non-entity mentions of the same words using cosine similarity. furthermore, we use microaverage precision, recall and f1 scores for the evaluation of our ner models. for language modelling, to compare the changes in the entity mention representations when using coreference information during training we conducted a series of experiments, taking into account the existence or absence of coreference annotation. specifically, for both models, for each entity we calculate the average similarity of its mentions with the other entity mentions, with and without the use of entity representations for gpt2e, and the average similarity between the entity representation and the entity mentions. we have limited the scope of the comparisons, using part-of-speech tags, to only nouns and proper nouns, as these will be the words that will be affected the most by our changes, given the dataset statistics presented in section 4.2. similarly, we calculated the average cosine similarity between the pronoun's representations of the two models as well as the differences between the two when entity representations are present. based on the results displayed in ## discussion and conclusions in this paper we demonstrated a novel architecture to use coreference information in transformer-based neural language models in order to create richer representations and its effects on downstream tasks. we introduced an extension over the transformer blocks of gpt2, labeled entity-transformer, that integrates coreference information to each entity mention. to that end, we also created an entity handling mechanism to create and update entity representations. furthermore, as our proposed architecture extends over the basic transformer block, it can be easily adapted to other transformer-based language models, such as bert, and also enables further research for transformer-based language models with explicit entity decisions which have far outperformed their rnn counterparts. in our experiments we showcased that in terms of language modelling, both gpt2e and gpt2, when trained on the same data, have indistinguishable performance in terms of perplexity and gpt2e has a small computational cost that translates into a slightly longer training time. however, the difference in the similarity between entity mention representations suggests that fewer iterations and mentions of each word are required to achieve the results, assuming a large enough number of mentions. this is due to the extended contextual information present at each mention occurrence, in the form of entity representations, used when training the model. what is more, the differences in these representations directly translates to an increase in tasks such as named entity recognition. as coreference is everpresent in natural language, with a better ability for a language model to understand and utilize the anaphoric phenomenon in text, we expect an increased performance in other tasks such as summarization and natural language inference. in order for language models to use coreference information, there are two requirements that need to be met. first, the models need to replace the transformer blocks with the entity-transformer blocks introduced and also adopt the entity handling mechanism to make use of entity information. second, annotated coreference information are required throughout the training corpus. while the changes described for the language models are trivial, language models require an enormous amount of training data, making it impossible to manually annotate coreference information. however, the entity handling mechanism we introduced is not affected by the lack of entity information in the training and is only boosted by the existence of them. as a result, even sparse annotations of high confidence will allow for improvements in the representations. in the future, we plan to extend our work, using noisy annotation provided by pretrained coreference resolvers so that we can train gpt2e to the wikitext dataset @xcite , creating a comparable model with the original gpt2 and other state-of-the-art language models in a wider range of tasks. furthermore, we aim to expand the abilities of our current approach to be able to make explicit entity decisions, similar to the previously cited work. for that purpose, attention head probing techniques, which have been found to model some anaphoric phenomena @xcite , and transfer learning through weight initialization from a pre-trained gpt2 model will be investigated as they can contribute to significant improvements while needing less annotated training data.