Named Entity Inclusion in Abstractive Text Summarization (2022)

Abstract: We address the named entity omission - the drawback of many current abstractive text summarizers. We suggest a custom pretraining objective to enhance the model’s attention on the named entities in a text. At first, the named entity recognition model RoBERTa is trained to determine named entities in the text. After that this model is used to mask named entities in the text and the BART model is trained to reconstruct them. Next, BART model is fine-tuned on the summarization task. Our experiments showed that this pretraining approach drastically improves named entity inclusion precision and recall metrics.

## introduction current state-of-the-art abstractive summarization methods achieved significant progress, yet they are still prone to hallucinations and substitution of the named entities with vague synonyms or omitting mention of some of them at all @xcite , @xcite , @xcite . such inconsistencies in the summary limit the practicability of abstractive models in realworld applications and carry a danger of misinformation. example in scientific texts are especially vulnerable to this issue. omitting or substituting the name of the metric used or the method applied can make a summary useless or, in the worst case scenario, totally misleading for a reader. we make the following contributions: without named entities with named entities famous north-american andrew ng from stanford scientist suggested suggested a new way a new way of training of training feed-ai algorithms. forward neural networks. for automatic summarization, one of the important issues is extrinsic entity hallucinations, when some entities appear in summary, but do not occur in the source text @xcite . a number of studies have been devoted to this problem, such as fixing entity-related errors @xcite , ensuring the factual consistency of generated summaries @xcite , and task-adaptive continued pertaining @xcite . in our paper, we address the problem of named entity awareness of the summarization model by first training it on the ner task before final finetuning to make the model entity aware. the idea of utilizing named entities during the pretraining phase first was described back in @xcite , where the authors proposed the usage of knowledge graphs by randomly masking some of the named entity alignments in the input text and asking the model to select the appropriate entities from the graphs to complete the alignments. one of the disadvantages of that approach is the need for a knowledge base, which is extremely difficult to build. only a limited number of domain-specific knowledge bases exist, and none of them can be considered complete. the study @xcite addresses the problem of the factual consistency of a generated summary by a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and a generated summary. training data is generated by applying a series of rule-based transformations to the sentences of the source documents. a similar approach is suggested by the authors of the paper @xcite who try to preserve the factual consistency of abstractive summarization by specifying tokens as constraints that must be present in the summary. they use a bert-based keyphrase extractor model to determine the most important spans in the text (akin to the extractive summarization) and then use these spans to constrain a generative algorithm. the big drawback of this approach is the vagueness of the keyphrases and the limited amount of training data. also, the use of the bert model leaves room for improvement. the analogous solution uses @xcite , where the authors suggest entity-level content planning, i.e. prepending target summaries with entity chains -ordered sequences of entities that should be mentioned in the summary. but, as the entity chains are extracted from the reference summaries during the training, this approach cannot be used in an unsupervised manner, like mnelm, proposed in this work. ## method we propose a three-step approach that aims to avoid all the aforementioned drawbacks: 1) at the first step the ner model is trained on a domain-specific dataset; 2) then the trained ner model is used for the mlm-like unsupervised pretraining of a language model; 3) the pretrained model is finetuned for the summarization task. by following these steps, we can use a large amount of unlabeled data for the pretraining model to select domain-specific named entities and therefore to include them in the generated summary. in comparison with a regular mlm pretraining, the suggested approach helps the model converge faster, shows an increased number of entities included in the generated summary, and drastically improves the avoiding of hallucinations, i.e. eliminates named entities that did not appear in the original text. ## datasets and evaluation metrics in this work, we use two datasets: scierc @xcite for training named entity extraction model and arxiv @xcite dataset for pretraining and training of the summarization model. the scierc dataset includes annotations for scientific entities for 500 scientific abstracts. these abstracts are taken from 12 ai conference/workshop proceedings in four ai communities from the semantic scholar corpus. these conferences include general ai (aaai, ijcai), nlp (acl, emnlp, ijcnlp), speech (icassp, interspeech), machine learning (nips, icml), and computer vision (cvpr, iccv, eccv) conferences. the dataset contains 8.089 named entities and defines six types for annotating scientific entities: task, method, metric, material, other-scientific-term and generic. scierc utilizes a greedy annotation approach for spans and always prefers the longer span whenever ambiguity occurs. nested spans are allowed when a subspan has a relation/coreference link with another term outside the span. the second dataset is the arxiv dataset which takes scientific papers as an example of long documents and their abstracts are used as ground-truth summaries. authors of the dataset removed the documents that are excessively long or too short, or do not have an abstract or some discourse structure. figures and tables were removed using regular expressions to only preserve the textual information. also, math formulae and citation markers were normalized with special tokens. only the sections up to the conclusion section of the document were kept for every paper. this dataset contains 215,912 scientific papers with the average length of 4,938 words and the average summary length of 220 words. to evaluate the performance of the model we used rouge-1, rouge-2, and rouge-l metrics. for scoring the occurrence of named entities and their soundness and completeness we use namedentity-wise precision and recall: the training procedure of our model consists of the three main stages, illustrated in figure 1 . ## results our model shows higher precision and recall in named entity inclusion in comparison to the same architecture, which was pretrained using regular masked language model objective -results of both models can be found in ## discussion during the training of our model, we noticed that increase in common metrics for text summarization causes a decrease in named entity inclusion. we believe the reason for this is the limited length of the generated summary -one can have only so many named entities, before they will displace other words from the original text, causing the model to reformulate sentences and miss more words from the source. therefore, during training, we tried to find the optimum point, at which the model will have high rouge scores and will still have high ne inclusion. at this point the mnelm-pretrained model, while keeping higher ne inclusion, converges faster than a regular mlm (in terms of rouge metrics). the comparison can be found in ## conclusion in this work, we described the task of preserving named entities in an automatically generated summary and presented the masked named entity language model (mnelm) pretraining task. we show that with the mnelm pretraining procedure the bart model can achieve higher precision and recall of named entity inclusion. pretraining with the mnelm task helps the model concentrate on domain-specific words, whereas mlm learns to reconstruct mostly common words. this leads to stronger attention on named entities, more likely preserving them in a generated text. the suggested model shows solid results in summarization metrics in comparison to the regular approach and converges faster. in further research, we plan to improve the quality of the pretraining by masking a sequence of named entities with a single mask -the step that could help the model, according to the original bart paper @xcite . also, we plan to conduct more experiments with different hyperparameters (such as masking probability), on more datasets, including pubmed @xcite and to train an even better ner model. in addition, we plan to improve the proposed model by overcoming the internal limitation on the number of input tokens (currently, it only has access to 1024 tokens).