Detection and Mitigation of the Negative Impact of Dataset Extractivity on Abstractive Summarization (2023)

Abstract: In text summarization, extractivity is defined as a measurement of the degree of overlap between a source document and its summary. Previous research has shown that the extractivity level of training data can influence both output extractivity and the amount of factual information (i.e. faithfulness) in outputs for abstractive summarization. However, it remains unclear if and how extractivity impacts the performance of abstractive models. In this work, we investigate the relationship between dataset extractivity and model performance by comparing the performance of trained models under different degrees of extractivity. We find that while low levels of extractivity can improve performance, as extractivity increases, performance is negatively impacted. Furthermore, through an analysis of the model’s copy continuity of content, we discover that higher extractivity leads to a greater tendency for the model to copy text continuously from the source document rather than identifying and summarizing important content that should be covered in the target summary. To address these issues, we propose a simple and effective method to design copy labels for fixing the model’s copying behaviors and train the model with a copy mechanism. The experimental results illustrate the effectiveness of our strategy in alleviating the negative impact on model performance resulting from high dataset extractivity, and that our method outperforms several competitive baselines.

## introduction text summarization is the task of reducing the content of an original text while preserving its salient content @xcite . a multitude of techniques to generate summaries exists, which can be categorized into extractive and abstractive approaches. with extractive techniques, a summary is generated by extracting salient sentences from the text @xcite . with abstractive techniques, new sentences or commonly called para-phrased sentences are generated so that the resulting summaries contain new words and expressions that do not occur in the original text @xcite . in addition to the extractivity for summarization models, previous work has proposed the idea of extractivity for summarization datasets specifically in terms of the summarization styles in datasets @xcite . more specifically, extractivity is a metric meant to capture the overlap between a text and its summary, and quantifies the extent to which a summary is a derivative of a text. summarization datasets typically contain aligned pairs of texts and human-generated or human verified summaries. as an intrinsic property of summarization datasets, extractivity has been shown to influence various characteristics of output summaries, such as output extractivity @xcite and the amount of factual information (i.e. faithfulness) @xcite . generally, datasets with high extractivity are treated as suboptimal for abstractive summarization @xcite , but there has been a lack of systematic research investigating the relationship between the magnitude of extractivity in training datasets and the performance of abstractive summarization models. to address the lack of understanding about the relationship between extractivity in summarization training data and the performance of abstractive summarization models, in this work, we first develop a framework for evaluating the change in model performance over baseline models (control) at three levels of amount of extractivity. specifically, we split the training data into different subsets based on their increasing extractivity and then train the control models on each resulting subset. by comparing the results, we find that as the extractivity in training data increases, model performance initially improves with increasing extractivity but then drops when extractivity is too high. this naturally raises the next question, which we also address in this paper: why and how does extractivity cause model performance to change? based on our observations, we posit that high extractivity in training data can lead to an increased likelihood of overfitting an abstractive model whereby the model continuously copies from source documents. to validate this hypothesis, we propose new metrics to measure how much content in output summaries is copied continuously from a source text, and how important this continuously copied content is by comparing it with target summaries. we apply these metrics to outputs from the control models defined above. the results support our hypothesis ahd provide a new direction for mitigating the negative effects of high extractivity by promoting the selection of truly important content that is covered in target summaries for copying. through our analysis, we have identified a primary harm associated with highly extractive training data, namely the propensity for models to excessively rely on continuous copying rather than focusing on the essential content that should be included in summaries. to alleviate this potential harm, we propose and test a simple and effective strategy based on a copy mechanism @xcite . our solution first identifies the salient parts in a source document that the model should copy into the summary. we do this by solving an integer linear programming. we then create ground-truth labels based on the optimal solution for the copy distribution, with the goal of guiding the model to focus on the content it should copy rather than blindly copying continuous text spans from the source document. lastly, we incorporate this copy mechanism into bart and train it with an auxiliary loss based on the copy distribution and the constructed copy labels. the experimental results demonstrate that our method not only is effective in mitigating the negative effects of high extractivity, but can also improve model performance as it outperformed several recent competitive baselines. in general, we aim to explore and answer the following research questions in this work: ## extractivity analysis in this section, we introduce our research design, following a similar process as @xcite , to answer our first research question: how does the extractivity in summarization datasets influence the performance of abstractive models? ## continuity analysis now that we have reached the conclusion that high extractivity can negatively influence abstractive model performance, our our second question to answer is: how does dataset extractivity cause the performance of abstractive models to change? after examining the model outputs, we formed a plausible hypothesis: extractivity could lead to an increased tendency or preference for an abstractive model to copy content continuously, potentially neglecting the inclusion of important content that should have been covered in a summary. to validate this hypothesis, it is necessary to develop new metrics to measure the continuity and salience of text spans that are continuously copied by the model. these metrics can be applied to the output summaries from all triplet models (as de-fined in sec. 2) and the results can be compared to test the validity of the our hypothesis. ## mitigating the negative impact of high extractivity on model performance a key takeaway from previous sections in this paper is that high dataset extractivity can result in a tendency to excessively copy text from source documents while neglecting important content that should be included in a summary. to address this issue, a natural solution is to focus the model's attention on important content during training to encourage the model to cover the important content when copying during inference. to achieve this, we first identified a mapping between tokens in a source document x and extractive fragments in the target summary y , where extractive fragments are the set of shared sequences of tokens in x and y @xcite . it should be noted that extractive fragments comprise all tokens the model can copy from the source document. then, a copy mechanism was implemented @xcite and we transformed the above mapping as the label for the copy distribution so that the model can attend to important texts when copying text from source documents. ## conclusion in this work, we have explored how the amount of extractivity in summarization training datasets influences the performance of abstractive models. by comparing model performance under different levels of dataset extractivity, we showed that low levels of extractivity can improve model performance while the impact becomes negative as extractivity is high. furthermore, with the analysis of the model's copy continuity of content, we found that high dataset extractivity encourages the model to copy text continuously, which can cause models to ignore important content. in order to mitigate these negative effects, we presented a novel, simple, and effective strategy that creates labels for fixing the model's copying behaviors. by training the model with a copy mechanism and our copy labels, our experimental results show that our method can effectively alleviate the harm resulting from dataset extractivity and outperforms several competitive baselines.