Global Learning with Triplet Relations in Abstractive Summarization (2024)

Abstract: Abstractive summarization models learned with token-level maximum likelihood estimation suffer from exposure bias, that the condition for predicting the next token is discrepant during training and inference. Existing solutions bridge this gap by learning to estimate semantic or lexical qualities of a candidate summary from the global view, namely global learning (GL), yet ignore maintaining rational triplet-relations among document, reference summary, and candidate summaries, e.g., the candidate and reference summaries should have a similar faithfulness degree judging by a source document. In this paper, we propose an iterative autoregressive summarization paradigm - IARSum, which fuses the learning of triplet relations into a GL framework and further enhances summarization performance. Specifically, IARSum develops a dual-encoder network to enable the simultaneous input of a document and its candidate (or reference) summary. On this basis, it learns to 1) model the relative semantics defined over tuples (candidate, document) and (reference, document) respectively and balance them; 2) reduce lexical differences between candidate and reference summaries. Furthermore, IARSum iteratively reprocesses a generated candidate at inference time to ground higher quality. We conduct extensive experiments on two widely used datasets to test our method, and IARSum shows the new or matched state-of-the-art on diverse metrics.

## introduction abstractive summarization is a classical natural language generation (nlg) task, which aims to rewrite a long document into a shorter version, retaining only the salient information (kumar and @xcite . in recent years, the advancement of pre-trained language models (plms) @xcite ; zhang a n ti c s lexical similarity s r c r c c r s semantic / lexical se m a n ti c semantic s e m a n ti c point movement in previous approaches point movement in our approach figure 1 : graphicalization of triplet relations among a document, the reference summary, and a candidate summary, where each point of the triangle △src presents source document (s), reference summary (r), and candidate summary (c), respectively. upper right: traditional gl methods only consider the edge rc and deem it the semantic or lexical gaps between the points r and c, which they learn to minimize. lower right: seqco further considers s and regards each edge as a semantic gap to be minimized. left: we extend traditional gl methods by treating the side edge sr (sc) as a relative semantics metric, and we highlight the balance of edges sr and sc. to address these problems, we highlight rational relations within the triplet (document, candidate summary, reference summary) in abstractive summarization. take the geometrical triplet in figure 1 for intuitive perception. traditional gl methods view the edge rc as lexical or semantic gaps between candidate and reference summaries, aiming to draw points r and close as possible. similarly, seqco considers from the semantics perspective and further aims to condense the triangle △src into a single point (draw all three points s, r, and c as close as possible). in this work, we base on the traditional gl assumption and further treat side edges sr and sc as relative semantics metrics between the document and summaries. we propose to balance the edges sr and sc while minimizing the edge rc. our inspiration is that an ideal model should generate candidate summaries similar to the reference on both semantic and lexical aspects @xcite . in particular, the limitations of gl-based methods, which tend to yield summaries with unsatisfactory relative semantics, such as faithfulness and abstractiveness @xcite measured by the source document, can be effectively fixed by balancing the two side edges of △src. according to the above insights, we propose an iterative autoregressive summarization paradigm (iarsum), which facilitates learning the mentioned triplet relations with a standard gl framework to enhance summarization performance. specifically, iarsum generates a summary through a series of iterations, during which the model re-inputs and reprocesses the previously generated summary in each iteration to get improved versions. this encourages assessing summaries' quality from a global view and effectively prevents exposure bias. we build iarsum on a double encoder-decoder network following transformer architecture to fulfill the desired properties. it uses two serial encoders to encode the document and reinput summaries, respectively, and uses the second encoder's outputs to model summary-document semantics. to learn the iarsum model aware of triplet relations, we reward the model to get similar outputs from the second encoder when provided with candidate and reference summaries as input, respectively. we also reward the model once a candidate achieves higher lexical overlap with the reference after reprocessing. furthermore, we adopt an offline mini-risk training strategy that enforces the model to maximize the mentioned rewards. in inference, a trained iarsum model can adaptively refine the generated summaries in sequential iterations for increased quality. in summary, we make three-fold contributions. first, we explore rational relations within the triplet (source document, reference summary, candidate summary) in summarization and propose to balance the relative semantics over tuples (candidate, document) and (reference, document) while reducing the lexical differences within @xcite . second, we propose iarsum, a novel summarization paradigm that facilitates learning our suggested triplet relations with a gl framework to boost summaries' quality. finally, we conduct extensive experiments on two public datasets to test our methods. results show that iarsum matches or outperforms previous state-of-the-art (sota) approaches in generating high-quality summaries measured by multiple metrics. furthermore, we transfer iarsum to few-shot settings and show its superior robustness. ## method in this section, we describe the details of our proposed methods. we introduce the iterative autoregressive text generation paradigm in section 3.1, describe the iarsum model architecture in section 3.2, and illustrate the offline global learning strategy in section 3.3. ## conclusion in this paper, we focus on improving the existing approaches that alleviate exposure bias suffered in abstractive summarization. specifically, we introduce a novel iterative autoregressive summarization paradigm, iarsum. it models the generation of an abstract summary as a series of transitions of intermediate results, ranging from coarse to refined quality. iarsum also enables learning rational relations among a document, the reference summary, and candidate summaries under a standard gl framework. extensive comparison experiments revealed the effectiveness and advancement of our method. 0.1 1.0 10.0 100.0 200.0 20 25 30 35 40 45 50 value @xmath0.1 1.0 10.0 100.0 200.0 @xmath1: the performance of iarsum with increasing λ on cnn/dm and xsum. unsurprisingly, increasing the beam width can indeed boost the model's performance. however, the rouge score improvement reduces once the k is over 16. we set k to 16 to save computational costs.