R ef S um: Refactoring Neural Summarization (2021)

Abstract: Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement. Our system can be directly used by other researchers as an off-the-shelf tool to achieve further performance improvements. We open-source all the code and provide a convenient interface to use it: https://github.com/yixinL7/Refactoring-Summarization .

## introduction in neural text summarization, system designers commonly have flexible choices in model architectures @xcite , decoding strategies @xcite (e.g. beam search) and etc. as a result, even on the same dataset, different selection biases of these choices will lead to diverse system outputs @xcite . to combine complementarity of system's output under different setups, researchers have made some preliminary efforts on two-stage learning @xcite ; gonzález-rubio "doc, hypo, ref" represent "input document, generated hypothesis, gold reference" respectively. "hypo'" represents texts generated during test phase. θ base and θ meta represent learnable parameters in two stages. @xcite , consisting of (i) a base-stage: first generates different outputs under different setups, and (ii) a meta-stage: then aggregates them in diverse ways, exemplified by stacking that uses a high-level model to combine multiple low-level models (ting and @xcite , or reranking @xcite , which aims to rerank different outputs of one system. although these methods each play a role in different scenarios, they suffer from following potential limitations: in this work, we aim to address these limitations by proposing a general framework, named refactor, which can not only serve as a base system to construct a summary by selecting sentences from the source document but also act as a meta system to select the best system output from multiple candidates. the unification of base and meta systems allows them to share a set of parameters, thereby alleviating the "base-meta learning gap". besides, we propose a pretrain-then-finetune paradigm for refactor that mitigates the "train-test distribution gap". in practice, our proposed refactor can be applied to different scenarios. for example, as a meta system, it can be used for multiple system combination or single system re-ranking. our contributions can be briefly summarized as: ## preliminaries existing works commonly design systems in an end-to-end fashion @xcite , which, though effective, also proves to be insufficient in some scenarios @xcite . instead of optimizing a system in an end-to-end fashion, one more flexible paradigm, stage-wise learning, is to break down the holistic process into different stages. the basic idea is to incorporate different types of inductive biases stage-wisely and two typical examples are: stacking and reranking. stacking stacking (a.k.a, stacked generalization) is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy (ting and @xcite . in nlp research, this method has been widely explored in machine translation (mt) task. traditionally, it is used to improve the performance of statistical mt systems @xcite @xcite . some recent work @xcite also extends this method to neural mt where the meta model and base systems are all neural models. there is a handful of works about system combination for summarization @xcite , in which a feature-based meta model is used for combining unsupervised text summarization systems. reranking reranking is a technique to improve performance by reranking the output of an existing system, which has been widely used across different nlp tasks, such as constituency parsing @xcite , dependency parsing @xcite , semantic parsing @xcite , machine translation @xcite . comparing reranking and stacking, both of them involve two-stage learning and the first stage would provide multiple candidate outputs as the input for the second stage. however, they differ in the way how multiple candidate outputs are generated at the first stage. specifically, reranking usually decodes k-most qualified results during inference, using one base system. by contrast, stacking generates multiple outputs that are usually from different base systems. ## summarization as two-stage learning in what follows, we detail how to formulate summarization as a two-stage learning task. base system the system in the base stage aims to generate a summary based on the input text. specifically, given a document @xmath0, • • • , s n } with n sentences, we refer to c as a candidate summary of d generated by a summarization system, which can be parameterized in diverse forms: meta system in practice, different choices of parameterized function base(•), training method t and decoding strategy s commonly lead to different candidate summaries, @xmath1, • • • , c k }, where c represents a set of different candidate summaries. the goal of the meta system is to utilize complementarities among c by popular techniques, such as reranking and system combination. specifically, given a set of candidate summaries c, a meta system is used to re-construct a new candidate summary c * ## refactoring text summarization despite effectiveness of existing meta systems, they, as briefly mentioned in §1, suffer from two major problems: (i) base-meta learning gap and (ii) train-test distribution gap. ## implications and future directions we present a general framework for utilizing the complementarity of modern text summarization systems by formulating text summarization as a two-stage learning problem. our proposed model, refactor, can be used either as a base system or a meta system, effectively mitigating the learning gaps introduced in the two-stage learning. experimental results show that refactor is able to boost the performance of the base systems, and achieves the state-of-the-art performance on cnndm and xsum datasets. we believe this work opens up a new direction for improving the performance of text summarization systems apart from an iterative process of searching for better model architectures -the gain of performance could be made by fully investigating and utilizing the complementarity of different systems with various architectures, problem formulations, decoding strategies, etc.