Conditional Neural Generation using Sub-Aspect Functions for Extractive News Summarization (2020)

Abstract: Much progress has been made in text summarization, fueled by neural architectures using large-scale training corpora. However, in the news domain, neural models easily overfit by leveraging position-related features due to the prevalence of the inverted pyramid writing style. In addition, there is an unmet need to generate a variety of summaries for different users. In this paper, we propose a neural framework that can flexibly control summary generation by introducing a set of sub-aspect functions (i.e. importance, diversity, position). These sub-aspect functions are regulated by a set of control codes to decide which sub-aspect to focus on during summary generation. We demonstrate that extracted summaries with minimal position bias is comparable with those generated by standard models that take advantage of position preference. We also show that news summaries generated with a focus on diversity can be more preferred by human raters. These results suggest that a more flexible neural summarization framework providing more control options could be desirable in tailoring to different user preferences, which is useful since it is often impractical to articulate such preferences for different applications a priori.

## introduction text summarization targets to automatically generate a shorter version of the source content while retaining the most important information. as a straightforward and effective method, extractive summarization creates a summary by selecting and subsequently concatenating the most salient semantic units in a document. recently, neural approaches, often trained in an end-to-end manner, have achieved favorable improvements on various large-scale benchmarks @xcite @xcite . despite renewed interest and avid development in extractive summarization, there are still long-standing, unresolved challenges. one major problem is position bias, which is especially common in the news domain, where the majority of research in summarization is studied. in many news articles, sentences appearing earlier tend to be more important for summarization tasks @xcite , and this preference is reflected in reference summaries of public datasets. however, while this tendency is common due to the classic textbook writing style of the "inverted pyramid" @xcite , news articles can be presented in various ways. other journalism writing styles include anecdotal lead, question-and-answer format, and chronological organization @xcite . therefore, salient information could also be scattered across the entire article, instead of being concentrated in the first few sentences, depending on the chosen writing style of the journalist. as the "inverted pyramid" style is widespread in news articles @xcite , neural models would easily overfit on position-related features in extractive summarization tasks because of the data-driven learning setup which tags on to features that correlate the most with the output. as a result, those models would select the sentences at the very beginning of a document as best candidates regardless of considering the full context, resulting in sub-optimal models with fancy neural architectures that do not generalize well to other domains @xcite . additionally, according to @xcite : "content selection is not a deterministic process @xcite @xcite . different people choose different sentences to include in a summary, and even the same person can select different sentences at different times @xcite . such observations lead to concerns about the advisability of using a single human model ...", such observations suggest that individuals differ on what she considers key information under different circumstances. this reflects the need to generate application-specific summaries, which is challenging without establishing appropriate expectations and knowledge of targeted readers prior to model development and ground-truth construction. however, publicly available datasets only provide one associated reference summary to a document. without any explicit instructions and targeted applications or user preferences, ground-truth construction for summarization becomes an under-constrained assignment @xcite . therefore, it is challenging for end-to-end models to generate alternative summaries without proper anchoring from reference summaries, making it harder for such models to reach their full potential. in this work, we propose a flexible neural summarization framework that is able to provide more explicit control options when automatically generating summaries (see figure 1 ). since summarization has been regarded as a combination of subaspect functions (e.g. information, layout) @xcite , we follow the spirit of sub-aspect theory and adopt control codes on sub-aspects to condition summary generation. the advantages are two-fold: (1) it provides a systematic approach to investigate and analyze how one might minimize position bias in extractive news summarization in neural modeling. most, if not all, previous work like @xcite only focus on analyzing the degree and prevalence of position bias. in this work, we take one step further to propose a research methodology direction to disentangle position bias from important and non-redundant summary content. (2) text summarization needs are often domain or application specific, and difficult to articulate a priori what the user-preferences are, thus requiring potential iterations to adapt and refine. however, human ground-truth construction for summarization is time-consuming and laborintensive. therefore, a more flexible summary generation framework could minimize manual labor and generate useful summaries more efficiently. an ideal set of sub-aspect control codes should characterize different aspects of summarization well in a comprehensive manner but at the same time delineate a relatively clear boundary between one another to minimize the set size @xcite . to achieve this, we adopt the subaspects defined in @xcite : importance, diversity, and position, and assess their characterization capability on the cnn/daily mail news corpus @xcite via quantitative analyses and unsupervised clustering. we utilize control codes based on these three sub-aspect functions to label the training data and implement our conditional generation approach with a neural selector model. empirical results show that given different control codes, the model can generate output summaries of alternative styles while maintaining performance comparable to the state-of-the-art model; modulation with semantic sub-aspects can reduce systemic bias learned on a news corpus and improve potential generality across domains. ## in relation to other work in text summarization, most benchmark datasets focus on the news domain, such as nyt @xcite and cnn/daily mail @xcite , where the human-written summaries are used in both abstractive and extractive paradigms @xcite . to improve the performance of extractive summarization, non-neural approaches explore various linguistic and statistical features such as lexical characteristics @xcite , latent topic information (ying-lang @xcite , discourse analysis @xcite , and graphbased modeling @xcite . in contrast, neural approaches learn the features in a data-driven manner. based on recurrent neural networks, summarunner is one of the earliest neural models @xcite . much development in extractive summarization has been made via reinforcement learning @xcite , jointly learning of scoring and ranking @xcite , and deep contex-tual language models @xcite . despite much development in recent neural approaches, there are still challenges such as corpus bias resulting from the prevalent "inverted pyramid" journalism writing style @xcite , and system bias @xcite stemming from position preference in the ground-truth. however, to date only analysis work has been done to characterize the position-bias problem and its ramifications, such as inability to generalize across corpora or domains @xcite . few, if any, have attempted to resolve this long-standing problem of position bias using neural approaches. in this work, we take a first stab to introduce sub-aspect functions for conditional extractive summarization. we explore the possibility of disentangling the three sub-aspects that are commonly used to characterize summarization: position for choosing sentences by their position, importance for choosing relevant and repeating content across the document, and diversity for ensuring minimal redundancy between summary sentences @xcite during the summary generation process. in particular, we use these three sub-aspects as control codes for conditional training. to the best of our knowledge, this is the first work in applying auxiliary conditional codes for extractive summary generation. in other nlp tasks, topic information is used as conditional signals and applied to dialogue response generation @xcite and pretraining of large-scale language models (keskar et al., 2019) while sentiment polarity is used in text style transfer @xcite . in image style transfer, codes specifying color or texture are used to train conditional generative models @xcite . ## conditional neural generation in this section, we construct a set of control codes to specify the three sub-aspect features described in section 4, and label the oracle summaries constructed in section 3, then we propose a neural extractive model with a conditional learning strategy for a more flexible summary generation. ## experimental results and analysis 6.1 quantitative analysis to test the possibility of reducing position bias by conditioning summary generation, we switched the position code to 0 and compared the position of selected sentences in summaries generated by our model to the state-of-the-art baseline bertext, based on fine-tuning bert @xcite . the results show that bertext has a 50% chance of choosing the first 10% of sentences in the document. while the proposed framework still has a stronger tendency to choose sentences from the first 30% of the sentences, its position distribution is flattened compared to that of bertext. we respectively switched importance and diversity codes to 1 and categorized the generated summaries into subset of each sub-aspect function as in section 4.2. as shown in figure 8 and 9, summaries in the subset of importance and diversity weigh higher when the corresponding control codes are on. together, these results demonstrate the feasibility of our proposed framework, which can generate output summaries of alternative styles when given different control codes. ## conclusion we proposed a neural framework for conditional extractive news summarization. in particular, subaspect functions of importance, diversity and position are used to condition summary generation. this framework enables us to reduce position bias, a long-standing problem in news summarization, in generated summaries while preserving comparable performance with other standard models. moreover, our results suggest that with conditional learning, summaries can be more efficiently tailored to different user preferences and application needs.