Controlled Text Reduction (2022)

Abstract: Producing a reduced version of a source text, as in generic or focused summarization, inherently involves two distinct subtasks: deciding on targeted content and generating a coherent text conveying it. While some popular approaches address summarization as a single end-to-end task, prominent works support decomposed modeling for individual subtasks. Further, semi-automated text reduction is also very appealing, where users may identify targeted content while models would generate a corresponding coherent summary.In this paper, we focus on the second subtask, of generating coherent text given pre-selected content. Concretely, we formalize Controlled Text Reduction as a standalone task, whose input is a source text with marked spans of targeted content (“highlighting”).A model then needs to generate a coherent text that includes all and only the target information.We advocate the potential of such models, both for modular fully-automatic summarization, as well as for semi-automated human-in-the-loop use cases.Facilitating proper research, we crowdsource high-quality dev and test datasets for the task. Further, we automatically generate a larger “silver” training dataset from available summarization benchmarks, leveraging a pretrained summary-source alignment model.Finally, employing these datasets, we present a supervised baseline model, showing promising results and insightful analyses.

## introduction abstractive text summarization takes one or more documents as input and aims at generating an accurate and coherent summary from it. it requires both locating salient information in the input and then generating a concise text covering it. while some modern state-of-the-art abstractive summarization models treat the task as a single end-to-end task, it has been common practice for summarization models to separate the salience detection phase from the text generation phase @xcite @xcite , with renewed popularity in recent years @xcite @xcite @xcite @xcite . but, though those proposed techniques comprised distinguishable subtasks, evaluation was performed on the whole summarization pipeline, rather than optimizing each step separately. in this paper, we focus on the text generation step, while addressing it as a standalone task at the sub-sentence level. to that end, we introduce a new task which we denote controlled text reduction. the task takes as input a document with pre-chosen salient spans in it, which we will henceforth call highlights. a model is then expected to reduce the document to a smaller coherent text which covers all and only the highlighted content, i.e., consolidating the highlighted spans into a fluent and coherent passage, as exemplified in figure 1 . this task poses a challenge, as it requires generating fluent and grammatical text from non-consecutive spans while keeping it faithful to the source document. hence, to balance the coherency and faithfulness constraints, models will be expected to use the context document to fill in implied details and to properly connect the different spans. focusing on this task can facilitate greater control over the generated text. it could lead to a modular summarization pipeline, where text-generation models can be trained once, and then used with different content selections to accommodate different needs. for example, we may envision a user (e.g., a student) pre-selecting the desirable textual content (either manually or via a designated model) while figure 1 : an example of an input, consisting of a source document and highlights (left), and the generated passage covering the highlighted content while preserving coherence (right). such highlights in realistic use cases may be produced either by a human user or by a salience detection model. to promote research on the advocated text reduction task, we first develop a suitable controlled crowdsourcing methodology, following @xcite , and apply it to produce high-quality dev and test datasets ( §4). next, we automatically generate a larger training dataset, by aligning propositional units of information @xcite , extracted with openie @xcite , between source documents and their summaries ( §5). we use this data to train an abstractive supervised model, and evaluate its performance against our testset while comparing it to an extractive reference baseline, which simply concatenates the highlights. we also perform analyses where we manipulate the highlights and show that the addition of highlights to a supervised model is helpful in steering the model toward the pre-selected content, in addition to improving overall faithfulness and fluency ( §8). hence, the contribution of this paper is manifold: 1. proposing the "controlled text reduction" task as a standalone module in automated or semi-automated use cases. 2. defining an intuitive and easy-to-reproduce crowd-sourcing method for the task. 3. constructing the first data suite for the task, including crowd-sourced dev and test sets and an automatically-generated train set. 4. developing a supervised baseline model for future work. ## background in this section, we briefly review related work and discuss the limitations of their framing. as mentioned above, much of the related previous work focused primarily on end-to-end summarization @xcite @xcite @xcite , with the vast majority of related datasets aimed at endto-end summarization @xcite @xcite , with only a source document as input. on the other hand, research on leveraging control through the injection of pre-chosen (rather than learned) signals in the seq-to-seq scenario focused mostly on semantic and syntactic signals, and also almost exclusively targeted machine translation models @xcite @xcite @xcite . attempts to leverage some control over the generation step in summarization received attention in recent years in the form of query-focused summarization @xcite @xcite and keywordsfocused summarization @xcite , with a few recently published corresponding datasets @xcite @xcite . a similar trend tried to leverage control through the addition of a planning step @xcite . although these lines of research allowed for some control over salience, this control was limited and mostly focused on biasing the summary's topic, style, or structure. the prevailing way to treat summarization in earlier works was to separate the salience detection phase from the text generation phase @xcite @xcite ), yet the evaluation was performed on the whole pipeline. some recent work focused on salience detection @xcite @xcite , whereas the generation step has mostly been explored in a fullsentence-fusion setting @xcite @xcite , rather than in a sub-sentence level. @xcite took it one step further, leveraging sentence fusion through a fine-grained content selection algorithm. but, though they did perform some analysis of this additional step by comparing different salience detection strategies, his evaluation focused on the full pipeline, similarly to his predecessors. there has also been some work on extracting salient information in source documents in the form of highlights @xcite . yet, though acknowledging the full potential of using highlights to mark salient information in the source document, it mainly focused on the process of obtaining these highlights, overlooking its actual usage in subsequent generation tasks, and in summarization in particular. moreover, these lines of work focused solely on automatic highlight detection, lacking any crowdsourced annotation scheme. there has also been work that pre-identified salient parts as input to the generation phase @xcite @xcite but, contrary to our work, the salience detection and generation tasks were addressed and evaluated jointly, without assessing the quality of each individual task. all those research directions recognized the potential of separating the summarization task into subtasks and performing each subtask explicitly. however, they all evaluated the subtasks jointly, and in doing so overlooked the potential laying in the optimization and characterization of each task individually, and specifically the generation task given content-selection. in this work, we propose to isolate the generation task given pre-selected content, treating it as a stand-alone task, thus promoting focused evaluation and model designing. ## task definition we define the controlled text reduction task as follows. given a document and a set of marked spans within that document, denoted as highlights, produce a coherent output text encompassing only the information provided within those highlights (see figure 1 ). the desirable output should adhere to two requirements beyond coherency: (1) its content has to be derived from the highlights alone, keeping any additional document premises to the minimum required for coherency; (2) the output has to retain all of the details covered by the highlighted spans. such requirements give rise to many interesting challenges, such as recognizing the connecting thread between disparate spans and faithfully representing the information contained within them. we forgo a strict definition for a highlighted span and allow possibly marking sub-sentence elements: an entity or a clause, even discontinuous descriptions of these (e.g., the last two highlights in figure 1 ). hence, the input highlights may be disconnected in both their surface realization (i.e. grammatically when the summary sentence is fully highlighted, we proceed to the next sentence, and so on. in this example, the summary consists of two facts, but steps 1 and 2 can be repeated as needed per sentence, until all its propositions (facts) are covered. figure 1 features an input-output example. the output covers exclusively and completely the highlighted information while using the source document's context to connect the disparate spans. ## gold dataset for evaluation we leverage different summarization datasets to annotate a high-quality dataset for the evaluation of controlled-reduction systems. in summarization, every summary arises from a set of salient document spans. exploiting this in our annotation process, we wish to "reverse-engineer" each summary and locate the spans in the document that led to its construction. this significantly reduces the annotation complexity and load, instead of compiling a new text given a set of highlighted spans, an annotator has to highlight document spans given the output text (i.e. the summary). to create our development and test partitions we sample 121 and 108 unique documents from duc 2001 and 2002 single-document-summarization (sds) datasets foot_1 respectively. each document is accompanied by up to 4 different reference summaries (with an average of 2.14 summaries per document), resulting in a total of 488 unique documentsummary pairs (see ## train dataset to acquire a larger dataset for training supervised models, we opt for an automatic approach to extract highlights. for that, we employ the superpal model @xcite , a proposition-based summary-source alignment model trained on a sentence alignment dataset @xcite @xcite based on the pyramid evaluation method @xcite . the model extracts propositions from the document and the summary, and then uses a roberta encoder fine-tuned on mnli and augmented with a binary classification layer to determine which propositions are aligned. we run the pre-trained superpal model on the sds duc 2001 and 2002 document-summary pairs that were not already manually annotated (see §4), consisting of 1911 such pairs (see ## baseline models we experiment with two methods for the controlled text reduction task: a supervised model, whose input is the full document, supplemented with indications of the highlighted spans ( §6.1) and another supervised model that receives as input only a concatenation of the highlights, without the surrounding context ( §6.2). both models are trained on our automatically-generated train dataset ( §5). ## experimental setup baseline models we use our training dataset ( §5) to finetune our two led variants ( §6). we employ the cnn-dm dataset together with our duc trainset for initial fine-tuning, which is then followed by further finetuning on the duc trainset alone. we avoid using the cnn-dm dataset in the latter finetuning phase since its quality is notably lower compared to the duc dataset. specifically, cnn-dm was generated automatically, in comparison to the expert-written summaries in duc, and it consists of standalone bullet points, lacking the desired discourse properties and flow of natural text. to avoid overfitting on the cnn-dm dataset, which is much larger than duc, we experimented with using only fractions of the cnn-dm data. optimal performance was achieved when using the full cnn-dm data for the initial finetuning of the led h model ( §6.1), while for the led only-h model it was best to finetune only on the duc data, avoiding the cnn-dm data altogether. in the led only-h , we preprocess our input, extracting the highlights and then using a dot (followed by a space) to separate spans originating in different sentences, and a white space otherwise. to model the highlights in the led h setting, we follow @xcite and add to the vocabulary two special tokens, <@xmath0, which are inserted as vec-tors into the source documents' embedding layer at the beginning and end of each highlighted span. also, we combine led's local attention with its global attention mechanism. as the global attention adds bias to the designated tokens, we mark @xmath1. our motivation stems from the assumption that allowing all the highlight tokens to attend to one another (through the symmetry of the global attention) will encourage the model to fuse the information they are attached to, under the assumption that the highlighted spans are related. though led supports inputs with up to 16384 tokens, for our purposes we limit it to 4096 tokens (see we also examined other techniques to represent the highlights @xcite @xcite , but as they introduced dependencies between their salience detection and generation components, we found them less fitting in our setting. as a reference point, we compare the abstractive models to an extractive text generated by simply concatenating the highlights, as described previously (i.e., the input to led only-h ). this version serves to demonstrate the necessity of our new abstractive task formulation, since without a system that bridges disparate texts, the concatenated spans are often unintelligible. no highlights in addition to the two baseline models for our text reduction task, we also examine led in a standard no-highlight summarization setting, where it is finetuned and evaluated on the original document without any highlights. in the absence of highlights, the global attention becomes unnecessary, hence this variant incorporates solely local attention. this no-highlight variant of led, denoted led nh , matches the classic summarization setting and provides insights into the ability of the model to pick up the highlighting signals. when optimizing the amount of cnn-dm data to use in the initial finetuning phase, as described above for the baseline models, we found it optimal to use 5% of the cnn-dm data. ## analysis and results first, we present the fluency results to validate the necessity of our task setting. as expected, it arises from in conclusion, to evaluate future progress on the text reduction task, we firstly propose combining manual evaluation of fluency, requiring sufficient fluency to make models acceptable, along with automatic evaluation of content preservation via common measures for this purpose such as rouge. while we also inspected less standard automatic evaluation measures, for both fluency ## conclusion in this paper, we promote the separation of the summarization task into the salience-detection and text-generation steps. we foresee applications where salient phrases will be highlighted by an avid reader, or selected by a model specialized in some domain, while a more general-purpose model would reformulate the disparate pieces into a coherent text. thus, we argue that controlled text reduction, the second step of summarization, is an interesting and useful research goal in its own right. to bolster the task, we release a high-quality evaluation dataset and a heuristically-generated training data, evaluation protocol, and the first baseline model. the latter clearly shows how the generated summary text benefits from the added salient span signals. future works may expand this to include multi-document settings in order to accommodate the task to a broader range of applications, and also focus on designing better evaluation metrics for the task.