Revisiting Sentence Union Generation as a Testbed for Text Consolidation (2023)

Abstract: Tasks involving text generation based on multiple input texts, such as multi-document summarization, long-form question answering and contemporary dialogue applications, challenge models for their ability to properly consolidate partly-overlapping multi-text information. However, these tasks entangle the consolidation phase with the often subjective and ill-defined content selection requirement, impeding proper assessment of models’ consolidation capabilities. In this paper, we suggest revisiting the sentence union generation task as an effective well-defined testbed for assessing text consolidation capabilities, decoupling the consolidation challenge from subjective content selection. To support research on this task, we present refined annotation methodology and tools for crowdsourcing sentence union, create the largest union dataset to date and provide an analysis of its rich coverage of various consolidation aspects. We then propose a comprehensive evaluation protocol for union generation, including both human and automatic evaluation. Finally, as baselines, we evaluate state-of-the-art language models on the task, along with a detailed analysis of their capacity to address multi-text consolidation challenges and their limitations.

## introduction in order to acquire knowledge on a new subject or find answers to complex questions, it is often necessary to consult multiple sources of written information. while information provided in a single document is usually consistent, textual materials from various sources often use different language expressions, which may vary in terms of level of specificity, to convey similar information. an illustration of this phenomenon can be seen in figure 1 . in this paper, we aim to address the process of combining such multiple partially overlapping textual 1 our data and code is available at: https://github.com/ eranhirs/sentence_union_generation text consolidation plays a crucial role in almost any text-based information access application, such as multi-document summarization (mds) @xcite , long-form question answering @xcite , and contemporary dialogue applications @xcite . it is important to point out here that content selection and consolidation manifest two distinct sub-tasks in such applications, where the former involves identifying the sought information in the source texts, based on considerations such as salience and user needs. consolidation, on the other hand, involves merging the selected information into a coherent output text. accordingly, we suggest that each sub-task deserves separate investigation, while focusing in this paper on the consolidation task, manifested as information union. this approach enables targeted investigation of information union capabilities of models, while enabling modular architectures, where an effective information consolidation model can be paired with different content selec-tion models and strategies, whether fully-automatic or interactively involving a user in the loop. to achieve a more controlled research environment, a sentence fusion task was introduced, which fuses a set of sentences into a single sentence @xcite @xcite . however, being similar to summarization, the general sentence fusion task is ill-defined, because it allows for subjective salience-based content selection decisions @xcite . in contrast, the sentence union generation task is strictly defined as generating a sentence that contains exactly all information from the source sentences (see fig. 1 ). while identifying the union task to be more attractive due to its more objective and semantically challenging nature, we found that datasets for this topic are relatively scarce @xcite @xcite , none of them sufficiently addressing the text consolidation setting. consequently, we revisit the sentence union generation task and propose that it can be used as an effective generic testbed for text consolidation. compared to the sentence intersection task, the union task is more challenging, as it requires merging both joint and disjoint information in the output and hence provides a more complete testbed for text consolidation. our input format is rich and challenging enough, as shown in our analyses, to support research on information merging models. further, this setting may already be of practical use for downstream text generation tasks, for example when combined with sentence compression or decontextualization models. our contributions are outlined as follows: (1) we suggest focusing on sentence union generation as a resource for studying cross-text consolidation capabilities, and point out that properly identifying informational relations between pairs of sentences is necessary for proper consolidation; (2) we provide the largest union fusion dataset to date, while proposing a controlled annotation protocol and interface for careful creation of a sentence union corpus; (3) we suggest evaluation protocols to assess the quality of a generated sentence union, accompanied by automatic metrics that can be used for comparing multiple systems; (4) we provide empirical results on the abilities of prominent neural generative models to address the union task, assessing their capabilities and limitations. ## background in multi-document summarization (mds) @xcite multipletexts are summarized into a single, shorter text. in a more controlled variant of mds, the task requires the fusion of partly-overlapping sentences @xcite @xcite . generally, the sentence fusion task included a saliency detection (or importance) component which requires identifying which pieces of information to preserve in the fused output. as a result, sentence fusion is generally ill-defined, as different possible content selections may be valid, making the task subjective to varying necessities of a user @xcite . its output could be seen as covering a "loose" intersection of the content of two sentences. ## task formulation the input for our sentence union task consists of two related sentences whose content partly overlap. the output union is then defined as a single sentence that follows two conditions: (a) it contains exactly the information from the two input sentences, and (b) it does not include any redundancies in its content. condition (a) implies that there cannot be any information missing from the union that is mentioned in the source sentences, while at the same time the union cannot contain information that is not mentioned in the source sentences (i.e., hallucinations). condition (b) implies that the union must avoid repetition of any units of information stemming from the source sentences, even if they are conveyed in different lexical terms. notably, the semantic content of the output union (condition (a)) can be defined objectively in strict textual entailment terms. formally, given an input of two related sentences s 1 and s 2 , and their union u, u should satisfy u |= s 1 , u |= s 2 and s 1 + s 2 |= u, where |= denotes textual entailment and + denotes concatenation of the two sentences. this definition, however, does not cover condition (b) of avoiding redundancies. identifying relevant informational links is crucial for producing a union, as demonstrated by the example in fig. 2 . we observe three types of relations between information units in the source sentences that affect the content of the resulting unit: (1) equivalent content, (2) uni-directional entailing content, and (3) disjoint content. equivalent content, such as lexical equivalence or paraphrases, needs to be identified and included exactly once in the union to avoid redundancy. uni-directional entailing content pertains to aligned text spans where one span can be implied from the other. in this case, only the entailing text unit should be included: including both spans would be redundant, while including only the less specific mention would result in missing information. disjoint content must be included in the union as it provides distinct information not mentioned in the other sentence. for example, in fig. 2 , sentence 1 mentions the reason for firing weightman while sentence 2 mentions that harvey resigned, each providing distinct information. in addition, according to our annotation scheme, we assume that the date of the publication is known, which means that when a phrase such as "the previous thursday" is mentioned, we can infer the specific date. thus, the text spans "on march 1st" and "the previous thursday" are equivalent, while "francis harvey" in sentence 1 is more specific than the text span "harvey" in sentence 2. by considering these three types of relations, a proper union can be produced. as noted earlier, we see the union generation task as a more comprehensive setup for information consolidation than the intersection generation task foot_0 . this is because the union output should combine all the content from both source sentences, while the output of the intersection task does not include information mentioned in only one of the sentences. as a result, the union is more informative than the intersection, which makes it more representative for downstream multi-text tasks requiring information consolidation, aiming to create an efficient, nonrepetitive output text. ## dataset analysis and assessment in the following subsections, we report various analyses of the quality and other properties of our dataset. dataset split statistics appear in moreover, our analysis of learning curves (see appendix g) suggests that the size of our training dataset is sufficient, and further expansion may not yield significant benefits. ## baseline models we present baseline models, aiming to test neural pretrained language models' for their ability to implicitly recognize relevant informational relations between input sentences and properly create their union. fine-tuned models as our first type of baseline we fine-tune a large pre-trained sequenceto-sequence model using our data. to that end, we picked two strong models: t 5 large @xcite , which is commonly applied to endto-end text generation tasks @xcite , and primera @xcite , which was pretrained in a cross-document fashion @xcite and achieves state-of-the-art results over multi-document summarization datasets. this makes this model appealing for our sentence fusion task, where the two sentences originate in different documents. see app. f for information about training details. in-context learning another current baseline approach is in-context learning, in which the instructions and examples to the task are provided as input (the prompt) at inference time to very large pre- score content redundancy 1 substantial information is missing. substantial information is repeated. 2 some information is missing. some information is repeated. 3 minor details are missing. minor details are repeated. 4 nothing is missing. nothing is repeated. ## model evaluation protocols we evaluate our baseline systems both through human evaluation ( §7.1) and with automatic metrics ( §7.2) suitable for the task, which can generally be used in the development cycles of union generation systems ( §7.2). ## conclusions in this paper, we advocate for using the sentence union task as a testbed for multi-text consolidation. we release a realistic dataset, together with a set of analyses that show that the dataset is of high quality, and challenging for multi-document consolidation efforts. we evaluate the performance of state-of-the-art pretrained large language models on text consolidation, where our findings suggest key challenges for future research. future research may expand upon our dataset to include consolidation beyond 2 input sentences, and may examine the use of explicit text consolidation structures for improving multi-text consolidation in large language models.