Multi-Review Fusion-in-Context (2024)

Abstract: Grounded text generation, encompassing tasks such as long-form question-answering and summarization, necessitates both content selection and content consolidation. Current end-to-end methods are difficult to control and interpret due to their opaqueness.Accordingly, recent works have proposed a modular approach, with separate components for each step. Specifically, we focus on the second subtask, of generating coherent text given pre-selected content in a multi-document setting. Concretely, we formalize Fusion-in-Context (FiC) as a standalone task, whose input consists of source texts with highlighted spans of targeted content. A model then needs to generate a coherent passage that includes all and only the target information.Our work includes the development of a curated dataset of 1000 instances in the reviews domain, alongside a novel evaluation framework for assessing the faithfulness and coverage of highlights, which strongly correlate to human judgment. Several baseline models exhibit promising outcomes and provide insightful analyses.This study lays the groundwork for further exploration of modular text generation in the multi-document setting, offering potential improvements in the quality and reliability of generated content. Our benchmark, FuseReviews, including the dataset, evaluation framework, and designated leaderboard, can be found at https://fusereviews.github.io/.

## introduction grounded text generation is the task of producing a passage from source texts, where the output is anchored around use-case-dependent spans within the source texts. although such a task involves two distinct subtasks -identifying relevant spans and fusing them -it is commonly handled in an end-toend approach, recently by using large language models (llms) @xcite @xcite . while effective, this approach often lacks flexibility and control over the generation process, given its opaque nature. addressing this, @xcite recently advocated splitting grounded generation tasks into their two subtasks, and particularly focused on the fusion step. they introduced controlled text reduction (ctr), a task where pre-selected spans in a source document ('highlights') are fused into a coherent text that exclusively covers the spans. this approach enhances control and modularity in text generation, enabling a single ctr model to work with various content selection strategies and user preferences, applicable in different contexts like summarization or long-form question-answering. it could also support human-in-the-loop scenarios for tailored outputs based on user preferences, as explored in @xcite . further, the direct access to the highlights that contribute to the output facilitates attributed generation @xcite , where models can cite source spans for generated text. despite its benefits, ctr's focus on single-input scenarios limits its applicability to the broader, and more complex, multi-document setting. in this paper, we bridge this gap and extend the task to the multi-document setting. for that, we introduce the task of fusion-in-context (fic), a generalized version of the ctr task, which processes multiple documents with pre-selected highlights, and aims to fuse them into a coherent, non-redundant text covering all and only the highlighted content, as demonstrated in figure 1 . in addition to the challenges of the single-input ctr task, including coreference resolution and proper discourse for coherence, the multi-document setting also requires handling repetitive, and sometimes conflicting information @xcite . specifically, our work focuses on the business reviews domain, where contradicting opinions are more prevalent than in other more fact-oriented domains, such as news. there is a metro stop within a couple blocks, which easily gets you to some prime shopping and restaurant areas. the rooms are not large, but the very high ceilings make the rooms feel roomy. we look forward to a return visit to montreal and to this charming hotel. the location of the hotel was highly appreciated, especially its proximity to the subway, though one reviewer said that there was too much noise in the area. also, most reviewers appreciated the rooms, stating they were comfortable, albeit small. ## background grounded text generation, an area focusing on generating text from source documents, requires identifying relevant task-specific details within the inputs, such as salient content for summarization, as well as their coherent fusion. this field includes tasks like long-form question-answering @xcite , summarization @xcite @xcite , and dialogue systems @xcite @xcite , with most related datasets aimed at end-to-end training @xcite @xcite . despite the prevalence of end-to-end systems, there has been a growing trend towards decom-posed pipeline approaches, particularly in summarization, with several recent studies focusing on content selection @xcite @xcite . conversely, content fusion was largely explored at the full-sentence fusion level @xcite , with less emphasis on sub-sentence fusion. recently, @xcite ) have proposed a distinct separation of content selection from fusion, treating each as an independent task. they specifically concentrated on fusion, defining it as a standalone task termed controlled text reduction (ctr). this task takes as input pre-selected spans, or 'highlights', within an input document, and requires a coherent merging of all the highlighted content, and nothing else. they also released a designated dataset and several ctr models showing strong adherence to these highlights. while these studies acknowledged the benefits of decomposing grounded generation to subtasks, they mainly focused on single-document inputs. our work builds on this decomposed approach, extending it to multi-document settings, which introduce new challenges such as managing longer inputs, handling redundant highlights @xcite , and dealing with potentially conflicting facts or opinions @xcite . additionally, previous ctr studies assessed highlight adherence by comparing outputs with the concatenated highlights, using lexical metrics like rouge @xcite and meteor @xcite , and semantic metrics like bertscore @xcite . these methods, while suitable for single-input scenarios, are less effective for multi-document contexts where redun-dant and conflicting highlights are more prevalent. additionally, these approaches did not distinctly evaluate faithfulness and coverage of highlights, typically assessing them jointly, with only manual evaluation for separate evaluation. addressing this, we explore more suitable metrics focusing separately on faithfulness and coverage of fused texts, inspired by recent progress in this area. several recent studies have used natural language inference (nli) models for faithfulness evaluation @xcite . there have also been advances in utilizing llms to evaluate faithfulness in a zero-shot setting with nli-style prompts @xcite @xcite , or after fine-tuning on synthetic data for faithfulness evaluation @xcite @xcite ). yet, these works mainly targeted overall source text faithfulness rather than to specific segments. moreover, they have not been widely applied to assess coverage, which has traditionally been evaluated using lexical metrics @xcite or manual evaluation @xcite . in our work, we adapt these methods to our highlights-focused setting, both for faithfulness and coverage, and assess their effectiveness. ## task definition the fusion in context (fic) task is defined as the process of synthesizing a coherent text from a given set of documents, specifically focusing on pre-selected spans within these documents, referred to as highlights. formally, given a document set d with marked spans @xmath0, h 2 , ..., h n } (such that h i may be non-contiguous), a coherent and non-redundant passage f is generated, adhering to the following two criteria: (1) highlight faithfulness f must be collectively entailed by the content in h, adding only minimal non-highlighted content required for coherence; (2) highlight coverageeach h @xmath1, either explicitly, or via a generalized reference. for instance, if a highlight states "the place serves great sushi", the output should either directly mention "great sushi" or refer to it in more general terms, such as "great food". moreover, the task permits the abstraction and aggregation of multiple highlights into a single, synthesized statement. for example, separate highlights noting "the beds were clean", "the bathrooms were spotless", and "the windows were clean" could be collectively abstracted to a general statement like "the rooms are clean". overall, the goal is to produce a faithful, non-redundant and non-omissive, yet potentially abstractive and aggregated, fusion of the highlighted content. next, we describe a dataset ( §4) and an evaluation framework ( §5) that comply with the task definition. these are released as the fusereviews benchmark for the fic task. ## dataset for fic to comply with the task definition, an instance in a fic dataset is expected to be a document set d with marked spans @xmath2, h 2 , ..., h n }, and a corresponding fused text f . to compile such data, we leverage existing multi-document summarization datasets and extract high-quality fic instances via controlled crowdsourcing @xcite , by adapting the method from @xcite to the multi-text setting, and the business reviews domain. ## evaluation framework consistent with the task definition in §3, a passage produced by a model as a fusion of highlights within source documents must uphold several criteria. (1) faithfulness: it must only contain information from the highlights; (2) coverage: it must cover all the information in the highlights, be it in an explicit, generalized, or aggregated form; (3) coherence and redundancy: it must convey the information in a well-structured and non-redundant form. in this section, we suggest several automatic metrics for faithfulness and coverage, and assess their effectiveness by correlating to human scores that we collected. coherence and redundancy are measured using manual evaluation. ## experiments 6.1 experimental setup we examine several baseline models for solving the fic task. the input to a model is a document set with spans marked within the documents (highlights), and the model is trained to generate a fused passage around the highlights. models with full input. using the training set of our dataset, we finetune a large language model, marking the highlights in the input via designated mark-ups, following @xcite . specifically, we finetune a flan-t5-large model @xcite , that exhibited enhanced performance in tasks requiring constrained generation @xcite . we will refer to this model as flan-t5 h ('h' for 'highlights'). we develop an additional variant of flan-t5 h , which we further finetune using reinforcement learning (rl), following the method in @xcite . 2022) combined with a dual-reward policy @xcite , alternating between our nli-based faithfulness and trained coverage metrics ( §5) as rewards. we also examine the performance of a one-shot gpt-4 model (openai, 2023), guided with an example of the task. 10 models with highlights only. to reveal the importance of the surrounding context, we also train a flan-t5-large model only with a concatenation of the highlights as the input (excluding surrounding context). we denote this variant flan-t5 only-h . ## conclusion in this paper, we further promote the decomposition of grounded text generation as presented in @xcite , extending it to the multidocument setting. to that end, we introduce the fusion-in-context (fic) task, an extension of the task from @xcite which focuses on the content fusion step, to the multi-document setting. the fic setting facilitates employing a single general-purpose fusion model for diverse content selection needs, capturing the challenges of repetitiveness and contradictions in source documents. it also supports interactive, user-driven generation, by allowing users to choose personalized content for the fic module to merge into a customized passage. moreover, direct access to the pre-selected "highlights" can facilitate attributed generation, where the pre-selected segments also serve as supporting cited content for the fused text. to advance the task, we introduce the fusereviews benchmark, which includes a high-quality dataset, an evaluation framework for faithfulness and coverage of selected spans, and several baseline models to stimulate further research and exploration. future work may include expanding the fic task to other multi-input contexts, e.g., the news do-main. we also plan to investigate ways to leverage the built-in traceability of the output text's origin, namely the highlights, for facilitating attributed generation.