Nutri-bullets Hybrid: Consensual Multi-document Summarization (2021)

Abstract: We present a method for generating comparative summaries that highlight similarities and contradictions in input documents. The key challenge in creating such summaries is the lack of large parallel training data required for training typical summarization systems. To this end, we introduce a hybrid generation approach inspired by traditional concept-to-text systems. To enable accurate comparison between different sources, the model first learns to extract pertinent relations from input documents. The content planning component uses deterministic operators to aggregate these relations after identifying a subset for inclusion into a summary. The surface realization component lexicalizes this information using a text-infilling language model. By separately modeling content selection and realization, we can effectively train them with limited annotations. We implemented and tested the model in the domain of nutrition and health – rife with inconsistencies. Compared to conventional methods, our framework leads to more faithful, relevant and aggregation-sensitive summarization – while being equally fluent.

## introduction articles written about the same topic rarely exhibit full agreement. to present an unbiased overview of such material, a summary has to identify points of consensus and highlight contradictions. for instance, in the healthcare domain, where studies often exhibit wide divergence of findings, such comparative summaries are generated by human experts for the benefit of the general public. 2 ideally, this capacity will be automated given a large number of relevant articles and continuous influx of new ones that require a summary update to keep it current. however, standard summarization architectures cannot be utilized for this task since the amount of comparative summaries is not sufficient for their training. in this paper, we propose a novel approach to multi-document summarization based on a neural interpretation of traditional concept-to-text generation systems. specifically, our work is inspired by the symbolic multi-document summarization system of @xcite which produces summaries that explicitly highlight agreements, contradictions and other relations across input documents. while their system was based on human-crafted templates and thus limited to a narrow domain, our approach learns different components of the generation pipeline from data. to fully control generated content, we frame the task of comparative summarization as concept-totext generation. as a pre-processing step, we ex-tract pertinent entity pairs and relations (see figure 1 ) from input documents. the content selection component identifies the key tuples to be presented in the final output and establishes their comparative relations (e.g., consensus) via aggregation operators. finally, the surface realization component utilizes a text-infilling language model to translate these relations into a summary. figure 1 exemplifies this pipeline, showing selected key pairs (marked in bold), their comparative relation -contradiction (rows 1 &3 and rows 4&5 conflict), and the final summary. 3 this generation architecture supports refined control over the summary content, but at the same time does not require large amounts of parallel data for training. the latter is achieved by separately training content selection and content realization components. since the content selection component operates over relational tuples, it can be robustly trained to identify salient relations utilizing limited parallel data. aggregation operators are implemented using simple deterministic rules over the database where comparative relations between different rows are apparent. on the other hand, to achieve a fluent summary we have to train a language model on large amounts of data, but such data is readily available. in addition to training benefits, this hybrid architecture enables human writers to explicitly guide content selection. this can be achieved by defining new aggregation operators and including new inference rules into the content selection component. moreover, this architecture can flexibly support other summarization tasks, such as generation of updates when new information on the topic becomes available. we apply our method for generating summaries of pubmed publications on nutrition and health. typically, a single topic in this domain is covered by multiple studies which often vary in their findings making it particularly appropriate for our model. we perform extensive automatic and human evaluation to compare our method against state-of-the-art summarization and text generation techniques. while seq2seq models receive competent fluency scores, our method performs stronger on task-specific metrics including relevance, content faithfulness and aggregation cognisance. our method is able to produce summaries that receive 3 we compare the selected content with other entries in the database, identifying two contradictions. an absolute 20% more on aggregation cognisance, an absolute 7% more on content relevance and 7% on faithfulness to input documents than the next best baseline in traditional and update settings. recent language modeling approaches @xcite @xcite can also be extended for text completion. our work is a text-infilling language model where we generate words in place of relation specific blanks to produce a faithful summary. prior work @xcite @xcite ) on text generation also control aspects of the produced text, such as style and length. while these typically utilize tokens to control the modification, using prototypes to generate text is also very common @xcite @xcite . in this work, we utilize aggregation specific prototypes to guide aggregation cognizant surface realization. data-to-text summrization traditional approaches for data-to-text generation have operated on symbolic data from databases. @xcite ; @xcite ; @xcite introduce two components of content selection and surface realization. content selection identifies and aggregates key symbolic data from the database which can then be realized into text using templates. unlike modern data-totext systems @xcite @xcite these approaches capture document consensus and aggregation cognisance. while the neural approaches alleviate the need for human intervention, they do need an abundance of parallel data, which are typically from one source only. hence, modern techniques do not deal with input documents' consensus in low resource settings. ## method our goal is to generate a text summary y for a food from a pool of multiple scientific abstracts x. in this section, we describe the framework of our nutribullets hybrid system, illustrated in figure 2 . ## summary and update setting in this section we describe the setting of summary updates. in a real world setting, we would often receive new input documents such as scientific studies about the same subject which necessitate a change in an old summary. in context of our food and health summarization task, the goal is to update an old summary about a food and health condition on receiving results from new scientific studies from pubmed. our model can accommodate this scenario fairly easily. we describe the minor changes to the content selection and aggregation and surface realization models for such a setting. we are provided an original summary and can extract it's content c and can also construct the mini-database x g from the text of the new documents. we identify the aggregation between the new studies' x g and original summary's content c first. depending on the aggregation identified, 6 summaries in our training data are labelled with o y m as belonging to one of the four categories of under-reported, population scoping, contradiction or agreement to accommodate such training.