Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation (2023)

Abstract: We present PeerSum, a novel dataset for generating meta-reviews of scientific papers. The meta-reviews can be interpreted as abstractive summaries of reviews, multi-turn discussions and the paper abstract. These source documents have a rich inter-document relationship with an explicit hierarchical conversational structure, cross-references and (occasionally) conflicting information. To introduce the structural inductive bias into pre-trained language models, we introduce RAMMER (Relationship-aware Multi-task Meta-review Generator), a model that uses sparse attention based on the conversational structure and a multi-task training objective that predicts metadata features (e.g., review ratings). Our experimental results show that RAMMER outperforms other strong baseline models in terms of a suite of automatic evaluation metrics. Further analyses, however, reveal that RAMMER and other models struggle to handle conflicts in source documents, suggesting meta-review generation is a challenging task and a promising avenue for further research.

## introduction text summarization systems need to recognize internal relationships among source texts and effectively aggregate and process information from them to generate high-quality summaries @xcite . it is particularly challenging in multidocument summarization (mds) due to the complexity of the relationships among (semi-)parallel source documents @xcite . however, existing mds datasets do not provide explicit interdocument relationships among the source documents @xcite @xcite although interdocument relationships may also exist in nature and should be considered in methodology @xcite automatically generated meta-review publicreview-1 authorcomment-1 officialreview-1 paper abstract authorresponse-1 authorresponse-3 authorresponse-4 authorresponse-2 officialresponse-1 officialresponse-2 publicresponse-1 public review thread official review thread paper abstract thread author comment thread et @xcite . this makes it hard to research interdocument relationship comprehension for information integration and aggregation in abstractive text summarization. to enable this, we introduce peersum, an mds dataset for automatic meta-review generation. we formulate the creation of meta-reviews as an abstractive mds task as the meta-reviewer needs to comprehend and carefully summarize information from individual reviews, multi-turn discussions between authors and reviewers and the paper abstract. from an application perspective, generating draft meta-reviews could serve to reduce the workload of meta-reviewers, as meta-reviewing is a highly timeconsuming process for many scientific publication venues. peersum features a hierarchical conversational structure among the source documents which includes the reviews, responses and the paper abstract in different threads as shown in figure 1 . it has several distinct advantages over existing mds datasets: (1) we show that the meta-reviews are largely faithful to the corresponding source documents despite being highly abstractive; (2) the source documents have rich inter-document relationships with an explicit conversational structure; however, there is limited study on abstractive mds methods that can recognize relationships among source documents. the most promising approaches are based on graph neural networks @xcite , but they introduce additional trainable parameters, and it is hard to find effective ways to construct graphs to represent source documents. to make pre-trained language models have the comprehension ability of complex relationships among source documents for mds, we propose rammer, which uses relationship-aware attention manipulation -a lightweight approach to introduce an inductive bias into pre-trained language models to capture the hierarchical conversational structure in the source documents. concretely, rammer replaces the full attention mechanism of transformer @xcite with sparse attention that follows a particular relationship in the conversational structure (e.g., the parentchild relation). to further improve the quality of generated meta-reviews by utilising the metadata information, rammer is trained with a multi-task objective to additionally predict source document types, review ratings/confidences and the paper acceptance outcome. we conduct experiments to compare the performance of rammer with a number of baseline models over automatic evaluation metrics including the proposed evaluation metric based on predicting the paper acceptance outcome and human evaluation. we found that rammer performs strongly, demonstrating the benefits of incorporating of the conversational structure and the metadata. further analyses on instances with conflicting source documents, however, reveal that it still struggle to recognise and resolve these conflicts, suggesting that meta-review generation is a challenging task and promising direction for future work. ## the rammer model we now describe rammer, a meta-review generation model that captures the conversational structure in the source documents (section 4.1) and uses a multi-task objective to leverage metadata information (section 4.2). rammer is built on an encoder-decoder plm to automatically generate a summary/meta-review y from a cluster of source documents d; its overall architecture is presented in figure 2 . the input to rammer is the concatenation of all source documents (d) and we insert a @xmath0. 7 ## conclusion we introduce peersum, an mds dataset for metareview generation. peersum is unique in that the summaries (meta-reviews) are grounded in the source documents despite being highly abstractive, it has a rich set of metadata and explicit inter-document structure, and it features explicit conflicting information in source documents that the summaries have to handle. in terms of modelling, we propose rammer, an approach that extends transformer-based pre-trained encoderdecoder models to capture inter-document relationships (through the sparse attention) and metadata information (through the multi-task objective). although rammer is designed for meta-review generation here, our approach of manipulating attention to incorporate the input structure can be easily adapted to other tasks where the input has interdocument relationships. compared with baselines over a suite of automatic metrics and human evaluation, we found that rammer performs favourably, outperforming most strong baselines consistently. that said, when we assess how well rammer does for situations where there are conflicting informa-tion/viewpoints in the source documents, the outlook is less encouraging. we found that rammer fail to recognise and resolve these conflicts in its meta-reviews in the vast majority of cases, suggesting this is a challenging problem and promising avenue for further research.