What’s New? Summarizing Contributions in Scientific Literature (2023)

Abstract: With thousands of academic articles shared on a daily basis, it has become increasingly difficult to keep up with the latest scientific findings. To overcome this problem, we introduce a new task of disentangled paper summarization, which seeks to generate separate summaries for the paper contributions and the context of the work, making it easier to identify the key findings shared in articles. For this purpose, we extend the S2ORC corpus of academic articles, which spans a diverse set of domains ranging from economics to psychology, by adding disentangled “contribution” and “context” reference labels. Together with the dataset, we introduce and analyze three baseline approaches: 1) a unified model controlled by input code prefixes, 2) a model with separate generation heads specialized in generating the disentangled outputs, and 3) a training strategy that guides the model using additional supervision coming from inbound and outbound citations. We also propose a comprehensive automatic evaluation protocol which reports the relevance, novelty, and disentanglement of generated outputs. Through a human study involving expert annotators, we show that in 79%, of cases our new task is considered more helpful than traditional scientific paper summarization.

## introduction with the growing popularity of open-access academic article repositories, such as arxiv or biorxiv, disseminating new research findings has become nearly effortless. through such services, tens of thousands of scientific papers are shared by the research community every month foot_0 . at the same time, the unreviewed nature of mentioned repositories and the sheer volume of new publications has made it nearly impossible to identify relevant work and keep up with the latest findings. scientific paper summarization, a subtask within automatic text summarization, aims to assist researchers in their work by automatically condensing articles into a short, human-readable form that contains only the most essential information. in recent years, abstractive summarization, an approach where models are trained to generate fluent summaries by paraphrasing the source article, has seen impressive progress. state-of-the-art methods leverage large, pre-trained models @xcite , define task-specific pre-training strategies @xcite , and scale to long input sequences @xcite . available large-scale benchmark datasets, such as arxiv and pubmed @xcite , were automatically collected from online archives and repurpose paper abstracts as reference summaries. however, the current form of scientific paper summarization where models are trained to generate paper abstracts has two caveats: 1) often, abstracts contain information which is not of primary importance, 2) the vast majority of scientific articles come with human-written abstracts, making the generated summaries superfluous. to address these shortcomings, we introduce the task of disentangled paper summarization. the new task's goal is to generate two summaries simultaneously, one strictly focused on the summarized article's novelties and contributions, the other introducing the context of the work and previous efforts. in this form, the generated summaries can target the needs of diverse audiences: senior researchers and field-experts who can benefit from reading the summarized contributions, and newcomers who can quickly get up to speed with the intricacies of the addressed problems by reading the context summary and get a perspective of the latest findings from the contribution summary. for this task, we introduce a new large-scale dataset by extending the s2orc @xcite corpus of scientific papers, which spans multiple scientific domains and offers rich citation-related metadata. we organize and process the data, and extend it with automatically generated contribution and context reference summaries, to enable supervised model training. we also introduce three abstractive baseline approaches: 1) a unified, controllable model manipulated with descriptive control codes @xcite @xcite , 2) a one-to-many sequence model with a branched decoder for multihead generation @xcite , and 3) an information-theoretic training strategy leveraging supervision coming from the citation metadata @xcite . to benchmark our models, we design a comprehensive automatic evaluation protocol that measures performance across three axes: relevance, novelty, and disentanglement. we thoroughly evaluate and analyze the baselines models and investigate the effects of the additional training objective on the model's behavior. to motivate the usefulness of the newly introduced task, we conducted a human study involving human annotators in a hypothetical paper-reviewing setting. the results find disentangled summaries more helpful in 79% of cases in comparison to abstractoriented outputs. code and data preparation scripts introduced in this work are available at https:// github.com/salesforce/disentangled-sum . ## related work recent trends in abstractive text summarization show a shift of focus from designing task-specific architectures trained from scratch @xcite to leveraging largescale transformer-based models pre-trained on vast amounts of data @xcite , often in multi-task settings @xcite . a similar shift can be seen in scientific paper summarization, where state-of-the-art approaches utilize custom pre-training strategies @xcite and tackle problems of summarizing long documents @xcite . other methods, at a smaller scale, seek to utilize the rich metadata associated with scientific articles and combine them with graph-based methods @xcite . in this work, we combine these two lines of work and propose models that benefit from pre-training procedures, but also take advantage of task-specific metadata. popular large-scale benchmark datasets in scientific paper summarization @xcite were automatically collected from open-access paper repositories and considered article abstracts as the reference summaries. other forms of supervision have also been investigated for the task, including author-written highlights @xcite and annotations @xcite , citations @xcite , and transcripts from conference presentations of the articles @xcite . in contrast, we introduce a large-scale automatically collected dataset with more fine-grained references than abstracts, which also offers rich citation-related metadata. update summarization @xcite defines a setting in a collection of documents with partially overlapping information is summarized, some of which are considered prior knowledge. the goal of the task is to focus the generated summaries on the novel information. work in this line of research mostly focuses on novelty detection in news articles @xcite and timeline summarization @xcite on news and social media domains. here, we propose a novel task that is analogous to update summarization in that it also requires contrasting the source article with the content of other related articles which are considered pre-existing knowledge. ## task given a source article d, the goal of disentangled paper summarization is to simultaneously summarize the contribution y con and context y ctx of the source article. here, contribution refers to the novelties introduced in the article d, such as new methods, theories, or resources, while context represents the background of the work d, such as a description of the problem or previous work on the topic. the task inherently requires a relative comparison of the article with other related papers to effectively disentangle its novelties from pre-existing knowledge. therefore, we also consider two sets of citations: inbound citations c i and outbound citations c o as potential sources of useful information for contrasting the article d with its broader field. inbound citations refer to the set of papers that cite d, i.e. relevant future papers, while outbound citations are the set of papers that d cites, i.e. content of d and related papers from c i and c o , 2) comparing the content of d with each document from the citations, and 3) summarizing the article along the two axes: contributions and context. ## models our goal is to build an abstractive summarization system which has the ability to generate contribution and context summaries based on the source article. to achieve the necessary level of controllability, we propose two independent approaches building on encoder-decoder architectures: controlcode (cc) a common approach to controlling model-generated text is by conditioning the generation procedure on a control code associated with the desired output. previous work on controllable generation @xcite @xcite showed that prepending a special token or descriptive prompt @xcite to the model's input during training and inference is sufficient to achieve fine-grained control over the generated content. following this line of work, we modify our training instances by prepending textual control codes, contribution: or context:, to the summarized articles. during training, all model parameters are updated for each data instance and the model is expected to learn to associate the provided prompt with the correct output mode. the approach does not require changes in the architecture, making it straightforward to combine with existing large-scale, pre-trained models. the architecture is shown on the left of figure 1 . ## experiments and results in this section, we describe the experimental environment and report automatic evaluation results. we consider four model variants: ## analysis 6.1 qualitative analysis to better understand the strengths and shortcomings of our models, we performed a qualitative study of model outputs. more generally, the described trends hold across summaries generated by our models. the model outputs are fluent, abstractive, offer good separation between modes, and are on topic. an artifact noticed in a few instances of the inspected outputs was leakage of contribution information into context summaries. factual correctness of summaries is discussed in section 6.4. other examples of generated summaries are included in the appendix c. ## conclusions in this paper, we propose disentangled paper summarization, a new task in scientific paper summarizing where models simultaneously generate contribution and context summaries. with the task in mind, we introduced a large-scale dataset with fine-grained reference summaries and rich metadata. along with the data, we introduced three abstractive baseline approaches to solving the new task and thoroughly assessed them using a comprehensive evaluation protocol design for the task at hand. through human studies involving expert annotators with motivated the usefulness of the task in comparison to the current scientific paper summarization setting. together with this paper, we release the code, trained model checkpoints, and data preprocessing scripts to support future work in this direction. we hope this work will positively contribute to creating ai-based tools for assisting scientists in the research process.