W HAT ’ S NEW ? S UMMARIZING C ONTRIBUTIONS IN S CIENTIFIC L ITERATURE
Anonymous authors
Paper under double-blind review
A BSTRACT
With thousands of academic articles shared on a daily basis, it has become increasingly difﬁcult to keep up with the latest scientiﬁc ﬁndings.
To overcome this problem, we introduce a new task of disentangled paper summarization , which seeks to generate separate summaries for the paper contributions and the context of the work, making it easier to identify the key ﬁndings shared in articles.
For this purpose, we extend the S2ORC corpus of academic articles, which spans a diverse set of domains ranging from economics to psychology, by adding disentangled “contribution” and “context” reference labels.
Together with the dataset, we introduce and analyze three baseline approaches: 1) a uniﬁed model controlled by input code preﬁxes, 2) a model with separate generation heads specialized in generating the disentangled outputs, and 3) a training strategy that guides the model using additional supervision coming from inbound and outbound citations.
We also propose a comprehensive automatic evaluation protocol which reports the relevance , novelty , and disentanglement of generated outputs.
Through a human study involving expert annotators, we show that in 79%, of cases our new task is considered more helpful than traditional scientiﬁc paper summarization.
I NTRODUCTION
With the growing popularity of open-access academic article repositories, such as arXiv or bioRxiv, disseminating new research ﬁndings has become nearly effortless. Through such services, tens of thousands of scientiﬁc papers are shared by the research community every month 1 . At the same time, the unreviewed nature of mentioned repositories and the sheer volume of new publications has made it nearly impossible to identify relevant work and keep up with the latest ﬁndings.
Scientiﬁc paper summarization, a subtask within automatic text summarization, aims to assist researchers in their work by automatically condensing articles into a short, human-readable form that contains only the most essential information.
In recent years, abstractive summarization, an approach where models are trained to generate ﬂuent summaries by paraphrasing the source article, has seen impressive progress. State-of-the-art methods leverage large, pre-trained models (Raffel et al., 2019; Lewis et al., 2020), deﬁne task-speciﬁc pre-training strategies (Zhang et al., 2019), and scale to long input sequences (Zhao et al., 2020; Zaheer et al., 2020).
Available large-scale benchmark datasets, such as arXiv and PubMed (Cohan et al., 2018), were automatically collected from online archives and repurpose paper abstracts as reference summaries. However, the current form of scientiﬁc paper summarization where models are trained to generate paper abstracts has two caveats: 1) often, abstracts contain information which is not of primary importance, 2) the vast majority of scientiﬁc articles come with human-written abstracts, making the generated summaries superﬂuous.
To address these shortcomings, we introduce the task of disentangled paper summarization. The new task’s goal is to generate two summaries simultaneously, one strictly focused on the summarized article’s novelties and contributions, the other introducing the context of the work and previous efforts. In this form, the generated summaries can target the needs of diverse audiences: senior researchers and ﬁeld-experts who can beneﬁt from reading the summarized contributions, and newcomers who can quickly get up to speed with the intricacies of the addressed problems by reading the context summary and get a perspective of the latest ﬁndings from the contribution summary.
For this task, we introduce a new large-scale dataset by extending the S2ORC (Lo et al., 2020) corpus of scientiﬁc papers, which spans multiple scientiﬁc domains and offers rich citation-related metadata.
We organize and process the data, and extend it with automatically generated contribution and context reference summaries, to enable supervised model training.
We also introduce three abstractive baseline approaches: 1) a uniﬁed, controllable model manipulated with descriptive control codes (Fan et al., 2018; Keskar et al., 2019), 2) a one-to-many sequence model with a branched decoder for multi-head generation (Luong et al., 2016; Guo et al., 2018), and 3) an informationtheoretic training strategy leveraging supervision coming from the citation metadata (Peyrard, 2019).
To benchmark our models, we design a comprehensive automatic evaluation protocol that measures performance across three axes: relevance, novelty, and disentanglement. We thoroughly evaluate and analyze the baselines models and investigate the effects of the additional training objective on the model’s behavior. To motivate the usefulness of the newly introduced task, we conducted a human study involving human annotators in a hypothetical paper-reviewing setting. The results ﬁnd disentangled summaries more helpful in 79% of cases in comparison to abstract-oriented outputs.
Code, model checkpoints, and data preparation scripts introduced in this work are available at omitted.link
R ELATED W ORK
Recent trends in abstractive text summarization show a shift of focus from designing task-speciﬁc architectures trained from scratch (See et al., 2017; Paulus et al., 2018) to levaraging large-scale
Transformer-based models pre-trained on vast amounts of data (Liu & Lapata, 2019; Lewis et al., 2020), often in multi-task settings (Raffel et al., 2019). A similar shift can be seen in scientiﬁc paper summarization, where state-of-the-art approaches utilize custom pre-training strategies (Zhang et al., 2019) and tackle problems of summarizing long documents (Zhao et al., 2020; Zaheer et al., 2020).
Other methods, at a smaller scale, seek to utilize the rich metadata associated with scientiﬁc articles and combine them with graph-based methods (Yasunaga et al., 2019).
In this work, we combine these two line of work and propose models that beneﬁt from pre-training procedures, but also take advantage of task-speciﬁc metadata. Popular large-scale benchmark datasets in scientiﬁc paper summarization (Cohan et al., 2018) were automatically collected from open-access paper repositories and consider article abstracts as the reference summaries. Other forms of supervision have also been investigated for the task, including author-written highlights (Collins et al., 2017), human annotations and citations (Yasunaga et al., 2019), and transcripts from conference presentations of the articles (Lev et al., 2019).
In contrast, we introduce a large-scale automatically collected dataset with more ﬁne-grained references than abstracts, which also offers rich citation-related metadata. Update summarization (Dang & Owczarzak) deﬁnes a setting in a collection of documents with partially overlapping information is summarized, some of which are considered prior knowledge.
The goal of the task is to focus the generated summaries on the novel information.
Work in this line of research mostly focuses on novelty detection in news articles (Bysani, 2010; Delort & Alfonseca, 2012) and timeline summarization (Martschat & Markert, 2018; Chang et al., 2016) on news and social media domains. Here, we propose a novel task that is analogous to update summarization in that it also requires contrasting the source article with the content of other related articles which are considered pre-existing knowledge.
T ASK
Given a source article D , the goal of disentangled paper summarization is to simultaneously summarize the contribution y con and context y ctx of the source article.
Here, contribution refers to the novelties introduced in the article D , such as new methods, theories, or resources, while context represents the background of the work D , such as a description of the problem or previous work on the topic.
The task inherently requires relative comparison of the article with other related papers to effectively disentangle its novelties from pre-existing knowledge.
Therefore, we also consider two sets of citations: inbound citations C I and outbound citations C O as potential sources of useful information for contrasting the article D with its broader ﬁeld.
Inbound citations refer to the set of papers that cite D , i.e. relevant future papers, while outbound citations are the set of papers that
[Table] D cites, i.e. relevant previous papers. With its unique set of goals, the task of disentangled paper summarization poses a novel set of challenges for automatic summarization systems to overcome: 1) identifying salient content of D and related papers from C I and C O , 2) comparing the content of D with each document from the citations, and 3) summarizing the article along the two axes: contributions and context.
3. D ATASET
Current benchmark datasets used for the task of scientiﬁc paper summarization, such as arXiv and PubMed (Cohan & Goharian, 2015), are limited in size, number of domains, and lack of citation metadata.
Thus, we construct a new dataset based on the S2ORC (Lo et al., 2020) corpus, which offers a large collection of scientiﬁc papers spanning multiple domains along with rich citationrelated metadata, such as citation links between papers and annotated citation spans.
Speciﬁcally, we carefully curate the data available in the S2ORC corpus and extend it with new reference labels.
Data Curation Some papers in the S2ORC corpus 2 do not contain a complete set of information required by our summarization task: paper text, abstract, and citation metadata.
We remove such instances and construct a paper summarization dataset in which each example a) has an abstract and body text, and b) has at least 5 or more inbound and outbound citations, C I and C O respectively.
In cases where a paper has more than 20 incoming or outgoing citations, we sort them in descending order by the number of their respective citation and keep the top 20 most relevant articles. [Table]
Extraction Each article in the set of inbound and outbound citations can be represented by its full text, abstract, or the span of text associated with the citation.
In this study, we follow Qazvinian & Radev (2008) and Cohan & Goharian (2015) in representing citations with the sentences in which the citation occurs. Thus, an outbound citation is represented by a sentence from the source paper.
Usually, such sentences directly refer to the cited paper and place its content in relation to the source paper.
Analogously, an inbound citation is represented by sentence from citing paper and relates its content with the source paper. Reference Generation
Our approach relies on the availability of reference summaries for both contributions and contexts. However, such annotations are not provided or easily extractable from the S2ORC corpus, and collecting expert annotations is infeasible due to the associated costs.
Therefore, we apply a data-driven approach to automatically extract contribution and context reference summaries from the available paper abstracts.
First, we manually label 400 abstracts sampled from the training set. Annotations are done on a sentence-level with binary labels indicating contributionand context- related sentences 4 .
This procedure yields 3341 sentences with associated binary labels, which we refer to as golden standard references.
Next we ﬁne-tune an automatic sentence classiﬁer using the golden standard data.
As our classiﬁer we use SciBERT (Beltagy et al., 2019), which after ﬁne-tuning achieves 86.3% accuracy in classifying contribution and context sentences on a held-out test set.
Finally, we apply the ﬁne-tuned classiﬁer to generate reference labels for all examples in our dataset, which we refer to as silver standard references.
The statistics of the resulting dataset are shown in Table 1.
M ODELS
Our goal is to build an abstractive summarization system which has the ability to generate contribution and context summaries based on the source article.
To achieve the necessary level of controllability, we propose two independent approaches building on encoder-decoder architectures:
C ONTROL C ODE (CC) A common approach to controlling model-generated text is by conditioning the generation procedure on a control code associated with the desired output.
Previous work on controllable generation (Fan et al., 2018; Keskar et al., 2019) showed that prepending a special token or descriptive prompt to the model’s input during training and inference is sufﬁcient to achieve ﬁne-grained control over the generated content.
Following this line of work, we modify our training instances by prepending textual control codes, contribution: or context: , to the summarized articles. During training, all model parameters are updated for each data instance and the model is expected to learn to associate the provided prompt with the correct output mode.
The approach does not require changes in the architecture, making it straightforward to combine with existing large-scale, pre-trained models. The architecture is shown on the left of Figure 1.
M ULTI H EAD (MH)
An alternative way of controlling generation is by explicitly allocating layers within the model speciﬁcally for the desired control aspects.
Prior work investigating multi-task models (Luong et al., 2016; Guo et al., 2018) showed the beneﬁts of combining shared and taskspeciﬁc layers within a single, multi-task architecture.
Here, the encoder shares all parameters between the two generation modes, while the decoder shares all parameters, apart from the ﬁnal layer, which splits into two generation branches.
During training, each branch is individually updated with gradients from the associated mode.
The model shares the softmax layer weights between the output branches under assumption that token-level vocabulary distributions are similar in the two generation modes due to the common domain. This approach is presented on the right of Figure 1.
4. I NFORMATIVENESS - GUIDED T RAINING
Peyrard (2019) proposed an information theoretic perspective on text summarization which decomposes the criteria of a good summary into redundancy, relevance, and informativeness.
Among these criteria, informativeness measures the user’s degree of surprise after reading a summary given their background knowledge, and can be formally deﬁned as:
[Equation]
where ! i is a primitive semantic unit, P K is the probability over the unit under the user’s knowledge, P D is the probability over the unit with respect to the source document, and i is an index over all semantic units within a summary.
As deﬁned by Peyrard (2019), informativeness is in direct correspondence to contribution summarization. Paper contributions are novel contents introduced to the community, which causes surprisal given the general knowledge about the state of the ﬁeld.
Therefore, in this work we explore utilizing this measure as an auxiliary objective that is optimized during training.
We deﬁne the semantic unit !
i as the summary itself 5 , which enables a simple interpretation of the corresponding probabilities.
We estimate P D as the likelihood of the summary given the paper content, P D ( ! i ) =
p ( y | D ) .
Since each paper is associated with a unique context and background knowledge, we treat the background knowledge as all relevant papers published before the source paper, i.e. , outbound citations C O .
Therefore, P K is estimated as the likelihood of the summary given the previous work, P K ( ! i ) =
p ( y | C O ) .
We formulate the informativeness function as:
[Equation]
where the conditioning depends on the generation mode of the model, and aim to maximize it during the training procedure. Combined with a cross entropy loss L CE , we obtain the ﬁnal objective which we aim to the minimize during training:
[Equation]
where  is a scaling hyperparameter determined through cross-validation.
E XPERIMENTS AND R ESULTS
In this section, we describe the experimental environment and report automatic evaluation results.
We consider four model variants:
• CC , CC+INF : C ONTROL C ODE model without and with the informativeness objective, • MH , MH+INF : M ULTI H EAD model without and with the informativeness objective.
5. E VALUATION
We perform automatic evaluation of the system outputs ( s con , s ctx ) against the silver standard references ( y con , y ctx ) . For this purpose, we have designed a comprehensive evaluation protocol, shown in Figure 2, based on existing metrics that evaluates the performance of models across 3 dimensions:
Relevance Generated summaries should closely correspond with the available reference summaries.
We measure the lexical overlap and semantic similarity between ( s con , y con ) and ( s ctx , y ctx ) using ROUGE (R- i ) (Lin, 2004) and BERTScore (Zhang et al. 2020; BS), respectively.
Purity Generated contribution summary should closely correspond with its respective reference summary, but should not overlap with the context reference summary.
We measure the lexical overlap between s con and ( y con , y ctx ) using NouveauROUGE con (N con - i ) (Conroy et al., 2011). The metric reports an aggregate score deﬁned as a linear combination between the two components:
[Equation]
where weights ↵ ij were set by the original authors to favor outputs with maximal and minimal overlap with related and unrelated references, accordingly.
Analogously, we calculate N ctx - i in reverse direction between s ctx and ( y ctx , y con ) .
Purity P- i is deﬁned as the average novelty in both directions: [Equation] [Table]
Disentanglement Generated contribution and context summaries should have minimal overlap. We measure the degree of lexical overlap and semantic similarity between ( s con , s ctx ) using ROUGE and BERTScore, respectively.
To maintain consistency across metrics (higher is better) we report disentanglement scores as [-100  M : D- i = 
-]complements of the associated metrics:
[Equation]
5. I MPLEMENTATION D ETAILS
Our models build upon distilBART 6 (Sanh et al., 2019; Wolf et al., 2019), a Transformerbased (Vaswani et al., 2017), pre-trained sequence-to-sequence architecture distilled from BART (Lewis et al., 2020).
Speciﬁcally, we used a model with 6 self-attention layers in both the Encoder and Decoder. Weights were initialized from a model ﬁne-tuned on a news summarization task.
For the M ULTI H EAD model, the ﬁnal layer of the decoder was duplicated and initialized with identical weights.
We ﬁne-tuned on the training set for 80000 gradient steps with a ﬁxed learning rate of 3 . 0 ⇥ 10  5 and choose the best checkpoints in terms of ROUGE-1 scores on the validation set. The loss scaling hyparameter  (Eq. 3) was set to 0.05 and 0.01 for the C ONTROL C ODE and M ULTI H EAD models, accordingly. Input and output lengths were set to 1024 and 200 , respectively.
At inference time, we decoded using beam search with beam size 5 .
Evaluation was performed using SummEval toolkit (Fabbri et al., 2020).
5. R ESULTS
In Table 2 we report results from the automatic evaluation protocol described in Subsection 5.1.
Relevance Across most models and metrics, relevance scores for context generation are higher than those for contribution summarization. Manual inspection revealed that in some cases generated context summaries also include article contribution information, while this effect was not observed in the reverse situation.
Considering that silver standard annotations may contain noisy examples with incorrectly separated references, we suspect that higher ROUGE scores for context summaries may be caused by noisy predictions coinciding with noisy references.
Examples of such summaries are shown in the Appendix E. We also observe that informativeness-guided models (+INF) perform on par with their respective base versions, and the additional training objective does not affect the performance on the relevance metric. This insight corroborates with Peyrard (2019) who deﬁnes informativeness and relevance as orthogonal criteria.
Purity While the informativeness objective was designed to improve novelty of generated summaries, results show an opposite effect, where informativeness-guided models slightly underperform their base counterparts.
The true reason for such behavior is unknown, however, it might be an indicator that the outbound citations C O are not a good approximation of reference context summaries
Original Abstract: Energy optimization in buildings by controlling the Heating Ventilation and Air Conditioning (HVAC) system is being researched extensively.
In this paper, a model-free actor-critic Reinforcement Learning (RL) controller is designed using a variant of artiﬁcial recurrentneuralnetworkscalledLong-Short-TermMemory(LSTM)networks.
Optimizationofthermalcomfortalongsideenergyconsumption is the goal in tuning this RL controller.
The test platform, our ofﬁce space, is designed using SketchUp.
Using OpenStudio, the HVAC system is installed in the ofﬁce. ...
Contribution:
In this work, we introduce novel control algorithms from a branch of machine learning called reinforcement learning.
In our current approach, the impetus is thermostat control.
Instead of traditional on/off heating and cooling control, reinforcement learning is utilized to set this schedule to obtain improved Predicted Mean Vote (PMV)-based thermal comfort at an optimal energy expenditure.
Hence, a thermostats schedule is computed using an RL controller.
The results show that the Q-learning algorithm can learn to adapt to time-varying and nonlinear system dynamics without explicit identiﬁcation of the plant model in both systems and controls.
Context: The Heating, Ventilation and Air Conditioning (HVAC) systems can account for up to 50% of total building energy demand.
In the hopes of moving toward a greener, more energy-efﬁcient future, a signiﬁcant improvement in energy efﬁciency is needed to achieve this goal.
Despite the advances in research on HVAC control algorithms, most ﬁeld equipment is controlled using classical methods that include hysteresis/on/off and Proportional Integral and Derivative controllers. ... y ctx , or the relationship between the two is weak. This effect is more evident in the Medical and
Biology domains, which are the two most frequent domains in the dataset Disentanglement Results indicate that C ONTROL C ODE -based models perform better than M ULTI H EAD approaches in terms of generating disentangled outputs. This comes as a surprise given that the CC models share all parameters between the two generation modes, but might indicate that the two tasks contain complementary training signals. We also noticed that, both informativenessguided models performed better in terms of D-1.
Based on both purity and disentanglement evaluations, we suspect that the informativeness objective does guide the models to output more disentangled summaries (second term in Eq 2), but the signal is not strong enough to focus on generating the appropriate content (ﬁrst term in Eq 2). It is also clear that the M ULTI H EAD model beneﬁts more from the additional training objective.  A NALYSIS 6. Q UALITATIVE A NALYSIS To better understand the strengths and shortcomings of our models, we performed a qualitative study of model outputs.
Table 3 shows an example of generated summaries compared with the original abstract of the summarized article.
Our model successfully separates the two generation modes and outputs coherent and easy to follow summaries. The contribution summary clearly lists the novelties of the work, while the context summary introduces the task at hand and explains its importance. In comparison, the original abstract brieﬂy touches on many aspects: the context, methods used, and contributions, but also offers details that are not of primary importance, such as the detailed about the simulation environment.
More generally, the described trends hold across summaries generated by our models. The model outputs are ﬂuent, abstractive, offer good separation between modes, and are on topic.
However, the factual correctness of summaries could not be assessed due to the highly specialized content and language of the summarized articles. An artifact noticed in a few instances of the inspected outputs was leakage of contribution information into context summaries. Other examples of generated summaries are included in the Appendix E.
6. P ER - DOMAIN P ERFORMANCE
Taking advantage of the rich metadata associated with the S2ORC corpus, we analyze the performance of models across the 10 most frequent scientiﬁc domains.
Table 4 shows the results of contribution summarization using the C ONTROL C ODE 8 model. summaries for the top 10 domains generated using the C ONTROL C ODE model. Performance on
Medicine domain is paricularly low [Table] While ROUGE-1 scores oscillate aroundpoints for most academic ﬁelds, the results indicate that summarizing documents from the Medical domain is particularly difﬁcult, with models scoring about 7 points below average.
Manual inspection of instances with low scores (R-1 < 20 ), exposed that contribution summaries in the Medical domain are highly quantitative ( e.g. “Among these treated . .
retinopathy was noted in X%”).
While other domains such as Biology also suffer from the same phenomenon, low-scoring quantitative summaries were 1.9 times more frequent in Medicine than in Biology. An investigation into the domain distribution in our dataset (Appendix) revealed that Biology and Medicine are the two best represented ﬁelds in the corpus, with Biology having over twice as many examples. We hypothesize that the poor performance of models stems from the fact that generating such quantitative summaries requires a deeper, domain-speciﬁc understanding of the source document and the available in-domain training data is insufﬁcient to achieve that goal.
6. H UMAN E VALUATION OF U SEFULNESS
[Table]
To assess the usefulness of the newly introduced task to the research community, we conducted a human study involving expert annotators.
The study aimed to compare disentangled papers summaries with traditional, abstractbased summaries in a hypothetical paper reviewing setting.
Judges were shown both types of summaries side by side and asked to pick one which would be more helpful for conducting the paper review.
Abstract-based summaries were generated by a model with a conﬁguration identical to the models previously introduced in this work, trained to generate full abstracts using the same training corpus.
Annotators that participated in this study hold graduate degrees in technical ﬁelds and are active in the research community, however, they were not involved or familiar with this work prior to this experiment. The study used 100 examples, out of which 50 were decoded on the test split of the adapted S2ORC dataset, while the other 50 were generated in a zero-shot fashion from articles in the CORD dataset (Wang et al., 2020), a recently introduced collection of papers related to
COVID-19.
Results in Table 5 show the proportion of all examples where the annotators preferred the disentangled summaries over the generated abstracts. The numbers indicate a strong preference from the judges for disentangled summaries, in case of both S2ORC and CORD examples. The values on CORD samples are slightly higher than those on S2ORC; we suspect this being due to the fact that the annotators were less familiar with the topics described in Covid-related publications and would require more help to review such articles.
C ONCLUSIONS
In this paper, we propose disentangled paper summarization , a new task in scientiﬁc paper summarizing where models simultaneously generate contribution and context summaries.
With the task in mind, we introduced a large-scale dataset with ﬁne-grained reference summaries and rich metadata. Along with the data, we introduced three abstractive baseline approaches to solving the new task and thoroughly assessed them using a comprehensive evaluation protocol design for the task at hand.
Through human studies involving expert annotators with motivated the usefulness of the task in comparison to the current scientiﬁc paper summarization setting. Together with this paper, we release the code, trained model checkpoints, and data preprocessing scripts to support future work in this direction. We hope this work will positively contribute to creating AI-based tools for assisting scientists in the research process.