A Corpus of Very Short Scientific Summaries (2020)

Abstract: We present a new summarisation task, taking scientific articles and producing journal table-of-contents entries in the chemistry domain. These are one- or two-sentence author-written summaries that present the key findings of a paper. This is a first look at this summarisation task with an open access publication corpus consisting of titles and abstracts, as input texts, and short author-written advertising blurbs, as the ground truth. We introduce the dataset and evaluate it with state-of-the-art summarisation methods.

## introduction table-of-contents (toc) entries are short summaries written by authors that are placed in the table of contents of journals, often with an eyecatching accompanying image, to advertise their paper to readers. they are meant to be a clear and concise summary of a paper's main contribution, but different from the title and abstract, which have a different communicative function. we take the titles and abstracts from chemistry papers published by the royal society of chemistry as input in this initial study, as they are freely available and more numerous, but will also release full text for the smaller subset of open access publications. as such, this particular corpus is different from other scientific corpora, which normally take the abstract as the summary. we include an analysis of the corpus properties, and experimental results with strong baselines and three different state-of-the-art deep learning models: an attention-based rnn, a reinforcement learning approach, and a bert-based transformer method. we also perform a human evaluation study to compare the models and to validate the usefulness of the summarisation task. the main aim of releasing this particular corpus is to see how useful deep learning models are in producing a latent semantic representation of chemical texts. within this paper we constrain ourselves to the question: can we decode this representation into a useful summary and perhaps aid in the journal editing process? but the real goal would be to transfer this representation into useful editing tasks like plagiarism detection or semantic search and discovery. in addition, chemistry has a complex domain lexicon involving a potentially infinite set of molecules that can be described both by formulae and (often multi-word) terms, as well as a set of other techniques. while everything from tokenisation onward in an nlp pipeline would benefit from customisation, there is a lack of domain-specific resources. in this paper we start addressing this problem by introducing a summarisation corpus, but we hope this is the first of many resources that will aid researchers in this field. ultimately, underlying all these tasks is an ability to produce representations that can accurately substitute multiword terms for chemical formulae or an accurate hyponym like ketone, without ontological knowledge. this corpus is one in a set of tools that will help us compare models' abilities to do this. ## related work supervised summarisation tasks are primarily evaluated on large news text corpora: cnn/daily mail @xcite , xsum @xcite , and newsroom @xcite . most of these use professionally written summaries consisting of one or more sentences provided by the publisher. there are also domain-specific datasets like arxiv/pubmed @xcite for mixed science summarisation and bigpatent @xcite , both of which use abstracts as the ground truth. the number of documents in these datasets varies between 200,000 and 1.3 million. while initial efforts in the field concentrated on unsupervised extractive summarisation methods @xcite @xcite , recent work has seen an explosion of deep learning-based models that leverage these large datasets to produce more abstractive and natural-sounding summaries. out of these we choose three methods that each take a different approach. first is the pointergenerator method by @xcite , which balances extractive and abstractive summarisation by substituting phrases according to a learned parameter. the second method from chen and bansal (2018) uses a reinforcement learning (rl) algorithm to extract informative sentences and then rewrites them using a sequence-2-sequence model with an additional re-ranking algorithm to avoid repetitive phrasings. the final method we test is a bert-based model introduced by @xcite which uses bert embeddings as the pretrained encoder, stacked transformer layers as a decoder and a fine-tuning process to produce more natural abstractive summaries. although @xcite designed a method for use with scientific text, it was specifically created for full text documents and our input text is much shorter. there are a few historic examples of extractive summarisation in the chemistry domain @xcite , but the research in this subfield was not as active as in other nlp applications. in their summarisation approach, @xcite highlight the need for keeping the case in chemical names and for using characterbased similarity measures for relevance ranking. we, likewise, employ customised tokenisation and named entity recognition while pre-processing the corpus @xcite to enable future researchers to forgo the ner step. further related applications of nlp include information retrieval @xcite and literature mining @xcite , while lessons learned from deep learning in nlp have been applied to strings representing chemical structures to successfully discover new potential antibiotics @xcite . ## corpus description the rscsum corpus contains 307,847 papers published in the chemistry domain between 2000 and 2019. foot_0 it is split into around 80% training set, 10% test set, and 10% validation set. the titles and abstracts comprise input documents, while the reference is the short table-of-contents summary. tab. 1 shows some vital statistics of the corpus. the compression ratio (r @xmath0. our compression ratio is 21, substantially higher than that of cnn and dm (14). on the other hand, titles and abstracts are very information-dense and provide a large part of the relevant information useful for the summary. our vocabulary size is similar to that of the cnn/dm and xsum corpora @xcite , although many of the tokens occur very rarely. of course, this is a count of lower case tokens, which in chemistry can lead to the undesired effect of collapsing significantly different chemical names into the same string. about 10% of the text is also taken up by chemical named entities, including chemical formulae. these are difficult to abstract, and this process may require knowledge from the full text documents or external sources. in fact about 1% of summaries have 3 or more chemical names that do not appear in the input text, while 26% of them have at least one. tab. 2 shows an example document where automatically extracted chemical named entities @xcite are highlighted. tab. 3 shows how our corpus compares to frequently used standard summarisation corpora. although we have a relatively high number of training instances, both the summary and document lengths are at the short end of the spectrum. this will obviously make abstractive summarisation a more difficult task. and extractive fragment density were introduced by @xcite . the extractive fragment coverage is the percentage of unigrams in the summary that were directly copied from the input text, regardless of their order. the extractive fragment density is the average of the sum of squares of lengths of the extracted n-grams. the higher the coverage score is, the more individual words the summary extracts from the input text, but this may indicate re-wording using the same tokens and not necessarily direct copying of phrases. on the other hand, high density suggests a higher number of copied n-grams and therefore a more extractive dataset. if we take as reasonable that chemical names should be copied we can adapt the idea of fragment coverage to this dataset. we use the following formula to detect the percentage overlap between the tokens in each sentence n in the abstract a i and the tokens in the summary s t i , disregarding the tokens that belong to chemical named entities t c . we take the maximum value of all sentence 2 the data for cnn/dm and xsum is gathered from hugging face ( https://huggingface.co/datasets ), and pubmed and arxiv from @xcite . for all datasets, we consider the training set only. overlaps. ## methods we implement three extractive baselines and three abstractive deep models. ## quantitative evaluation we perform a set of classic quantitative experiments by training and tuning on the train and validation portions of the data and testing once on the test portion. we evaluate the performance of the models using four standard variations of rouge @xcite and report the f1 values and the confidence interval (ci) in tab. 4. rouge-1 and rouge-2 measure the unigram and bigram overlap between the automatic and reference summaries, whereas rouge-l measures the longest extended matching sequence of words using the longest common subsequence (lcs). an advantage of using lcs is that it uses all in-sequence matches that reflect sentence-level word order, rather than requiring consecutive matches. since it automatically includes the longest in-sequence common n-grams, a predefined n-gram length is not necessary. finally, the skip-bigram/unigram variation rouge-su measures overlap of word pairs that have a maximum of two gaps between words combined with unigram overlap. ## qualitative evaluation reference-based automatic evaluation has accepted limitations such as the fact that any given reference is not the only possible summary or even necessarily the best one. we, therefore, perform a smallscale human evaluation using three participants with a background in the chemical sciences. they volunteered to help. we choose to compare scib-erta as the best-performing system, in addition to rl-ea, which is less similar to the lead-2 baseline than pgn-e and also has a different learning objective. in order to test the quality of the gold standard reference summaries objectively, we also evaluate them in our setup. we perform a 3×3 latin square design with 42 items (documents) and three conditions (systems). the advantage of the latin square design (cf. tab. 6) is that each participant sees each item in only one condition, thus avoiding repetition bias. the systems (α, β, γ) are run on different batches of 14 items (1, 2, 3) and then shown to judges (a 1 , a 2 , a 3 ), so that a 1 is seeing summaries from batch 1 generated by system α, batch 2 generated by system β, and batch 3 summaries by system γ. we then randomise the order of items shown. so that the conditions are not readily distinguishable, we aim to provide a distribution of summary lengths which is as even as possible. fig. 6 shows the natural distribution of the summary lengths produced on the test set. the participants were presented with the document title and abstract (concatenated together) and a summary, which they rated on scale of 0-5 according to the criteria in tab. 7. dimensions considered are grammaticality, informativeness, relevance and overall quality. we report the systems' mean scores ## discussion in this paper we introduced a corpus that consists of titles and abstracts of papers in the scientific domain as input text, and author-written table-ofcontents summaries. the summaries are meant to be distinct from the titles and abstracts, but at least one fifth of the training data contains significant extractive elements. overall the corpus compares favourably to others in its abstractive qualities. it contains scientific terminology in the chemistry domain, and some of the terminology in the summaries does not occur in the available input text. consequently, as part of a larger effort of enriching chemical nlp, in the future we will also release about 40k full text open access articles that have corresponding toc summaries. we tested three state-of-the-art deep summarisation methods and found that a transformer-based method that uses pretrained scientific bert embeddings produces the best overall results in both quantitative evaluation and a qualitative study with three domain expert participants. it is surprisingly resistant to the strong extractive component in the training data, and produces novel content despite the short input text. a qualitative study showed that automatic summaries yield acceptable results and are in some aspects significantly better than author-written summaries. however, in order to be truly useful, the summarisation methods and associated embeddings need to be adapted to deal with the cased text, which may lead to difficulties with an expanded vocabulary. in conclusion, this study paves the way for future exploration of summarisation and semantics in the chemistry domain. candidate p gn -e : hydrogen peroxide ( h2o2 ) plays a significant role in regulating the redox balance in the living body . this work illustrated a high sensitivity hydrophilic photoacoustic probe for ratiometric imaging of h2o2 in living mice . candidate rl-ea : a high sensitivity hydrophilic photoacoustic probe for ratiometric imaging of hydrogen peroxide in vitro and vivo . candidate scibert a : a hydrophilic photoacoustic probe was developed for ratiometric imaging of h2o2 in vitro and vivo . ground truth: hydrogen peroxide ( h2o2 ) plays a significant role in regulating the redox balance in the living body . abstract: enhancing hydrophilicity of photoacoustic probes for effective ratiometric imaging of hydrogen peroxide . hydrogen peroxide ( h2o2 ) plays a significant role in regulating the redox balance in the living body . compared with traditional imaging techniques , ratiometric photoacoustic imaging is no doubt a superior choice for h2o2 visualization . however , the difficult design of ratiometric probes with only one activatable agent that exhibits two changeable absorption peaks under h2o2 activation remains a big challenge . in this work , we developed a near -infrared absorbing probe , which responded to h2o2 selectively and permitted the ratiometric photoacoustic imaging of h2o2 in living mice . the probe was constructed from an aza-bodipy backbone attached with a benzeneboronic acid pinacol ester moiety though a quaternization reaction . oligo(ethylene glycol ) ( oeg ) was introduced into hydrophobic aza-bodipy to enhance the water -solubility of the probe . the oeg-aza-bodipy-bape probe exhibited sensitive and specific ratiometric pa signals towards h2o2 . in vivo experiments also showed that the oeg-aza-bodipy-bape probe can be used for ratiometric pa imaging . overall , our work illustrated a high sensitivity hydrophilic photoacoustic probe for ratiometric imaging of hydrogen peroxide in vitro and vivo . candidate f ull : β-amyloid (aβ) plays a central role in alzheimer's disease (ad), but the specific molecular mechanism and associated structures remain unknown. candidate t 80 : β-amyloid (aβ) plays a central role in alzheimer's disease (ad), but the specific molecular mechanism and associated structures remain unknown in contrast to structured conformations and associated structures. ground truth: direct correlation of alzheimer patient data to a spectrum of nmr structures and chemical properties of beta amyloid ( aβ ) variants allows identification of conformation -dependent disease properties . abstract: pathogenic properties of alzheimer's β-amyloid identified from structure -property patientphenotype correlations. β-amyloid (aβ) plays a central role in alzheimer's disease (ad), but the specific molecular mechanism and associated structures remain unknown. we compiled patient data for carriers......we conclude that disordered monomers are likely to be pathogenically important in contrast to structured conformations...... candidate f ull : the addition of inbr3 to the oxidative sonogashira cross -coupling reaction of 2ethynylaniline with ( e ) -trimethyl ( 3,3,3 -trifluoroprop -1-enyl ) silane led the subsequent cyclization of these 1,3 -enynes under palladium catalysis provides access to the corresponding indoles bearing a 3,3,3 -trifluoroprop -1-enyl group at their 2 -position . candidate t 80 : the addition of inbr3 to the oxidative sonogashira cross -coupling reaction of 2 -ethynylaniline with ( e ) -trimethyl ( 3,3,3 -trifluoroprop -1-enyl ) silane led the oxidative sonogashira coupling of 2 -ethynylanilineselectronic with a affords a dramatic enhancing effect of inbr3 . ground truth: a dramatic enhancing effect of inbr3 was observed towards the oxidative sonogashira cross -coupling reaction of 2-ethynylaniline with (e)-trimethyl(3,3,3-trifluoroprop-1-enyl)silane . abstract: the addition of inbr3 to the oxidative sonogashira cross -coupling reaction of 2-ethynylaniline with (e)-trimethyl(3,3,3-trifluoroprop-1-enyl)silane led to a dramatic increase in the reactivity to afford the corresponding 1,3-enynes bearing a trifluoromethyl group on their terminal sp2 carbon . the subsequent cyclization of these 1,3-enynes under palladium catalysis provides access to the corresponding indoles bearing a 3,3,3-trifluoroprop-1-enyl group at their 2-position . candidate f ull : a novel visible light promoted carbodifluoroalkylation of allylic alcohols is disclosed. a series of difluoro 1,5 -dicarbonyl compounds were obtained through a tandem radical addition and 1,2-aryl migration process. candidate t 80 : a novel visible light promoted carbodifluoroalkylation of allylic alcohols was developed via a tandem radical addition and 1,2-aryl migration process , which proceeds via a radical intermediate. ground truth: a novel visible light promoted carbodifluoroalkylation of allylic alcohols is disclosed. abstract: visible light promoted carbodifluoroalkylation of allylic alcohols via concomitant 1,2-aryl migration. a novel visible light promoted carbodifluoroalkylation of allylic alcohols is disclosed. a series of difluoro 1,5-dicarbonyl compounds were obtained through a tandem radical addition and 1,2-aryl migration process. mechanistic analysis indicated that the 1,2-aryl rearrangement proceeded via a radical intermediate.