SumPubMed : Summarization Dataset of P ub M ed Scientific Articles (2021)

Abstract: Most earlier work on text summarization is carried out on news article datasets. The summary in these datasets is naturally located at the beginning of the text. Hence, a model can spuriously utilize this correlation for summary generation instead of truly learning to summarize. To address this issue, we constructed a new dataset, SumPubMed , using scientific articles from the PubMed archive. We conducted a human analysis of summary coverage, redundancy, readability, coherence, and informativeness on SumPubMed . SumPubMed is challenging because (a) the summary is distributed throughout the text (not-localized on top), and (b) it contains rare domain-specific scientific terms. We observe that seq2seq models that adequately summarize news articles struggle to summarize SumPubMed . Thus, SumPubMed opens new avenues for the future improvement of models as well as the development of new evaluation metrics.

## introduction most of the existing summarization datasets, i.e., cnn daily mail and duc are news article datasets. that is, the article acts as a document, and the summary is a short (10-15 lines) manually written highlight (i.e., headlines). in many cases, these highlights have significant lexical overlap with the few lines at the top of the article. thus, any model which can extract the top few lines, e.g., extractive methods, performs adequately on these datasets. however, the task of summarization is not merely limited to short-length news articles. one could also summarize long and complex documents such as essays, research papers, and books. in such cases, an extractive approach will most likely fail. for successful summarization on these documents, one needs to (a) find information from the distributed (non-localized) locale in the large text, (b) perform paraphrasing, simplifying, and shortening of longer sentences and (c) combine information from multiple sentences to generate the summary. hence, an abstractive approach will perform better on such large documents. one obvious source that contains such complex documents is the medline biomedical scientific articles, which are publicly available. furthermore, these articles are accompanied by abstracts and conclusions which summarize the documents. therefore, we constructed a scientific summarization dataset from pre-processed pubmed articles, named sumpubmed. in comparison to the previous news-article based datasets, sumpubmed documents are longer, and the corresponding summaries cannot be extracted by selecting a few sentences from fixed locations in the document. the dataset, along with associated scripts, are available at https://github.com/vgupta123/ sumpubmed . our contributions in this paper are: in section 1, we provided a brief introduction. the remaining parts of the paper are organized as follows: in section 2 we explain how sumpubmed was created. in section 3, we explain how summaries were annotated by human experts. we then move on to experiments in section 4. we next discuss the results and analysis in section 5, followed by the related work in section 6. lastly, we move on to the conclusions in final section 7. ## sumpubmed creation sumpubmed is created from pubmed biomedical research papers, which has 26 million documents. the documents are sourced from diverse literature, including medline, life science journals, and online books. for sumpubmed creation we took 33, 772 documents from bio med central (bmc). bmc incorporates research papers related to medicine, pharmacy, nursing, dentistry, health care, health services, etc. the research documents in bmc contain two subsections: front and body. the front part of the document is basically the abstract and taken as the gold summary. the body part which is taken as the main document contains three subsections: background, results, and conclusion. ## human annotation of sumpubmed inspired from work on human evaluation of summaries by @xcite , we distributed 50 randomly chosen summaries from the noun-phrase versions of sumpubmed to 10 expert annotators (graduate nlp students) such that we have 3 annotation for each summary. we asked these humanannotators to rate the summaries on a scale of 1 to 10. we created different document files, each having 10 pairs of summaries where we randomly shuffled between reference and generated summaries with respect to the placement on the page (left or right). the annotators evaluated the summaries based on the following criteria: the average scores and standard deviations are shown in rouge and human scores for the 50 summaries evaluated by expert annotators, we calculated the pearson's correlation @xcite between rouge @xcite scores (rouge-1 (r-1), rouge-2 (r-2) and rouge-l (r-l)) in terms of precision, recall and f1 score with the humanevaluated scores. rouge-n is an n-gram similarity measure that computes uni/bi/trigram and higher n-gram overlaps. in r-l, l refers to the longest common subsequence (lcs) overlap: a subsequence of matching words with the maximal length that is common in both texts with the order of words being preserved. pearson's correlation value (between -1 and +1) quantifies the degree to which quantitative and continuous variables are related to each other. the pearson's correlations values are shown in rouge scores assume that a high-quality summary generated by a model should have common words and phrases with a gold-standard summary. however, this is not always true because (a) there can be semantically similar meaning (synonymous) word usage, and (b) there can be the usage of text paraphrases (similar information conveyed) with a little lexical overlap in the reference summary text. therefore, merely considering lexical overlaps to evaluate summary quality is not sufficient. a high rouge score may indicate a good summary, but a low rouge score does not necessarily indicate a bad summary. furthermore, while summarizing large documents, humans tend to utilize different paraphrasing/words to convey the same meaning in a shorter form. several studies by @xcite ; @xcite argue that rouge is not an accurate estimator of the quality of a summary for scientific input, e.g., biomedical text. hence, a weak correlation of rouge scores with human ratings on sumpubmed, as reported in ## experiments we have used the noun phrase version of sumpubmed in the abstractive summarization settings and the hybrid version of sumpubmed in the extractive and the hybrid settings, i.e., (extractive + abstractive) summarizations. we split the dataset into train (93%), test (3%), and validation (4%) sets. before training, we wrote a script that first tokenizes all input files and then forms the vocabulary and chunked files for the train, test, and validation sets. this step converts the input into a suitable format for the seq2seq models. ## results and analysis results on sumpubmed for abstractive methods, i.e., seq2seq models (with and without coverage), the extractive method of textrank, and the hybrid approach, i.e., textrank + seq2seq (with and without coverage) are shown in tables 6, 7, and 8, respectively. we also evaluated the seq2seq models on news datasets (cnn/daily mail and duc 2001) for comparison, as shown in analysis: in all three approaches, abstractive in we notice in both tables 6 and 8 that by adding the coverage (+cov) mechanism, the problem of repetition in summaries is solved to a great extent. the rouge scores also show improvement after applying coverage to pointer-generator networks. thus, one can conclude that pointer generator networks effectively handle named entities and outof-vocabulary words, and the coverage mechanism is useful to avoid repetitive generation, which is essential for scientific summarization. in ## conclusion we created a non-news, sumpubmed dataset, from the pubmed archive to study how various summarization techniques perform on task of scientific summarization on domain specific scientific texts. these texts have essential information scattered throughout the whole text. in contrast, earlier datasets with news stories appear to mostly have useful information in the first few lines of the document text. we also conducted a human evaluation on aspects such as repetition, readability, coherence, and informativeness for 50 summaries of 250 words. each summary is evaluated by 3 different individuals on the basis of four parameters: readability, coherence, non-repetition, and informativeness. due to the unavailability of any state-ofthe-art results on this new dataset, we built several baseline models (extractive, abstractive, and hybrid model) for sumpubmed. to check the significance of our results, we studied the effectiveness of rouge through pearson's correlation analysis with human-evaluation and observed that many variants of rouge scores correlate poorly with human evaluation. our results indicate that rouge is possibly not a proper metric for sumpubmed. eva sharma, chen li, and lu wang. 2019. bigpatent: a large-scale dataset for abstractive and coherent summarization. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 2204-2213. michael völske, martin potthast, shahbaz syed, and benno stein. 2017. tl; dr: mining reddit to learn automatic summarization. in proceedings of the workshop on new frontiers in summarization, pages 59-63.