C ite S um: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision (2022)

Abstract: Scientific extreme summarization (TLDR) aims to form ultra-short summaries of scientific papers. Previous efforts on curating scientific TLDR datasets failed to scale up due to the heavy human annotation and domain expertise required. In this paper, we propose a simple yet effective approach to automatically extracting TLDR summaries for scientific papers from their citation texts. Based on the proposed approach, we create a new benchmark CiteSum without human annotation, which is around 30 times larger than the previous human-curated dataset SciTLDR. We conduct a comprehensive analysis of CiteSum, examining its data characteristics and establishing strong baselines. We further demonstrate the usefulness of CiteSum by adapting models pre-trained on CiteSum (named CITES) to new tasks and domains with limited supervision. For scientific extreme summarization, CITES outperforms most fully-supervised methods on SciTLDR without any fine-tuning and obtains state-of-the-art results with only 128 examples. For news extreme summarization, CITES achieves significant gains on XSum over its base model (not pre-trained on CiteSum), e.g., +7.2 ROUGE-1 zero-shot performance and state-of-the-art few-shot performance. For news headline generation, CITES performs the best among unsupervised and zero-shot methods on Gigaword.

## introduction scientific summarization typically regards paper abstract as the ground-truth summary, as it is written by the authors themselves with relatively high quality and readily available in most scientific documents. however, paper abstract may not always be the ideal summary because it often involves certain details such as task description, background information, and experiment results (cf. the abstract of this paper). as a result, recent work @xcite paper abstract: we study the problem of transferring a sample in one domain to an analog sample in another domain. given two related domains, s and t , we would like to learn a generative function g that maps an input sample from s to the domain t , such that the output of a given function f , which accepts inputs in either domains, would remain unchanged. other than the function f , the training data is unsupervised and consist of a set of samples from each domain. the domain transfer network (dtn) we present employs a compound loss function that includes a multiclass gan loss, an f -constancy component, and a regularizing component that encourages g to map samples from t to themselves. we apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity. citation text: taigman et al. @xcite proposed the domain transfer network (dtn) to map a sample from one domain to an analog sample in another domain and achieved favorable performance on small resolution face and digit images. however, unlike paper abstracts, ultra-short paper summaries are far from being universally available. only certain scientific venues such as open-review.net support a tldr field during paper submission, which is completely optional, and not all submitted papers provide such information. in addition, human-annotated summaries of scientific documents are rather costly and require domain expertise. as a consequence, the previous scitldr dataset @xcite , using a combination of author-provided tldr and human-annotated tldr (rephrased from paper reviews on openre-view), only collected around 2,000 examples for training and 600 for testing. in this paper, we argue that citation texts can often serve as high-quality short summaries of the cited papers. in we conduct a comprehensive analysis of cite-sum regarding its data characteristics and quality, meanwhile establishing strong baselines as the reference for future studies. we further verify the usefulness of citesum by demonstrating that models pre-trained on citesum, which we name as cites (citation text-guided summarizer), exhibit superior generalizability during low-resource adaptation to new tasks and domains. on the human-annotated scientific extreme summarization dataset scitldr @xcite , our zero-shot bart-based @xcite cites, without any fine-tuning, performs better than most fully-supervised baselines, including the fully-supervised bart model (without pre-training on citesum). our few-shot cites achieves state-of-the-art performance with only 128 labeled examples from scitldr. in addition, cites outperforms its base model (bart) on two more diverse scientific tasks -discipline classification and title generation. when transferring to news extreme summarization, despite the domain mismatch, cites achieves significantly better zero-shot performance than bart and pega-sus @xcite ) (e.g., +7.2 rouge-1) and state-of-the-art few-shot performance on the xsum dataset @xcite . furthermore, cites performs the best among unsupervised and zero-shot methods on the gigaword news headline generation dataset @xcite . contributions. (1) we propose a simple yet effective approach to automatically extracting ultra-citation example 1: we take the publicly available semantic scholar open research corpus (s2orc) @xcite as the source for data creation. citation example 2: unlike wikitransfer @xcite , cites does not involve any downstream task-specific data selection or model tuning. @xcite as the source for data creation. in the latest version of s2orc, there are 136m scientific papers from different academic disciplines and the number of papers with full-text access is 12m. we further remove papers without citation information, resulting in 9m papers as the candidates. quality control not all citation texts are of high quality and can be used as summaries of the cited papers. in to ensure data quality, we carefully locate, extract, and filter the citation texts of papers in the following manner. first, we only take citation texts in the related work section of a paper, which largely ensures that they describe the content of the cited paper instead of irrelevant information, such as task gigaword @xcite 3,803,957 / 189,651 / 1,951 32 9 ✓ xsum @xcite 204,045 / 11,332 / 11,334 431 23 ✓ arxiv @xcite 203,037 / 6,436 / 6,440 4.9k 220 ✓ scisummnet @xcite 1,000 / -/ -4.7k 150 ✗ talksumm @xcite 1,716 / -/ -4.8k 150 ✓ scitldr @xcite al., 2020) 1,992 / 619 / 618 159 21 ✗ citesum 83,304 / 4,721 / 4,921 255 23 ✓ next, we measure the similarity between the citation texts and the cited papers and filter dissimilar pairs. intuitively, if a citation sentence can serve as a high-quality summary, certain amount of its content should be from the cited paper. prior work @xcite also showed that authors tend to cite a paper using the information in the abstract of the cited paper. we thus calculate the overlap between paper abstracts and their citation sentences, and filter those below a threshold t . we set t to 50/20/40 for rouge-1/2/l recall through manual examination, resulting in a rouge-1/2/l recall of 73.1/39.4/58.5 after filtering. 3 as a reference, the rouge-1/2/l recall between paper abstracts and reference summaries on scitldr @xcite ) is 81.1/38.9/62.0 and 65.2/17.9/45.7 for author-provided (scitldr-auth) and peer reviewderived (scitldr-pr) tldr, respectively. that is, the abstraction level of citesum is between scitldr-auth and scitldr-pr. this filtering step is rather strict as we prefer quality to quantity of the data and only 93k of the 426k examples (21.8%) are kept. we further replace each citation span (e.g., "taigman et al. [8]") with a special token "ref" as they vary in different papers but essentially have the same meaning (i.e., referring to a cited paper). 3 we also experimented with semantic metrics such as bertscore @xcite but they did not function as well as rouge-based metrics in our human evaluation. dataset split after data filtering and preprocessing, there are 92,946 examples in the final citation text-guided summarization dataset, which we name as citesum. we take about 5% of the data as the validation and test sets respectively, and the remaining 90% as the training set. as one paper may be cited multiple times in different papers, we ensure that there is no label leakage by excluding papers used for evaluation from the training set. ## experiments on citesum in this section, we experiment on citesum with state-of-the-art baselines and analyze their performance under different setups to provide references for future studies. implementation and training details are provided in app. b. ## transferring to new tasks and domains with cites to further verify the quality and usefulness of cite-sum, we adapt models pre-trained on citesum to new tasks and domains, some of which are rather different from citesum and make model transfer with limited supervision very challenging. specifically, we name our pre-trained model as cites (citation text-guided summarizer). cites uses the simplest form in sec. 3 with paper abstract as input and tldr as target output. we evaluate cites on various downstream tasks with no fine-tuning (zero-shot) or limited training examples (few-shot), including scientific extreme summarization on scitldr @xcite , news extreme summarization on xsum @xcite , and news headline generation on gigaword @xcite . additionally, we evaluate cites on two more diverse tasks in the scientific domain, namely discipline classification and title generation, in a fully-supervised setting. ## related work citation text generation there have been prior studies utilizing citation texts for different purposes. one popular line of work focuses on the generation of the citation texts for writing assistance or paper comparison @xcite @xcite . however, they typically do not distinguish the citation texts that can serve as summaries of the cited paper from those used for other purposes, e.g., background or result comparison @xcite . for example, @xcite treat citation text generation as a multi-document summarization task, where the target output is a paragraph with more than two citations and the model input is the abstracts of all cited papers. there is no filtering regarding the citation texts and all the paragraphs with enough citations are included. besides including citation texts with various intents and the lack of quality control, prior studies differ from citesum in that they target longer outputs, e.g., multiple sentences @xcite or the entire related work section @xcite . citation text for paper summarization another line of work does not generate but extracts the citation texts and either uses them to form a summary directly @xcite @xcite or treats them as a bridge to the cited paper @xcite . specifically, the citation texts in the latter studies are used to find relevant contexts in the cited paper (called citation contexts). then, a long summary is formulated primarily using the cited paper, e.g., by selecting sentences from the citation contexts @xcite . unlike cites, prior citationbased summarization methods require (often multiple) citation texts of a paper as input, which are unavailable for new papers. in addition, they do not target ultra-short but abstract-long summaries. extreme summarization extreme summarization aims to form ultra-short summaries of the documents. notable benchmarks in this direction include xsum @xcite and new-shead (gu et al., 2020) in the news domain, sc-itldr @xcite in the scientific domain, and webis-tldr-17 @xcite for social media summarization. compared to sc-itldr, our citesum dataset is significantly larger in scale, from more venues than openreview, and composed of various disciplines. summarization with limited supervision our work is also related to unsupervised and zero/fewshot summarization that constructs weakly supervised guidance signals using e.g., data characteristics @xcite , domain knowledge @xcite , or pseudo labeled data @xcite . compared to prior studies, cites shows great crossdomain capability that has not been well explored. ## conclusion in this paper, we propose a simple yet effective approach to automatically extracting ultra-short paper summaries from citation texts. based on the proposed approach, we create a large-scale, highquality benchmark for scientific extreme summarization. we conduct a comprehensive analysis on the created benchmark and further demonstrate that models pre-trained on it exhibit superior generalizability to new tasks and domains such as news extreme summarization and headline generation with limited supervision.