Explaining Relationships Between Scientific Documents (2021)

Abstract: We address the task of explaining relationships between two scientific documents using natural language text. This task requires modeling the complex content of long technical documents, deducing a relationship between these documents, and expressing the details of that relationship in text. In addition to the theoretical interest of this task, successful solutions can help improve researcher efficiency in search and review. In this paper we establish a dataset of 622K examples from 154K documents. We pretrain a large language model to serve as the foundation for autoregressive approaches to the task. We explore the impact of taking different views on the two documents, including the use of dense representations extracted with scientific IE systems. We provide extensive automatic and human evaluations which show the promise of such models, but make clear challenges for future work.

## introduction the output of the world's scientists doubles roughly every nine years @xcite . consequently, researchers must devote significant energy to quickly understand how a new piece of research fits with a rapidly changing research landscape. several lines of research seek to reduce this burden on scientists. citation recommendation systems suggest references to relevant published work @xcite . intent classification systems help determine the type and importance of a citation in a work @xcite . summarization systems aim to help researchers more quickly understand the basic ideas in a piece of research @xcite . we draw inspiration from these works as well as automatically describing inter-document relationships could decrease the time researchers devote to literature review. for instance, explanations for a new paper can be personalized to a particular reader by relating the new work to ones they have read before. further, such technology could be incorporated into writing assistance systems to help less experienced or non-native writers better articulate the connection between their work and prior art. additionally, users of citation recommendation systems can benefit from natural language explanations of recommendation system choices. in addition to the utility of this task to scientists, it presents several interesting technical challenges. these include effectively representing the important information in a document, generating from a long-tailed technical vocabulary, and expressing the variety of connections between related scientific papers. figure 1 illustrates how the same document is described differently in relation to different documents. in this paper we use citing sentences to operationalize the problem of generating natural language explanations of the relationships between two scientific papers. authors, when citing other work, oftentimes describe how their work relates to the cited work. to this end, we use in-text citation sentences as a naturally occurring proxy explanations for how two documents relate to each other. however, we generate such sentences from general representations of document content rather than the specific in-text locations where these sentences occur, as this task formulation can better facilitate the applications described above. we approximate the explanation objective by having a gpt2 language model generate sentences containing citations given a pair of documents. this approach relies on providing dense but informative representations of documents to use as conditioning context for the generation model. we explore the use of sentence-based contexts as input including document abstracts, introductions, and sampled sentences from the full document; we find that using introductions and abstracts works well. finally, we improve our model's performance on automated metrics by using informative entities and terms to both construct dense input and rank the output relationship explanations. in addition to standard automatic metrics, we perform human evaluations of technical outputs with a pool of annotators. in this work, we describe a series of stages of model development, each with its own experiments that, together, informed the task and our series of solutions. our contributions include: a novel dataset for the relationship explanation task; a domain-adapted gpt2 we release for left-to-right language modeling of scientific text; the scigen model for describing document relationships; and an extensive expert evaluation and analysis of machine generated technical text. 1 ## related work the current work builds on recent research in scientific document understanding, including citation recommendation, intent categorization, and scientific document summarization. citation recommendation systems suggest related works given a 1 https://github.com/kel-lu/scigen document or a span of text @xcite @xcite . recently, researchers have sought to categorize citations using various ontologies of citation intents. @xcite develop an annotation scheme and corresponding classification model for citation functions. @xcite seek to discern "highly influential" citations from others. @xcite use six categories including "motivation," "uses," and "future work" among others. @xcite condense this ontology to just three: "background," "method," and "result comparison." intent classification can identify relationships between documents; our relationship explanation task extends this in two ways. first, datadriven freeform generation can express a wider array of relationships compared to a manuallydefined label set. further, our task framework could be used to describe relationships between works which do not actually cite each other, such as contemporaneous works. unlike categorization techniques, we require no task-specific annotated data as we supervise with citing sentences that are readily available in scientific documents. in practice, citation classification is used to assist in suggesting relevant works to researchers; our work complements this goal by providing rationales for the recommendation and furthering progress toward explainable ai. our work is also connected to a long history of research on summarizing scientific documents @xcite . work in this area has mostly used used abstracts or peer reviews as targets @xcite @xcite . in particular, @xcite show that using a simple extractive summary as input for abstractive summarization of scholarly texts work well. researchers have also used citing sentences as part of the input for summarization, recognizing the explanatory power of these texts @xcite @xcite . ours is the first work to focus on learning to express the specific relationship between two documents from such sentences. the closest work to our own is @xcite , who pilot a task of in-line citation generation. their goal is a model which can insert a citing sentence into a particular context within a document. our work, on the other hand, aims to learn from citing sentences how to describe general relationships between documents independent of particular indocument contexts. while the xing et al. ( 2020 ) method may facilitate writing assistance, our task has applications in search and summarization. because our task does not rely on a specific location in a document where the citation will go, solutions can be used at scale to provide users with general explanations of document relationships. our models rely heavily on recent advances in transfer learning in nlp. large pretrained models such as bert @xcite and gpt2 @xcite have made strong advances on a number of tasks @xcite . it has also been shown that pretraining these models on domain-specific data further improves results on domain-specific tasks @xcite . in this work, we apply that methodology by adding a pretraining phase on in-domain data before finetuning a gpt2 model toward the explanation generation task. a key challenge when using pretrained language models for documentlevel tasks is how to select document content to fit within the limited context window of the model, which is a major focus of our work. ## task overview we aim to generate an explanation: a natural language sentence which expresses how one document relates to another. explicit examples of such sentences are nontrivial to find in corpora, especially when annotation for a highly technical task is expensive. to this end, we use in-text citations in a scientific document to prior work as proxies for relationship explanations. we use these citing sentences as partial supervision for our task, and refer to them as "explanations." 2 we distinguish one document as the principal document, from which we will draw explanations that reference the cited document. let t denote an explanation drawn from principal document s, and s denote s without t. then let describes the relationship between new documents ŝ and ĉ. optimizing equation 1 is made easier by modern representation learning. pretrained neural language models like gpt2 have shown strong performance when generating sentences conditioned on a context. however, existing implementations of gpt2 limit the context window to 512 or 1024 tokens, far smaller than scientific documents. in this work, we explore ways to represent the documents' content for use with language models. data we use english-language computer science articles and annotation from s2orc dataset @xcite . s2orc is a large citation graph which includes full texts of 8.1 million scientific documents. we use 154k connected computer science articles, from which we extract 622k explanations with a single reference that link back to other documents in our corpus. we omit any sentences that cite more than one reference. we hold 5000 sentences for each of the validation and test sets. detailed statistics can be found in ## models we develop several models for explaining document relationships. following current work in neural text generation, we finetune the predictions of a large pretrained language model to our task (section 4.1). in order to bring the language model into the scientific text domain, we do additional language model pretraining over full scientific texts. we also investigate approximate nearest neighbor methods to retrieve plausible human-authored explanations from the training data as a baseline (section 4.2). ## representing documents with sentence selection methods for the related task of citation recommendation have made use of abstracts, which perhaps act as sufficient summaries of document content. building on this, we represent the principal and cited documents with the first 450 tokens of either their abstracts, introductions, or sentences randomly sampled from throughout the full document. 4 in this section, we answer two questions: 1) do neural generation models with sentence-based context outperform the ir baseline and 2) does the type of sentence-based context (abstract, introduction, sampled) matter? we answer these questions by performing both automatic and human evaluations. ## using ie-extracted term lists compared to the gold explanations, we found that our generated explanations miss important phrases such as unique model or dataset names and other lower-frequency terms; generally, they lacked specificity. the missing phrases typically appear in the cited document after the abstract and introduction. 8 naïvely sampling from the full text does not capture them due to sparsity. to address this issue, we explore more sophisticated information extraction (ie) techniques for constructing the conditioning context for scigen. recent work has shown that pretrained language models can adapt to disfluent inputs such as linearized trees and graphs @xcite . inspired by this, we investigate whether we can use lists of salient words and phrases to effect a dense representation of the cited document in the conditioning context. specifically, we construct a list of document-specific terms using tf-idf to score unigrams and entities extracted with a state-of-the-art scientific ner system. the paradigm is illustrated in figure 3 . tf-idf tf-idf is a measure of the frequency of a term in a document, normalized by the document frequency of that term. in our use, we calculate the tf-idf score for each unigram in the cited document. we keep the 100 highest scoring terms w i sorted in descending order of scores. the terms of this list are concatenated with a special token ξ tf to signal that this part of the input is structured as a list rather than conventional text. the resulting context x @xmath0, ξ tf , w 2 , ξ tf , ..., ξ tf , w 100 is used to represent the cited document to the scigen model. principal cited intro entity/terms scigen input intro x entities/terms scigen candidate 1 candidate 2 candidate 20 candidate 2 ## discussion example system outputs for selected test datapoints are shown in a case where the model identifies a correct relationship between the two documents. in this instance, they both use the pinyin representation for chinese characters in their transliteration models. output 2 demonstrates a failure of the explanation generation system. the principal document deals with the topic of discourse relations, the automatic identification of which is a long-standing machine learning task. however, this particular document is an analysis paper, and does not involve any training. output 3 is an example of a "too vague (but not incorrect)" case from the analysis in section 6.2. here again the explanation generated by scigen is topical, dealing with the concept of "distant supervision" that is key to both input documents. however, this sentence fails to capture the specific use that the principal makes of the research described in cited document. the final example, output 4, showcases potential for our system to explain concurrent work. the generated text summarizes the cited and implies that principal will build on that work. however, selected papers are both concurrent generation papers published in the same venue and do not cite each other. this appears to be a weakness in using citation sentences as proxies for relationship explanations. citations of contemporaneous work occur less frequently, so these types of sentences appear less often in training. similarly, relationship explanations between papers with more distant connections (e.g., "multi-hop" in the citation graph) principal: a syllable-based name transliteration system 1 cited: a joint source-channel model for machine transliteration scigen: following cited , chinese characters are considered as pinyin sequence. principal: recovering discourse relations: varying influence of discourse adverbials 2 cited: the benefits of a model of annotation scigen: the two text collections provided by cited were used for training, and the other two text collections were used for evaluation. principal: coreference resolution for swedish and german using distant supervision 3 cited: collective cross-document relation extraction without labelled data scigen: it is one of the most widely used distant supervision techniques and is inspired by techniques proposed by cited. principal: neural text generation in stories using entity representations as context 4 cited: delete, retrieve, generate: a simple approach to sentiment and style transfer scigen: the authors of cited proposed a model that combines neural generation with user interaction to create an objectcentric reading experience. it is notable that the scigen model usually out-puts syntactically correct and topical explanations, even given the difficulty of the vocabulary in this domain. this is consistent with many recent findings using domain-specific language models. the fluency and appropriateness of scigen's generations shows the promise of generating explanations which accurately capture the relationship between two documents. based on the results obtained here, we expect pretrained scientific language models to persist as a foundation. future work should focus on two complementary goals: ensuring the factual accuracy of the generated text and improved modeling of the cited document. factual accuracy is difficult to enforce in language model-based text generation systems, especially where inference includes sampling procedures. the use of information extraction for contexts showed promise in section 6; other methods of incorporating information like grounding to knowledge bases could help prune false or irrelevant statements. combining knowledge graphs with language models and generation is an active research area that has shown promise in other domains @xcite @xcite . applying this line of work to scientific text by modeling input documents as knowledge graphs of their content may help algorithms better understand the cited document, provide distant supervision for concurrent work, and result in better outputs. ## conclusion we have described a task of explaining the relationship between two scientific texts and its connections to facilitating researcher productivity. we employ a large, publicly available dataset of scientific documents to train a domain-adapted leftto-right language model for use in text generation applications and beyond. we explore a collection of techniques for representing document content including using abstracts, introductions, sampled sentences, and lists of informative terms and entities. we conduct thorough human and automatic evaluations to determine the relative strengths of each representation for expressing document relationships in natural language text. padded with a special token. the two contexts and the target citation sentence are then concatenated together with special separator tokens. figure 4 depicts our dataset construction. to construct the data splits, we randomly select 500 principal documents for both test and validation sets. the citation sentences that occur in these principal documents are used as examples in the test (5310 examples) and validation (5164 examples) sets. of the remaining examples where the principal documents were not in evaluation sets, we throw out any citation sentences that use an evaluation document as the cited document. the resultant examples are used for training (609509 examples). this construction allows us to ensure that the cited document is unseen at test time. 15