CORWA : A Citation-Oriented Related Work Annotation Dataset (2022)

Abstract: Academic research is an exploratory activity to discover new solutions to problems. By this nature, academic research works perform literature reviews to distinguish their novelties from prior work. In natural language processing, this literature review is usually conducted under the “Related Work” section. The task of related work generation aims to automatically generate the related work section given the rest of the research paper and a list of papers to cite. Prior work on this task has focused on the sentence as the basic unit of generation, neglecting the fact that related work sections consist of variable length text fragments derived from different information sources. As a first step toward a linguistically-motivated related work generation framework, we present a Citation Oriented Related Work Annotation (CORWA) dataset that labels different types of citation text fragments from different information sources. We train a strong baseline model that automatically tags the CORWA labels on massive unlabeled related work section texts. We further suggest a novel framework for human-in-the-loop, iterative, abstractive related work generation.

## introduction academic research is an exploratory activity to solve problems that have never been solved before. by this nature, each academic research work must sit at the frontier of its field and present novel contributions that have not been addressed by prior work; in order to convince readers of the novelty of the current work, the authors must compare against the prior work. while the format may vary among different fields, in natural language processing (nlp), this literature review is usually conducted under the "related work" section. since each paper must review the relevant prior work in its field, which is shared among papers on the same topic or task, many related work sections in a given field can be similar in both content and format. therefore, figure 1 : an example of corwa labels displayed using the brat interface @xcite . the task of automatic related work generation is that of generating the related work section of a target paper given the rest of the target paper and a set of papers to cite. prior works (hoang and @xcite @xcite @xcite @xcite mostly simplify related work generation as a general summarization task, generating related work sections using sentence-level models. this approach ignores the nature of the related work section, which consists of variable-length text fragments derived from different information sources. these text fragments refer to different cited papers, and they range in length from a few words to multiple sentences. there are also non-citation, supporting sentences that serve various discursive roles, such as introducing new topics, transitioning between topics, or reflecting on the current work. we argue it is necessary to dis-tinguish these heterogeneous text fragments, rather than treating related work sections as concatenations of homogeneous sentences. in addition to the heterogeneous information sources for related work section sentences, the writing styles of these sentences also vary. @xcite classify literature reviews to be integrative or descriptive, depending on whether they focus on high-level ideas or provide more detailed information on specific studies. however, this documentlevel classification scheme was intended as a descriptive, information science study of related work sections, and it has not been previously used in automatic related work generation. inspired by these observations, as a first step towards linguistically-motivated related work generation, we present a citation oriented related work annotation (corwa) dataset of related work sections from nlp papers. we distinguish text fragments from different information sources by tagging each sentence with discourse labels and identifying the spans of tokens belonging to each citation. we further distinguish citations that give detailed explanations of cited papers and those that illustrate high-level concepts. our main contributions are as follows: (1) we collect a corwa dataset that decomposes the related work section with three inter-related annotation tasks -discourse tagging, citation span detection, and citation type recognition -and demonstrate the significance of corwa with analyses from multiple perspectives ( §3). ( 2 ) we propose a strong baseline model that automatically tags the corwa annotation scheme on massive unlabeled related work section texts ( §4). (3) we show that citation spans are a better target than citation sentences with two example tasks ( §5). (4) we discuss a novel framework for human-in-the-loop, iterative, abstractive related work generation ( §6). ## related work extractive related work generation. early related work generation systems employed the extractive summarization approach. hoang and kan (2010) pioneered the task, developing rules to select sentences following a topic hierarchy tree that was assumed to be given as input. @xcite grouped sentences into topic-biased clusters with plsa, modeled sentence importance with svr, and applied a global optimization framework to select sentences. @xcite se-lected sentences from papers that co-cited the same cited papers as the target paper in order to cover a minimum steiner tree constructed from the paper's keywords. @xcite extracted cited text spans (cts), the matched text spans in the cited paper that are most related to a given citation. however, these extractive approaches aim to maximally cover the citation texts with the extracted sentences, thus mostly ignoring the reference type citations that are concise and abstractive ( §3.1.3). abstractive related work generation. recently, xing et al. (2020) extend the pointergenerator @xcite to take two text inputs, allowing them to recover a masked citation sentence given its neighboring context sentences. @xcite encode the citation context, cited paper's abstract, and citation network and train their model with multiple objectives: sentence salience score regression of the cited paper's abstract, functional role classification of the citation sentence, and citation sentence generation. @xcite propose a relation-aware, multi-document encoder to generate a related work paragraph given a set of cited papers. @xcite fine-tune gpt2 @xcite on scientific texts and explore several techniques for representing documents, such as using extracted named entities. all of the works described above focus on the generation aspect, while neglecting dataset collection; their datasets are mostly extracted automatically. moreover, the datasets are not reused, though they are publicly available, because these works all use slightly different problem definitions, and thus the models are not directly comparable @xcite . in this work, we focus on collecting a dataset that is widely applicable to various related work generation settings, rather than proposing another incomparable approach. ## corwa dataset in this work, we limit our scope to publications from the nlp domain for ease of automatically extracting the related work section; existing work on related work generation has also focused on nlp in the past. we build our dataset on top of the nlp partition of the s2orc dataset @xcite , a large-scale corpus of scientific papers derived from l a t e x source code and pdf files. we extract the related work section by matching the section titles. because not all papers cited in the extracted related work sections are available in s2orc, we prioritize annotating related work sections where the majority of their cited papers are available. ## spans as an alternative to sentences we argue that the citation spans annotated in corwa are a better alternative to the citation sentences that have previously been used for the tasks of rouge-based retrieval and citation text generation. ## toward full related work generation existing extractive related work generation systems @xcite @xcite select sentences from the target paper and/or the cited papers, which can be concatenated to form a full related work section; neural network-based, abstractive related work generation systems generate individual citation sentences @xcite @xcite or paragraphs @xcite . however, none of these prior works address the ordering of the extracted/generated sentences or the grouping of sentences into paragraphs, nor are they able to produce rhetorical sentences to smooth the transitions between citations. no prior work bridges the gap from generating individual citation texts to generating a full related work section. we suggest a bottom-up, iterative approach to generate full related work sections. the process would begin with generating citation spans under the settings proposed in §5.2. then, multiple generated citation spans would be aggregated and rewritten into citation text blocks in either the summarization or narrative style. these blocks would be further aggregated and rewritten into paragraphs by generating transition and reflection sentences. generating and rewriting in this pipeline fashion has the following benefits: (1) it mitigates the practical issue of computational resource limitations, given that state-of-the-art models do not perform well on long text generation. (2) the auxiliary inputs, such as citation functions or discourse tags, may vary for each stage of generation. (3) as a practical system to assist researchers, it is crucial to allow user involvement in the iterative generation process. due to the large search space, consisting of multiple valid related work section candidates with different writing styles, it is extremely challenging to precisely generate a satisfying text with a one-shot, end-to-end system. a human-in-the-loop approach allows the user to significantly prune the search space and simultaneously reduces the errorpropagation issue caused by the pipeline design. ## conclusion we present the corwa dataset of three interrelated annotation tasks: discourse tagging, citation span detection, and citation type recognition. we demonstrate the significance of corwa with analyses from multiple perspectives, such as writing style and discourse patterns. we propose a strong baseline model that can automatically propagate the corwa annotation scheme to massive unlabeled related work sections. furthermore, we show that citation spans are a better alternative to citation sentences for both the relevant sentence retrieval and citation generation tasks. finally, we discuss a novel framework for human-in-the-loop iterative abstractive related work generation.