S ci REX : A Challenge Dataset for Document-Level Information Extraction (2020)

Abstract: Extracting information from full documents is an important problem in many domains, but most previous work focus on identifying relationships within a sentence or a paragraph. It is challenging to create a large-scale information extraction (IE) dataset at the document level since it requires an understanding of the whole document to annotate entities and their document-level relationships that usually span beyond sentences or even sections. In this paper, we introduce SciREX, a document level IE dataset that encompasses multiple IE tasks, including salient entity identification and document level N-ary relation identification from scientific articles. We annotate our dataset by integrating automatic and human annotations, leveraging existing scientific knowledge resources. We develop a neural model as a strong baseline that extends previous state-of-the-art IE models to document-level IE. Analyzing the model performance shows a significant gap between human performance and current baselines, inviting the community to use our dataset as a challenge to develop document-level IE models. Our data and code are publicly available at https://github.com/allenai/SciREX .

## introduction extracting information about entities and their relationships from unstructured text is an important problem in nlp. conventional datasets and methods for information extraction (ie) focus on within-sentence relations from general newswire text @xcite . however, recent work started studying the development of full ie models and datasets for short paragraphs (e.g., information extraction from abstracts of scientific articles as in scierc @xcite ), or only extracting relations (given ground truth entities) on long documents (e.g. @xcite ). while these tasks provide a reasonable testbed for developing ie models, a significant amount of information can only be gleaned from analyzing the full document. to this end, not much work has been done on developing full ie datasets and model for long documents. creating datasets for information extraction at the document level is challenging because it requires domain expertise and considerable annotation effort to comprehensively annotate a full document for multiple ie tasks. in addition to local relationships between entities, it requires identifying document-level relationships that go beyond sentences and even sections. figure 1 shows an example of such document level relation (dataset: squad, metric: em, method: bidaf, task:machine comprehension). in this paper, we introduce scirex, a new comprehensive dataset for information extraction from scientific articles. our dataset focuses on the task of identifying the main results of a scientific article as a tuple (dataset, metric, task, method) from raw text. it consists of three major subtasks, identifying individual entities, their document level relationships, and predicting their saliency in the document (i.e., entities that take part in the results of the article and are not merely, for example, mentioned in related work). our dataset is fully annotated with entities, their mentions, their coreferences, and their document level relations. to overcome the annotation challenges for large documents, we perform both automatic and manual annotations, leveraging external scientific knowledge bases. an automatic annotation stage identifies candidate mentions of entities with high recall, then an expert annotator corrects these extracted mentions by referring to the text of the article and an external knowledge base. 1 this strategy significantly reduces the time necessary to fully annotate large documents for multiple ie tasks. in addition, we introduce a neural model as a strong baseline to perform this task end-to-end. our model identifies mentions, their saliency, and their coreference links. it then clusters salient mentions into entities and identifies document level relations. we did not find other models that can perform the full task, so we evaluated existing stateof-the-art models on subtasks, and found our baseline model to outperform them. experiments also show that our end-to-end document level ie task is challenging, with the most challenging subtasks being identifying salient entities, and to a lesser extent, discovering document level relations. the contributions of our paper are as follows, 1. we introduce scirex, a dataset that evaluates a comprehensive list of ie tasks, including n -ary relations that span long documents. this is a unique setting compared to prior work that focuses on short paragraphs or a single ie task. 2. we develop a baseline model that, to the best of our knowledge, is the first attempt toward a neural full document ie. our analysis emphasizes the need for better ie models that can overcome the new challenges posed by our dataset. we invite the research community to focus on this important, challenging task. most structured extraction tasks from among these have revolved around extraction from sentences or abstracts of the articles. a recent example is scierc @xcite , a dataset of 500 richly annotated scientific abstracts containing mention spans and their types, coreference information between mentions, and binary relations annotations. we use scierc to bootstrap our data annotation procedure (section 3.2). there has been a lack of comprehensive ie datasets annotated at the document level. recent work by @xcite tried to rectify this by using distant supervision annotations to build datasets for document-level relation extraction. in both datasets, the task of relation extraction is formulated as a binary classification to check if a triplet of ground-truth entities is expressed in the document or not. instead, our work focuses on a comprehensive list of information extraction tasks "from scratch", where the input is the raw document. this makes the ie model more interesting as it requires to perform entity extraction, coreference resolution, saliency detection in addition to the relation extraction. 2 general ie most work in general domain ie focus on sentence-level information extraction @xcite @xcite . recently, however, @xcite introduced docred, a dataset of cross-sentence relation extractions on wikipedia paragraphs. the paragraphs are of a comparable length to that of scierc, which is significantly shorter than documents in our dataset. previous ie work on the tac kbp competitions @xcite comprise multiple knowledge base population tasks. our task can be considered a variant of the tac kbp "cold start" task that discovers new entities and entity attributes (slot filling) from scratch. two aspects of our task make it more interesting, 1) our model needs to be able to extract facts that are mentioned once or twice rather than rely on the redundancy of information in their documents (e.g rahman et al. (2016) ), 2) tac kbp relations are usually sentence-level binary relations between a query entity and an attribute (e.g angeli et al. (2015) ), while our relations are 4-ary, span the whole document, and can't be split into multiple binary relations as discussed in section 3.1. end-to-end neural ie models with neural networks, a few end-to-end models have been proposed that perform multiple ie tasks jointly @xcite @xcite . the closest to our work is dygie++ @xcite , which does named entity recognition, binary relation extraction, and event extraction in one model. dygie++ is a span-enumeration based model which works well for short paragraphs but does not scale well to long documents. instead, we use a crf sequence tagger, which scales well. our model also extracts 4-ary relations between salient entity clusters, which requires a more global view of the document than that needed to extract binary relations between all pairs of entity mentions. ## model we develop a neural model that performs documentlevel ie tasks jointly in an end-to-end fashion. 4 this section details our model design (also summarized in figure 2 ). ## evaluation we compare our model with other recently introduced models. since we cannot apply previous models directly to our task, we evaluate on subtasks of our dataset and also evaluate on scierc (section 5.2). the other goal of the evaluation is to establish a baseline performance on our dataset and to provide insights into the difficulty of each subtask. to that end, we evaluate the performance of each component separately (section 5.3), and in the overall end-to-end system (section 5.4). in addition, we perform diagnostic experiments to identify the bottlenecks in the model performance. we report experimental setup and hyperparameters in appendix a. ## conclusion we introduce scirex, a comprehensive and challenging dataset for information extraction on full documents. we also develop a baseline model for our dataset, which, to the best of our knowledge, is the first attempt toward a neural document level ie that can perform all the necessary subtasks in an end-to-end manner. we show that using a document level model gave a significant improvement in terms of recall, compared to existing paragraphlevel approaches. this task poses multiple technical and modeling challenges, including 1. the use of transformer-based models on long documents and related device memory issues, 2. aggregating coreference information from across documents in an end-to-end manner, 3. identifying salient entities in a document and 4. performing n-ary relation extraction of these entities. each of these tasks challenges existing methodologies in the information extraction domain, which, by and large, focus on short text sequences. an analysis of the performance of our model emphasizes the need for better document-level models that can overcome the new challenges posed by our dataset. as our research community moves towards document level ie and discourse modeling, we position this dataset as a testing ground to focus on this important and challenging task.