S ci ER : An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents (2024)

Abstract: Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations). Several datasets have been proposed for training and validating SciIE models. However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context. In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations. To capture the intricate use and interactions among entities in full texts, our dataset contains a fine-grained tag set for relations. Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation. We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM-based baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.

## introduction scientific information extraction (sciie) is a core topic of scientific literature mining @xcite @xcite @xcite . it typically includes scientific named entity extraction (sciner) and scientific relation extraction (scire), and plays a critical role in downstream applications, including scientific knowledge graph construction @xcite , data searching @xcite , academic question answering @xcite , and method recommendation @xcite . scientific large 1dataset and code are publicly available: https:// github.com/edzq/scier 𝑆 1 : we train a deep cnn for semantic segmentation. 𝐸 1 : method 𝐸 2 : task used-for task input output ner 𝑆 1 𝐸 1 , 𝐸 2 re 𝑆 1 , [𝐸 1 , 𝐸 2 ] used-for ere 𝑆 1 [𝐸 1 , used-for, 𝐸 2 ] figure 1: top: an annotation sample of our scier dataset, illustrating the labeling process and data structure. the sentence 𝑆 1 contains two annotated spans denoting two entities 𝐸 1 and 𝐸 2 , with respective types method and task. bottom: a table detailing the input and output of the three tasks supported by our scier dataset, including named entity recognition (ner), relation extraction (re), and entity and relation extraction (ere). existing sciie datasets and benchmarks that support both sciner and scire are limited to extracting information from specific parts of papers, such as particular paragraphs @xcite or abstracts @xcite . however, scientific entities like datasets, methods, and tasks entities, are distributed throughout the entire text of papers. sentences in the body of a paper exhibit diverse linguistic styles and ways to mention entities @xcite and semantics @xcite , which allows the extraction of more fine-grained and precise relation types. for example, abstracts do not say that method x is trained on dataset y, but experimental sections give such details. therefore, focusing on specific parts of scientific articles is likely to miss important information. several datasets @xcite @xcite attempt to create sciie benchmarks with full-text annotation, but they ignore the scire task. in this paper, we present scier, an entity and relation extraction dataset for identifying dataset, method, and task entities in scientific documents as well as the relations between them. our dataset is large, with 24k entities and 12k relations from 106 scientific articles, enabling the evaluation and development of sciie models. these documents are taken from the publications included in papers with code (pwc)2, covering artificial intelligence (ai) topics, such as natural language processing (nlp), machine learning (ml), computer vision (cv), and ai for science (ai4science). figure 1 shows an annotated sentence from our dataset, which gives the entities, their types, i.e., method and task, respectively, and the relation between them used-for. our dataset can be used to evaluate ner and re as separate tasks, but it can also support the evaluation of end-to-end entity and relation extraction (ere) from scientific publications @xcite . the table in figure 1 describes those settings. for example, in ner the input is a sentence and the output is the set of entities in the sentence. in re, the input is the sentence along with the entities and the output is the relation between those entities. finally, in ere the @xmath0, relation, @xmath1. we address the limitations of existing datasets by annotating entire scientific papers for both entity and their relations. this is a much harder task 2https ://paperswithcode.com/ compared to annotating abstracts. furthermore, comparing with existing datasets @xcite @xcite , we provide more fine-grained relation types to describe the interactions between datasets, methods, and tasks. for example, we use trained-with and evaluated-with to describe the interactions between methods and datasets. these relation types need to be extracted from the body of a paper, and are not supported by previous datasets. §3.3 gives a detailed comparison between our dataset and existing ones. finally, to evaluate the model's robustness to temporal and conceptual shifts in the sciie, we set in-distributed (id) and out-of-distribution (ood) test sets. the documents in the ood set were all published after the training documents and feature entirely different topics. we conduct evaluation experiments by employing three stateof-the-art supervised methods and llms-based in-context learning (icl) methods and provide analysis. specifically, for llms-based methods, we tested both pipeline and joint approaches, optimizing the prompts through retrieval-based icl, tag-based entity extraction, and the incorporation of annotation guidelines. the experimental results show that for llms, pipeline modeling, which splits the ere task into two sub-tasks of ner and re, outperformas joint extraction. in the challenging ere task, the best supervised method achieves an f1 score of 61.10%, while the best llm method achieves an f1 score of 41.22%. our contributions can be summarized as follows: ## related work many datasets for sciner have been proposed. @xcite and dmdd @xcite are two datasets for dataset mention detection. the @xcite dataset comprises 6000 annotated sentences selected based on the occurrence of dataset related word patterns from four major ai conference publications. dmdd is annotated on the full text and comprises 31219 scientific articles automatically annotated with distant supervision @xcite . tdmsci @xcite supports three types of entities: task, dataset, and method. it has 2000 sentences extracted from nlp papers. scirex @xcite offers comprehensive coverage with 438 full text annotated documents and supports four entity types: task, dataset, method, and metric. scirex does not annotate relations between pairs of those entity types. @xcite manually annotates 100 documents for fine-grained sciner by defining 10 different entity types in 3 categories: mlmodel related, dataset related and miscellaneous. scidmt @xcite uses the pwc as knowledge created a very large scale dataset for data, method, and task. scidmt includes 48 thousand scientific articles with over 1.8 million weakly annotated mention annotations in their main corpus. however, given the inherent complexity of the ner task, employing weak labels may cause models to overfit on noisy data, thereby substantially impacting their performance @xcite @xcite . although there has been growing interest in research on developing methods and datasets for sciie, very few datasets support both ner and re tasks for scientific text. an overview of existing sciie benchmarks that support both sciner and scire is shown in compared to those resources, our dataset contains 106 scientific publications with minute manual annotations. the dataset has nine relation types, allowing for more nuanced relations between entities. the scale of our dataset, which contains more than 24k entities and over 12k relations, which is significantly larger than previous datasets, except for those that are created with distant supervision. ## scier in this section, we detail the curation of our dataset, including data collection process in §3.1, the data annotation process in §3.2, and present the final dataset statistics and comparisons in §3.3. ## experiments in this section, we provide the details of evaluation experiments of both state-of-the-art supervised baselines and llms-based baselines on the proposed dataset. we first formally define the problem of end-to-end relation extraction in §4.1, then describe the supervised methods in §4.2 and the llms-based methods in §4.3. finally, we present our implementation details in §4.4 and evaluation settings in §4.5. ## conclusion we introduce scier, a dataset for entity and relation extraction in scientific documents, specifically focusing on datasets, methods, and task entities. to address the limitations of existing datasets, we annotate entire scientific papers for both entities and relations, resulting in a large-scale dataset comprising 106 full-text scientific publications from various ai topics, containing over 24,000 entities and 12,000 relations. additionally, we introduce a fine-grained relation set to describe the interactions between datasets, methods, and tasks. to evaluate the model's robustness to emporal and conceptual shifts in the sciie, we also set an ood test set. we conduct comprehensive evaluation experiments, including supervised state-of-the-art (sota) models and llm-based icl baselines, to highlight the challenges in this task. specifically, for llm-based methods, we tested both pipeline and joint approaches, optimizing the prompts through retrieval-based icl, tag-based entity extraction, and the incorporation of annotation guidelines. the experimental results of llms-based methods show that: ❶ for the ere task, pipeline modeling, which decomposes the task into ner and re sub-tasks, significantly outperforms joint modeling; ❷ although llm-based approaches require less labeled data, there remains a performance gap compared to supervised methods. for future work, we aim to further optimize prompts to enhance the performance of llms in scientific information extraction (sciie) and domain-specific ie tasks. additionally, a llm-in-the-loop data annotation system to reduce the high costs of creating domain-specific ie datasets is feasible.