GSAP - NER : A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets (2023)

Abstract: Named Entity Recognition (NER) models play a crucial role in various NLP tasks, including information extraction (IE) and text understanding. In academic writing, references to machine learning models and datasets are fundamental components of various computer science publications and necessitate accurate models for identification. Despite the advancements in NER, existing ground truth datasets do not treat fine-grained types like ML model and model architecture as separate entity types, and consequently, baseline models cannot recognize them as such. In this paper, we release a corpus of 100 manually annotated full-text scientific publications and a first baseline model for 10 entity types centered around ML models and datasets. In order to provide a nuanced understanding of how ML models and datasets are mentioned and utilized, our dataset also contains annotations for informal mentions like “our BERT-based model” or “an image CNN”. You can find the ground truth dataset and code to replicate model training at https://data.gesis.org/gsap/gsap-ner.

## introduction throughout various disciplines, the scientific process constantly produces new knowledge, innovative discoveries, and valuable insights, which typically are published in conference proceedings and journal articles. the increasing volume of scholarly artifacts underscores the importance for scientists to efficiently locate, comprehend, and utilize these resources in their daily work. consequently, the nlp community is constantly creating methods to extract named entities from the scholarly domain, recognizing its significance in facilitating scientific understanding. with the advent of artificial intelligence (ai), the landscape of machine learning (ml) approaches has evolved, incorporating techniques such as deep learning (dl) and fine-tuning of large language mlmodelgeneric our model dubbed mlmodel xlm-r , significantly outperforms mlmodel multilingual bert ( mlmodel mbert ) on a variety of datasetgeneric cross-lingual benchmarks , including +14.6% average accuracy on dataset xnli , +13% average f1 score on dataset mlqa , and +2.4% f1 score on dataset ner . figure 1 : example of machine learning models and dataset related mentions annotated according to our tag set. "our model" and "cross-lingual benchmarks" are considered informal mentions, whereas the others are noun phrases mentioning named entities. models (llm). consequently, to effectively comprehend scientific articles related to ml, ai, or data science, it becomes crucial to identify and comprehend these emerging entity types. developing effective ner models for these entities requires annotation guidelines and ground truth datasets to train robust language models @xcite . however, existing guidelines and ground truth datasets for scholarly entities have not adequately addressed the finer-grained entity types, such as ml models and datasets as distinct entities. instead, state-of-the-art works treat ml models as methods @xcite , failing to differentiate between the model instance, type, and underlying architecture. similarly, dataset mentions are typically categorized as material, overlooking the fact that this can also encompass knowledge bases, resources, or other corpora. this paper presents gsap-ner 1 , a ground truth dataset specifically designed to enable the development of language models tailored for identifying named entities associated with the interplay between machine learning models and datasets. it benefits from a detailed annotation scheme that is customized for the discussion and use of machine learning models and the data used. we address the limitation of existing datasets by emphasizing comprehensive annotation of full scientific paper annotation rather than solely focusing on annotated abstracts or pre-selected sections. our dataset offers two significant advantages. firstly, we place particular emphasis on capturing informal mentions of named entities. these unnamed, descriptive mentions indirectly relate to named entities (e.g., "cross-lingual benchmarks" in figure 1 ), providing valuable training data for co-reference resolution tasks. secondly, our dataset features nested entity annotations @xcite , enabling the annotation of multiple sub-parts of a text span within a single noun phrase. based on our ground truth we fine-tune a first baseline model for our task of ml model and dataset named entity recognition. we employ three state-of-the-art baseline models for that, scib-ert @xcite , roberta @xcite , and a more recent pre-trained language model called scideberta-cs @xcite . we have found that scideberta-cs performs best on the entity types mlmodel and dataset, with an f1 score of 0.71 and 0.81, respectively. creating a sizable ground truth dataset like gsap-ner is costly in terms of effort. therefore, as a final experiment, we explore the minimum quantity of fully-annotated texts required to observe a noteworthy improvement in performance by incrementally increasing the size of the training data. our dataset enables researchers and practitioners to extract precise and domain-specific information, contributing to fields such as information retrieval, scientific knowledge mining, automated literature analysis, and knowledge graph creation. our research presents four key contributions, each aimed at advancing the state of scholarly entity and concept detection: all materials, such as the ground truth dataset and the code to replicate model training can be found at https://data.gesis.org/gsap/gsap-ner . ## related work among the works dealing with the task of scholarly information extraction, in this section we focus on those named entity recognition foot_1 methods which are machine-learning-based and not rule-based approaches. as a general overview, @xcite gives a comprehensive list of approaches on information extraction from scientific publications. multiple datasets serve as ground truth datasets for named entity recognition (ner), each cater-ing to specific tasks. please consult among datasets working on abstracts, scierc stands out as a prominent dataset, comprising 500 abstracts extracted from 12 ai conference and workshop proceedings @xcite . this rich dataset contains annotations for scientific entities, their relationships, and co-reference clusters, which are invaluable for related nlp tasks. another dataset, scirex, offers comprehensive coverage with 438 fully annotated documents, specifically targeting mention identification and relationship extraction of entities related to methods, tasks, datasets, and metrics @xcite . to prepare their ground truth dataset for the ner task, they combined distant supervision and manual correction of automatically pre-annotated full-texts. in contrast to their work, we created a fully manually annotated corpus. this enables us to define our tag set independently of current approaches and to avoid potential model bias introduced by pre-annotation. @xcite contributed significantly to this area by presenting tdmsci, a corpus containing domain expert annotations for tdm entities in 2000 sentences extracted from nlp papers, alongside a dedicated tdm tagger designed for this specific task. @xcite developed a ground truth dataset for dataset mention detection, comprising 6,000 annotated sentences selected by the occurrence of dataset related word patterns that were sourced from four major ai conference publications. approximately half of them containing one or more named datasets. an emerging nlp task known as leaderboard extraction focuses on extracting task-dataset-metric-score (tdms) tuples from scholarly papers, enabling the generation of an aggregated comparison view of the main entities of interest @xcite . along this direction, @xcite introduced an extraction pipeline, axcell, for extracting results from tables listed in scientific articles. in 2022, d'souza and auer (2022) created cs-ner, a corpus of contribution-centric information extraction targets, namely research problem, method, solution, dataset, metric, and more. recent lines of research have explored end-toend frameworks based on nlp extraction tasks, such as ner, which involve a series of intercon-nected methods aimed at creating knowledge bases or knowledge graphs. @xcite focused on extracting the aim, method, and result sections from scientific articles, utilizing this information to construct a scientific knowledge graph. similarly, @xcite developed scinlp-kg, a framework designed to extract tdm entities and relations from papers in the nlp domain. furthermore, @xcite presented a computer science knowledge graph (cs-kg) that is automatically generated and periodically updated. they achieved this by applying an information extraction pipeline to a vast repository of research papers, offering a comprehensive and up-to-date resource for the computer science domain. ## dataset machine learning models and datasets are essential scholarly entities that are discussed in various scientific disciplines. for named entities, it is frequently observed that, depending on context, identical string spans refer to different semantics. take "bert" in natural language processing as an example: it can refer to a particular pre-trained model with fixed parameters like "bert base" or its architecture, depending on the context. in addition to named entities, unnamed or informal mentions of machine learning models or datasets are more common in scientific text. but those informal mentions often contain nested references to other named entities and thereby carry extra information linking not only to machine learning models or datasets but also to other scholarly entities such as methods, model architectures and tasks. an illustrative example can be found in ## conclusion we introduce gsap-ner, a manually annotated corpus over full-text scholarly publications from the computer science domain, designed for information extraction of ml models and datasets. by distinguishing ml models from methods and datasets from materials, our dataset enables researchers and services to gain deeper insights into the specific methods and materials employed in computer science research. we utilized our data and fine-tuned three state-of-the-art baseline models. the experiments showed that scideberta-cs reaches best performance on the majority of entities types, with an overall f1 score of 0.64 and 0.73 on exact span matches and partial span matches, respectively. despite the challenges involved in its creation, we believe gsap-ner remains a valuable resource for the development, evaluation, and benchmarking of ner models in the computer science domain. it offers researchers and practitioners a comprehensive and domain-specific dataset, addressing the limitations of existing datasets that often lack specialized entity differentiation. furthermore, this dataset can contribute to advancing research in areas such as information retrieval, scientific knowledge mining, automated literature analysis, and knowledge graph creation. as future work, we aim to employ a multi-stage model training approach and leverage additional background knowledge to disambiguate syntactic mentions from their semantic context. such background knowledge could come from ml ontologies and knowledge graphs, such as the orkg or cs-kg. we also envision exploring entity relationships, co-reference resolution, and entity attributes as future directions to enhance the value of this dataset.