DMDD : A Large-Scale Dataset for Dataset Mentions Detection (2023)

Abstract: The recognition of dataset names is a critical task for automatic information extraction in scientific literature, enabling researchers to understand and identify research opportunities. However, existing corpora for dataset mention detection are limited in size and naming diversity. In this paper, we introduce the Dataset Mentions Detection Dataset (DMDD), the largest publicly available corpus for this task. DMDD consists of the DMDD main corpus, comprising 31,219 scientific articles with over 449,000 dataset mentions weakly annotated in the format of in-text spans, and an evaluation set, which comprises 450 scientific articles manually annotated for evaluation purposes. We use DMDD to establish baseline performance for dataset mention detection and linking. By analyzing the performance of various models on DMDD, we are able to identify open problems in dataset mention detection. We invite the community to use our dataset as a challenge to develop novel dataset mention detection models.

## introduction as the volume of scientific literature continues to grow, the automatic extraction of scientific entities from publications becomes increasingly valuable for knowledge management and scientific discovery. in particular, accurate and efficient detection of dataset mentions in the literature can greatly improve the accessibility and usability of scientific data. dataset mention detection (dmd) presents distinct challenges compared to other named entity recognition (ner) tasks, because the vocabulary used in scientific literature is complex and diverse across different subjects. domain expertise is often needed to recognize datasets mentioned in the literature. furthermore, the mention of dataset entities can be potentially ambiguous as the same names may be used to refer to method or task entities. for example, 'sgd' is used to represent the stochastic gradient descent method and the schema-guided dialogue dataset. lastly, it is crucial for dataset mentions to include linking annotation, which allows linking dataset mention to its definition website, such as on github. however, datasets have very diverse ways of being mentioned in the literature, which creates additional challenges for linking. all of these unique issues pose non-trivial challenges both to the annotation generation process and to mention recognition models. there are a number of manually compiled corpora that aim to aid the study of dataset name detection in scientific literature @xcite @xcite @xcite @xcite @xcite . these corpora have certain limitations. for instance, many of them have only a few hundred instances, which limits their usefulness for training and evaluating dataset mention detection models. additionally, some of these corpora lack diversity in dataset naming conventions, as they may include mostly capitalized dataset names, whereas many dataset names are not capitalized. furthermore, these corpora lack entity linking information. those limitations limit their usefulness for developing dataset mention detection algorithms, despite being suitable for testing such algorithms. manual labeling can be prohibitively expensive for ner in general, and for dataset mention detection in particular, because it requires domain expertise. consequently, there is a need to develop labeled dataset mention data with less human effort to facilitate the training of robust detection models. one potential alternative to manual labeling is to utilize web pages such as github and the papers with code (pwc) website, 1 which provide metadata about scientific papers. however, these metadata are not always sufficient for building robust dataset mention detection models for two reasons. first, they are often created based on optional manual inputs, which may be prone to human errors and may not be available for new papers or arxiv papers. second, the metadata is only available at the document level, whereas in-text level mention annotations are required for mention detection models. in this paper, our objective is to enable the creation of robust dataset detection models by creating a large corpus, called dataset mentions detection dataset (dmdd). to construct the main corpus of dmdd, we employ distant supervision @xcite to develop in-paper annotation by taking the dataset mentions from the pwc website and matching them with those in papers. we also add entity linking annotations to each dataset mention. although the labels obtained by distant supervision may not be as precise as human-generated labels, their vast quantity and diversity of mentions provide an advantage in (pre)training competitive models @xcite . dmdd comprises 31,219 scientific articles with over 449,000 dataset mentions automatically annotated in the format of in-text span. additionally, it includes an evaluation set of 450 scientific papers where we carefully annotated the occurrence of each dataset mention. the dmdd corpus can be accessed at the following url: https://www.kaggle.com/datasets /panhuitong/dmdd-corpus . overall, our paper makes the following contributions: ## dmdd in this section, we provide a detailed description of the development of dmdd's main corpus and dmdd's evaluation set. we also present a comprehensive comparison between dmdd and related corpora. for the purposes of this study, we define dataset mentions as in-text spans that exclusively comprise the necessary texts (i.e., the name of the dataset) for finding the datasets in knowledge bases. as such, we exclude pronominal references to dataset entities, such as 'the dataset' and 'the images'. additionally, it is important to note that the dataset entity in dmdd is distinct from the material entity in other related works, such as scirex @xcite , as the dataset entity is only one of the components of the experimental materials that is annotated as the material entity. ## experimental setup the experiments are designed to address the task of dataset entity mentions and entity linking, with three primary objectives in mind: establishing baseline performance on our dataset, providing insights into the difficulty of each task, and evaluating the effectiveness of using dmdd for training. ## experimental results this section describes the experimental setup and results of the conducted experiments. ## limitations and future work the dmdd corpus is annotated through distant supervision, which prioritizes scale over accuracy. the current scope of dmdd is limited to dataset mentions that can be linked to the dmdd dictionary, resulting in missing labels for dataset mentions that are not listed on pwc websites or that have variations not included in the regular expression. this limitation may introduce annotation noise, especially when dealing with dataset subversions that are not explicitly listed in pwc. furthermore, dmdd does not include annotations for ambiguous cases, where distinct datasets have the same name or share acronyms, nor does it consider changes in naming conventions over time. similar limitations apply to other corpora created using distant supervision, as annotation accuracy heavily relies on manual correction. to address these limitations, future work can focus on developing more advanced methods for mention detection and exploring alternative approaches to distant supervision. additionally, dmdd can be extended to include annotations for more challenging test instances, such as unseen mentions, ambiguous mentions, and mentions with diverse sub-versions. we also plan to periodically release revised versions that have larger sizes and additional (manual) annotations for scientific entities such as model and method names. in terms of model performance, the baseline models showed limitations when presented with unseen entities, lengthy inputs, and multiple entities. these challenges highlight the difficulties of dataset mention detection and linking in scientific literature. to develop a more robust mention detection method, future research may also explore end-to-end framework for dataset entity mention detection and linking, advanced detection networks that are robust to noise in training data, or how to leverage the context out of the mention sentence to boost the performance of el. in addition to these approaches, future work may also explore the use of footnotes and citations in literature to improve dataset entity recognition. ## conclusion in conclusion, dmdd is a valuable resource for studying dataset mention detection in scientific literature. as the largest corpus created for this purpose, it addresses the limitations of existing corpora in terms of size, diversity of dataset mentions, and entity linking information. our experiments with baseline models show that dmdd enables the training of more robust models with a small number of manual labels, as demonstrated by the improved performance of scibert trained on dmdd compared to other corpora. the analysis of dmdd instances and experimental results highlight the challenges and open problems in the task of dataset mention detection. we believe that dmdd will stimulate further research in this important area of scientific information extraction.