Data Gatherer: LLM -Powered Dataset Reference Extraction from Scientific Literature (2025)

Abstract: Despite growing emphasis on data sharing and the proliferation of open datasets, researchers face significant challenges in discovering relevant datasets for reuse and systematically identifying dataset references within scientific literature. We present Data Gatherer, an automated system that leverages large language models to identify and extract dataset references from scientific publications. To evaluate our approach, we developed and curated two high-quality benchmark datasets specifically designed for dataset identification tasks. Our experimental evaluation demonstrates that Data Gatherer achieves high precision and recall in automated dataset reference extraction, reducing the time and effort required for dataset discovery while improving the systematic identification of data sources in scholarly literature.

## introduction the increasing availability of data has accelerated scientific progress. genomic and proteomic data sharing, for example, has enabled scientists to develop approaches that rely on access to large amounts of data @xcite . policies and frameworks like the fair principles @xcite ) and force11's joint declaration of data citation principles @xcite and changing researchers practices have contributed to increasing the amount of data available. yet, finding datasets for reuse and identifying datasets referenced in research papers remain a challenging and labor-intensive task @xcite @xcite . in contrast to journal article and book citation practices that use standardized formats (e.g., citation styles, dois), dataset references are inconsistent, ambiguous, and dispersed throughout scholarly documents, making systematic discovery difficult. pubmed and pubmed central, for exam-ple, make some dataset mentions available through linkout resources which links to external resources. they also allow researchers to search for articles that contain associated data in their data availability statement (das), a structured section of articles that describe datasets used, or inside similar sections. however, these indexes are not currently able to surface dataset mentions fully, especially those embedded in the article text. even when datasets are explicitly referenced, their mentions are often ambiguous. the same dataset may be cited under different names, abbreviations, or project titles across multiple papers. some papers provide only partial accession codes or omit repository information, making it difficult to resolve the dataset's location. das's, for example, may erroneously state that all data from a study is included in the paper @xcite . common issues like typos, incorrect identifiers, and broken links further hinder discovery. to locate datasets included in papers, researchers, librarians and data curators then have to undertake the labor-intensive process of manually searching, cross-referencing, and verifying dataset mentions. mentions may include metadata such as accession codes, repository names, urls, or informal descriptions. they can be embedded in figure captions, tables, supplementary materials, citations, or structured article sections like a das rather than explicitly listed in the main text. recent advances in large language models (llms) present unprecedented opportunities for automating the discovery and extraction of dataset mentions from scientific literature. llms demonstrate superior capability in recognizing complex patterns in natural language text, enabling them to identify dataset references across diverse formats and naming conventions while distinguishing them from superficially similar entities such as gene and experiment identifiers. this can lead to significant improvements in automated extraction. contributions. we introduce data gatherer 1 , an open-source, llm-powered system that automates the identification and extraction of structured dataset records from scientific publications. our system addresses the labor-intensive manual processes currently employed by researchers and librarians for dataset discovery. the design and development of data gatherer was informed by a collaboration with biomedical researchers specializing in proteomics and genomics, ensuring the tool addresses real-world requirements. to evaluate the effectiveness of data gatherer, we develop two benchmark datasets: (1) a high-quality collection carefully curated and validated by an expert librarian to ensure accuracy and completeness, and (2) a larger-scale dataset constructed through the systematic integration of existing databases that maintain associations between research articles and their referenced datasets. these benchmarks enable a comprehensive evaluation across different scales and quality standards. we present the results of an experimental evaluation which show that data gatherer achieves recall of up to 99.4% and precision up to 91.1% across our benchmark datasets. in summary, our main contributions are: (1) an llm-powered pipeline for automated identification and extraction of dataset references from scholarly documents (section 5); (2) development and curation of two benchmark datasets for evaluation of dataset extraction methods (section 4); and (3) an experimental evaluation of our data extraction methods using different llms (section 6). the related work on the extraction of dataset mention from scientific literature falls in two main categories, which we describe below. datasets for information extraction from scientific literature. several datasets have been developed to facilitate research in scientific information extraction. anzaroot and mccallum (2013) introduced a dataset for fine-grained citation field extraction, focusing on segmenting citation strings into components like title and authors. cheung et al. (2024) presented polyie, a dataset for extracting entities and relations specific to polymer materials. zhang et al. (2024) created scier, a dataset for entity and relation extraction with a focus on datasets, methods, and tasks. while these datasets facilitate various aspects of scientific information extraction, 1 https://github.com/vida-nyu/data-gatherer such as citation parsing and domain-specific entity extraction, we focus on a different task: the extraction of dataset references from the scientific literature on proteomics and genomics. the data citation corpus, created by make data count in collaboration with the chan zuckerberg initiative, is a comprehensive list of data citations from articles and preprints meant to facilitate the creation and use of data metrics similar to bibliometrics used to measure the impact of other scholarly outputs (e.g., h-index, impact factor, and the rcr) (make data count, 2025). the data citation corpus is in part compiled using machine learning methods that leverage scibert-based named entity recognition @xcite . make data count does not make these data citation location tools publicly available. in contrast, our tool is open source and freely accessible to researchers, and instead of focusing on the creation of data metrics, our goal is to enable users to identify all mentions of datasets within a collection of articles to facilitate data discovery and reuse. ## problem definition we aim to automatically discover and extract dataset references from scholarly publications, focusing on citations accessible in academic documents available on the web. definition 1. given a publication p (e.g., a url or doi that refers to a scholarly article), the goal is to build a function f that extracts a structured set of records {(d i , r i )} from p , i.e., f(p ) = {(d 1 , r 1 ), (d 2 , r 2 ), ..., (d n , r n )}, where d i is the dataset identifier, typically an accession code or another type of dataset reference, and r i is the repository name or reference (e.g., a plain text string or a url pointing to the repository). we consider a dataset reference valid if its identifier d i exists in the repository r i . to evaluate the ability of different approaches to identify and extract valid dataset references correctly, we built two benchmark datasets that are detailed in section 4. ## the dataref benchmarks to evaluate data gatherer, we constructed two datasets using distinct methodologies: ## the data gatherer tool data gatherer was designed to automatically extract dataset references from scientific publications by processing both html web pages and xml responses as discussed in section 3. it employs llms to identify references and construct links that enable the retrieval of the dataset. we have explored two main strategies: full-document read (fdr) and retrieve-then-read (rtr). ## experimental evaluation we evaluated data gatherer's performance using dataref-exp and dataref-rev (section 4). due to the size of the dataref-rev dataset, we used a sample consisting of 1,883 dataset citation records from 1,242 pubmed central articles. performance was assessed by using precision and recall metrics calculated for each paper and then averaged over the set of papers to compute the average precision and average recall. since identifiers are typically unique across repositories (e.g., dois), we compare only the identifiers to determine matches. to account for common identifier variations (e.g., doi: 10.6019/pxd123456 vs. accession code: pxd123456), we consider both exact or partial matches (e.g., substring match) for dataset identifiers. we report the results in while not conclusive, these results suggest that reducing the input size can help improve the cost (due to smaller input size) and the precision of long-context models (at the cost of decreasing the recall in some cases). thus, more accurate and general rtr methods could be beneficial to improve the overall results. we also note that the results reported on the dataref-rev dataset are limited, specially precision, since it may be possible that the models identify correct datasets that are not included in the ground truth (see section 4). ## conclusion researchers, librarians, and data curators currently spend significant amounts of time locating dataset mentions in scholarly papers. they perform this work both to locate datasets for secondary analysis projects and also to ensure a paper's conclusions are well-supported by the data. to ease this time-intensive and difficult task, we designed data gatherer to automatically find and parse dataset mentions in articles. as new methodologies in the sciences increasingly rely on access to large amounts of open data this tool can have a notable impact on the way that researchers, data curators, and librarians find, review, and aggregate data to meet the promise of these new methods.