MOLE : Metadata Extraction and Validation in Scientific Papers Using LLM s (2025)

Abstract: Metadata extraction is essential for cataloging and preserving datasets, enabling effective research discovery and reproducibility, especially given the current exponential growth in scientific research. While Masader (CITATION) laid the groundwork for extracting a wide range of metadata attributes from Arabic NLP datasets’ scholarly articles, it relies heavily on manual annotation. In this paper, we present MOLE, a framework that leverages Large Language Models (LLMs) to automatically extract metadata attributes from scientific papers covering datasets of languages other than Arabic. Our schema-driven methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output. Additionally, we introduce a new benchmark to evaluate the research progress on this task. Through systematic analysis of context length, few-shot learning, and web browsing integration, we demonstrate that modern LLMs show promising results in automating this task, highlighting the need for further future work improvements to ensure consistent and reliable performance.

## introduction we release the mole dataset under the cc0 license ... in vastly and rapidly growing domains such as natural language processing (nlp). these datasets vary widely in structure, size, format, purpose, and language. without robust metadata extraction, valuable datasets remain underutilized, research efforts are duplicated, and reproducibility is compromised @xcite . with hundreds of thousands of new publications annually foot_2 , automating metadata extraction is essential to maintain the scalability of the scientific ecosystem. this work attempts to approach this problem utilizing llms to extract the metadata. we define metadata as a json object that holds many attributes, such as year, license, and paper and dataset links. such attributes vary in constraints: some fixed options (license), some free form (dataset description), and some context-dependent @xcite . while existing automated approaches typically extract around 5-10 attributes @xcite , @xcite 17 svm ✗ ✗ 1 ✗ @xcite 18 rule-based ✗ ✗ 1 ✗ @xcite 23 crf ✗ 40 1 ✗ our work automatically extracts around 30 different attributes per paper, providing a substantially more comprehensive metadata profile (see figure 1 overviews our work of extracting sample attributes from a paper as a simplified example. current metadata extraction approaches typically rely on rule-based systems, supervised machine learning, or combinations thereof @xcite @xcite @xcite . while effective for structured documents, these methods struggle with the heterogeneity of scientific papers and require domain knowledge and maintenance to accommodate evolving document structures @xcite . recent advances in large language models (llms) have opened new possibilities for information extraction @xcite , with models showing promising results in extracting structured data, @xcite @xcite . a prominent advantage of modern llms is their ability to handle long contexts @xcite @xcite , allowing our methodology to process entire papers. we summarize our contributions as follows: ## methodology figure 2 illustrates our mole framework. this framework processes scientific papers in either la-tex source or pdf format, leveraging the power of llms' extraction capabilities to identify dataset metadata attributes. when processing latex, the framework directly analyzes the source; for pdf, we study extracting text manually using the available pdf tools vs. prompting an llm (with vision capabilities) to extract a structured output from the paper. then, an llm identifies and structures the metadata according to our schema, which is subsequently validated before producing the final json output. this pipeline enables an automated, efficient, and reliable extraction of a comprehensive dataset metadata from diverse document formats and across multiple languages. ## dataset we manually annotated 126 papers covering datasets in different languages for testing. we additionally annotate 6 more papers for validation. two authors participated in the annotation process foot_4 . we annotate each metadata with two values; the first one is the value of the metadata, and the second with a binary value of 1 if the attribute exists in the paper; otherwise, it is 0. the binary annotation will help us in measuring which metadata exists in the papers and which ones require browsing the web. for example, the license of a given dataset might not exist in the paper itself, but most likely it can be accessed through the link. the articles collected span six different categories: arabic (ar), english (en), french (fr), japanese (jp), russian (ru), and multilingual (multi) datasets. the annotation process included annotating a subset and calculating the agreement percentage of the two annotations. the percentage of agreement between the two authors was found to be 85.93. in ## conclusion this paper introduced a methodology for using llms to extract and validate metadata from scientific papers. through our framework, which extracts around 30 distinct metadata attributes, we offer a more robust approach to automating metadata extraction. we experiment with multiple approaches, including different input formats, fewshot examples, and browsing. we also highlight the effect of length constraints and how llms still struggle to follow strict instructions. throughout our experiments, we show recent advancements in processing long context, especially in flagship models like gemini 2.5 pro and gpt-4o that continue to achieve better results. we also release a benchmark of 126 papers manually annotated to facilitate research in metadata extraction. this work contributes not only to the advancement of metadata extraction techniques but also to the broader goal of making scientific research more transparent, accessible, and reusable.