Metadata Enhancement Using Large Language Models (2024)

Abstract: In the natural sciences, a common form of scholarly document is a physical sample record, which provides categorical and textual metadata for specimens collected and analyzed for scientific research. Physical sample archives like museums and repositories publish these records in data repositories to support reproducible science and enable the discovery of physical samples. However, the success of resource discovery in such interfaces depends on the completeness of the sample records. We investigate approaches for automatically completing the scientific metadata fields of sample records. We apply large language models in zero and few-shot settings and incorporate the hierarchical structure of the taxonomy. We show that a combination of record summarization, bottom-up taxonomy traversal, and few-shot prompting yield F1 as high as 0.928 on metadata completion in the Earth science domain.

## introduction not all scholarly documents are formal scientific articles. in the natural sciences (e.g. earth science, biodiversity science, archaeology), a common form of scholarly document is a physical sample record. also called catalog records or specimen records, sample records are documents written by scientists that describe samples (for instance, fossils, soil samples, sediment cores, etc.) collected and analyzed for scientific research. since the 1980s, millions of dollars of grant funds have been dedicated to projects digitizing and sharing physical sample records through online data repositories @xcite . the specific contents of records varies by domain, but they typically contain metadata describing the type, material, and provenance of samples. sample records are needed to find and reuse physical samples, and, when aggregated, can be treated as a scientific dataset in and figure 1 : an illustration of our automatic metadata assignment workflow utilizing text classification with llms. the example record material metadata field is predicted as result of tree traversal done on the taxonomy in a bottom-up manner. sample record metadata may be incomplete for a variety of reasons, including data entry issues or schema mismatches when sample records are aggregated across different repositories. for example, among the sample records that describe the resources registered in the system for earth sample registration (sesar2) data repository foot_0 (fig 2 ), 78% of the records, totaling 3,565,478 records, exhibited missing values in the material metadata field. automatic metadata assignment has consequently emerged as a critical task in the processing of sample records and other metadata aggregations. for instance, some data repositories automatically assign geocoordinates and higher geographic classifications based on text-based locality descriptions using rules-based workflows @xcite . other approaches to inferring missing categorical fields often represent metadata fields as vectors and utilized classifiers to fill in missing values from a controlled vocabulary @xcite . often sample records contain many text fields that can be used to predict or infer missing values; thus, we approach automatic metadata assignment as a taxonomy-driven text classification problem. we propose a solution for metadata assignment based on large language models (llms) (fig. 1 ). our contributions are as follows: our evaluation focuses on sample records from the earth science domain, though our approaches are general enough to be applied to many other domains. our findings suggest that llms could provide a viable solution for digital libraries or aggregations facing missing metadata issues. metadata enhancement in digital libraries a common challenge in digital libraries or metadata aggregations is the "missing data" problem @xcite . to tackle this issue, various automatic metadata assignment approaches including machine learning techniques have been employed. @xcite ; @xcite used classification methods to automatically assign metadata values. unlike these supervised learning methods, our approach employs llms with zero or few-shot samples to offer a more domain-agnostic solution. llms for text classification with the advancement of larger-scale pre-trained language models trained on vast unlabeled corpora, approaches that utilize language models with instructions (zeroshot) and task-related examples directly in the prompt (few-shot) have been used. prompting methods @xcite and domain-specific llms @xcite have been investigated for text classification. our work explores text classification with llms to assign metadata values. ## dataset we are working with physical sample records from the internet of samples (isamples) project, which aggregates physical sample metadata records from data repositories within the domains of earth science, bioscience, and archaeology @xcite . to aggregate these records, isamples developed a common metadata schema and controlled vocabularies for the material and object type fields of each sample record @xcite , to which each repository's records are crosswalked. this makes it possible to easily search across a diverse range of records within the isamples interface. the material and object type fields are used as two primary facets in the search interface. here, we focus on records from sesar2, which describe specimens from the earth science domain, encompassing rocks, fossils, fluids, and other materials registered. a substantial portion of these records contains missing values in the material field, which negatively impacts the discoverability within both isamples and sesar2. however, the records contain other metadata which can be used to infer the material type. to frame this as a text classification dataset, for each sample record, we construct the output from the metadata field to be predicted, and construct the input from all other available metadata fields. in section 4 we explore various ways of concatenating the many metadata fields into a single text for input, but in all cases, we include the following text in the text classification prompt: you are a scientist . your task is to analyze the description of a material sample and determine the kind of material that constitutes it @xmath0: { taxonomy } you will only respond with the material type . do not include the word material type . do not provide explanations or notes . ### <<< description : { text } material type : >>> where the taxonomy is a list of possible labels in the taxonomy to predict and the text is the input constructed from metadata fields. for the output, we use a multi-label formulation that includes all appropriate labels within the taxonomy. for example, if a sample should be labeled as basalt, then it should also be labeled as rock since basalt is a type of rock in the isamples taxonomy (figure 3 ). for evaluation on this multi-label dataset, we use the macro f1 score. for our training and testing purposes, we utilized the 988,426 (22%) of records that contained the material field value. due to the dataset's highly imbalanced nature, we randomly selected and discarded sample records such that the maximum occurrence of a label could be 10,000, while ensuring that each label appeared at least 10 times in the entire dataset. as a result, the training dataset contained 294,420 records, the development dataset contained 63,090 records, and the test dataset contained 63,091 records. for the test dataset, we focused solely on labels that appeared at least 10 times and records that was annotated up to the leaf level to estimate performance of our experiments. the number of tokens in fields of these records is listed in appendix a. ## experiments we explore three llms for our multi-label text classification formulation of the metadata assignment problem: llama-2-7b-chat-hf, mistral-7b-instruct, and mistral-7b-openorca. using these models, we explore the following research questions: to address each of these questions, we utilize the test dataset for evaluation purposes. our findings indicate that the bottom-up search approach is the most effective for integrating the label hierarchy, and thus, we apply this method when conducting experiments for the first two questions. the following sections detail the experiments conducted to answer each of these research questions. selected values input metadata field values long cylindrical cores, ship, lamont-doherty core repository at columbia university (ldcr), ... metadata field name and value description_sampletype is long cylindrical cores, descrip-tion_supplementmetadata_platformtype is ship, descrip-tion_supplementmetadata_currentarchive is lamont-doherty core repository at columbia university (ldcr), ... metadata field description and value the object type of sample is long cylindrical cores, the type of platform for the cruise is ship, the name of institution, museum, or repository where the sample is currently stored is lamont-doherty core repository at columbia university (ldcr), ... ## conclusion physical sample records, a form of scholarly document created by natural scientists, play a crucial role in ensuring the reproducibility and reusability of sample-based scientific knowledge; additionally, they are often used as scientific datasets in and of themselves. however, they are often incomplete, thereby impacting their usability and the usability of the data repositories that store them. in this research, we explore various methodologies aimed at leveraging recent developments in llms to address this issue. to transform a sample record into an input suitable for text classification by llms, we find that maximizing verbosity by using descriptions of metadata field names and their corresponding values is beneficial. additionally, we demonstrate that harnessing the llm's inherent knowledge and summarization capabilities enhances the comprehensibility of inputs for text classification tasks. our results also reveal the effectiveness of integrating taxonomy hierarchies through bottom-up tree search, which we expect to be particularly beneficial for domain-specific taxonomies with which the llm has limited exposure. furthermore, incorporating few-shot examples into the prompt leads to substantial improvements. notably, the best fewshot learning performance surpasses that of a finetuned roberta textual entailment classifier, indicating a promising approach for embedding training data with minimal computational resources, applicable across diverse domains of sample records. overall, our experimental results suggest that llms hold potential as a solution for digital libraries and aggregations grappling with metadata quality issues, particularly in domain-specific cases. our approach offers promise for addressing metadata quality challenges stemming from incomplete records across various domains.