Empowering Knowledge Discovery from Scientific Literature: A novel approach to Research Artifact Analysis (2023)

Abstract: Knowledge extraction from scientific literature is a major issue, crucial to promoting transparency, reproducibility, and innovation in the research community. In this work, we present a novel approach towards the identification, extraction and analysis of dataset and code/software mentions within scientific literature. We introduce a comprehensive dataset, synthetically generated by ChatGPT and meticulously curated, augmented, and expanded with real snippets of scientific text from full-text publications in Computer Science using a human-in-the-loop process. The dataset contains snippets highlighting mentions of the two research artifact (RA) types: dataset and code/software, along with insightful metadata including their Name, Version, License, URL as well as the intended Usage and Provenance. We also fine-tune a simple Large Language Model (LLM) using Low-Rank Adaptation (LoRA) to transform the Research Artifact Analysis (RAA) into an instruction-based Question Answering (QA) task. Ultimately, we report the improvements in performance on the test set of our dataset when compared to other base LLM models. Our method provides a significant step towards facilitating accurate, effective, and efficient extraction of datasets and software from scientific papers, contributing to the challenges of reproducibility and reusability in scientific research.

## introduction scientific research is dynamically and rapidly evolving, generating an overwhelming amount of knowledge in the form of research outputs. the vastness and the intricacies of scientific literature makes it impossible for researchers to keep up with all advancements in their respective fields, which is critical to their research. consequently, knowledge discovery from scientific literature and research artifact analysis (raa) have gained significant prominence as fields. in recent years, many new research artifact (ra) datasets have been constructed, with the goal of training models and creating benchmarks for the identification of a variety of ras, both intangible (e.g. methods, tasks) and tangible (e.g. datasets, software) @xcite . moreover, recent efforts have been directed towards identifying corresponding metadata and classifying those ras based on their functions, such as their usage and provenance @xcite . in this paper, our primary focus lies in introducing a novel ra dataset specifically designed for dataset and software extraction. our ra dataset is constructed by leveraging chatgpt foot_0 and the full-text of scientific publications in the field of computer science. it employs a human-in-the-loop manual curation process and comprises snippets of scientific text that encompass mentions of datasets and software (ra mentions). each snippet includes a trigger keyword or keyphrase indicating the ra mention, as well as a curated list of essential metadata, such as the name, version, license, url, usage, and provenance of the respective dataset or software. this ra dataset stands out from conventional ra datasets commonly used in the academic literature, due to its unique formulation. unlike conventional approaches to raa that focus on named ras through named entity recognition (ner) and entity linking, our methodology addresses a significant oversight. those approaches often neglect unnamed and undocumented resources, leading to implications for open science and reproducibility, and rendering them out of scope. our approach unifies those tasks, including both named and unnamed ra mentions. each ra mention is systematically mapped to a corresponding ra, along with its associated metadata, as defined by the context of the sentence (fig. 1 ). the primary reason for this approach is to create a dataset where all ra men-tions, whether explicitly named or not, are treated with equal importance. this allows models trained on this ra dataset to effectively identify the presence of ras even in more complex and ambiguous scenarios. additionally, we utilized the lora @xcite method to power the human-in-the-loop process and fine-tune base llm models on the constructed ra dataset. employing a model trained on this ra dataset can streamline the detection of ras within the body of scientific publications, consequently improving the reproducibility of experiments and fostering a comprehensive understanding of the research process. furthermore, this approach contributes to resource reusability by establishing a collection of crucial resources, accelerating scientific progress, mitigating repetition, and encouraging cross-disciplinary collaborations. in the subsequent sections we provide detailed insights into the methodology employed for dataset construction (secs. 3.1, 3.2, 3.3, 4.1). we also discuss the training and evaluation of our lora models (secs. 3.5, 4.2), showcasing their effectiveness in extracting ras through comprehensive benchmarking on our dataset's test set (sec. 6). by comparing them to other base llms, we demonstrate the feasibility of employing simpler llm models for successful and reliable raa. our key contributions 2 are as follows: ## related work raa has gained significant attention in recent years. this extensive research has led to the introduction of many important ra datasets related to the disciplines of computer science, biology, sociology and more. at the same time, important breakthroughs have been made in the construction of new novel machine learning models aimed to achieve this task, taking advantage of a variety of different technologies like recurrent neural networks (rnns) @xcite @xcite and bert-like architectures @xcite . the aforementioned ra datasets can be differentiated into two types with respect to their formulation and goal. the first type is characterized by abstract ras in the form of tasks, processes, or materials like semeval 2017 task 10 (augenstein et al., 2017), scierc @xcite , scirex @xcite , the methods dataset from @xcite and scires @xcite . in contrast, the second type is aimed to the identification of more strictly defined ras, with the rich context competition dataset created by the coleridge initiative foot_1 at new york university, ner dataset recognition @xcite and dmdd @xcite being characteristic examples for dataset extraction and somesci @xcite , softcite @xcite being characteristic examples for software extraction. nevertheless, a mapping between those two cases is not always possible. this divides raa into two tasks that are similar in concept but very different in practice. furthermore, some ra datasets further encompass the collection of ra metadata. two notable works that address the lack of such ra datasets are softcite and somesci. our dataset differs significantly from common standards in ra datasets through a holistic approach that prioritizes both named and unnamed ra mentions. we annotate all ra mentions within a snippet, effectively creating a connection between those sharing the same name. that approach, similar to those employed in the construction of the softcite and somesci datasets, allows us to utilize the information from all ra mentions for a specific ra, providing a more comprehensive view. considering the array of definitions for ra mention "validity" across various ra datasets, it is essential for us to demonstrate the robustness and applicability of our annotation schema. therefore, we compared our ra dataset with five highly regarded datasets of ra mentions, in order to highlight its novelty and differences, and to explore how our broader scope could benefit future applications. for example, we observed that many of the existing ra datasets lack annotations of unnamed ra mentions in a given snippet or even aim only for the identification of named datasets or software. the ra datasets we used for comparison were softcite and somesci for software mentions, and the rich context competition dataset, ner dataset recognition and dmdd for dataset mentions. the exploitation of those ra datasets has been largely demonstrated through the utilization of rnn-and bert-based models handling ner tasks embodied by those datasets. in contrast, the non-ner configuration of our ra dataset fosters the training and deployment of alternative model types. in our work, we use llms, fine-tuned with the lora method, and tackle the raa task as an instruction-based qa task. this approach serves a dual purpose: it enables us to evaluate those models' performance when deployed on such tasks, while simultaneously assessing the quality of our newly introduced ra dataset. ## synthetic dataset in this section we describe the construction process of the synthetic dataset (subsecs. 3.1, 3.2) and the conversion of the ra mentions to question-answer (qa) pairs (subsec. 3.3). next, we detail the splitting of the dataset into training, development, and testing sets (subsec. 3.4), and conclude with the training of the lora model (subsec. 3.5). ## hybrid dataset this section explores the creation of a hybrid dataset, focusing on the integration of ra mentions from synthetic and real snippets from scientific publications (subsec. 4.1), and fine-tuning a lora model on the hybrid dataset to enhance generalizability and performance for the raa task (subsec. 4.2). ## dataset analysis & comparisons the previously introduced synthetic and hybrid datasets consist of 35475 and 45136 qa pairs respectively. each qa pair revolves around a question pertaining to a ra mention, as delineated in tab. 9 (app. c). although large in quantity, they encompass only 5446 and 7112 ra mentions respectively, distinguishing them from large-scale collections like dmdd with 449798 dataset mentions. however, our aim deviates from such vast ra datasets, prioritizing quality over sheer quantity due to the different annotation methodology. a key characteristic of our dataset is that it is comprised of qa pairs that have been generated from a smaller pool of snippets. those snippets, containing references to ras, have undergone various rounds of annotation and paraphrasing. this process produces a multitude of differently phrased snippets, each providing unique ra mentions of the same ra while preserving the original contextual information. this depth of ra mentions is unique to our ra dataset and sets it apart from others. to further illustrate where our dataset sits within the existing research landscape, we offer a comparison of our dataset's key statistics with those from other established ra datasets in tab. 2. our manual annotation process has been executed rigorously to ensure quality enhancements across several aspects: the emphasis on manual annotation in our ra dataset is a result of our efforts to address the issues we have encountered in other ra datasets. those issues include the following: finally, it is important to acknowledge that our definition of a ra mention within the context of a snippet aligns significantly with the construct employed by the somesci dataset. the somesci dataset, which is dedicated exclusively to software mentions, classifies those mentions into four categories: application, plugin, operating system, and programming environment, while also annotating an extensive array of metadata. in our case, we have adopted this formalism to encompass ra mentions, without delving further into the corresponding subtypes. the metadata we have annotated includes the url, version, license, provenance, and usage of each ra mention. ## experimental results on test set we conducted a series of tests using the topperforming versions of the aforementioned lora fine-tuned models as well as the original flan-t5 base and xl models. those were employed to evaluate the quality of information included within our synthetic and hybrid datasets. 7 specifically, we computed the respective scores of those four models on the test sets of both ra datasets, aiming to discern the advantages of the fine-tuning process and to perform an ablation study to investigate the effect of fine-tuning a model using synthetic data versus synthetic data expanded by real data. to ensure a balanced comparison between the original and fine-tuned models, we modified the prompts utilized by the original models (tab. 10, app. c). those adjusted prompts refrain from presupposing any specific training of the models to 7 due to its substantial size, the flan-t5 xxl model was not incorporated in this comparison. answer in a particular manner, and consequently, provide more comprehensive instructions. the f1 score measures successful identification of a valid ra mention or presence of specific metadata (name, license, version, url). for usage and provenance metadata, it denotes successful identification of the ra's use or creation by authors. the exact match (em) score, applicable for name, license, version, and url metadata, determines exact lowercase match of the metadata text from a provided snippet, provided the model correctly identifies the presence of that metadata. the lenient match (lm) score checks if the model's answer in lowercase is within the gold truth, or vice versa. the results achieved by the four models on both ra datasets are outlined in tabs. 3 and 4. moreover, detailed evaluation results of named versus unnamed ra mentions can be found in app. d. the original flan-t5 base and xl models achieved a high level of success in discerning the validity of ra mentions, as well as the associated metadata such as the license, version, and url. as anticipated, superior performance was observed on the less complicated synthetic dataset. furthermore, the xl model surpassed the base model, particularly in identifying the name, usage, and provenance. those outcomes endorse the high semantic quality of both our ra datasets and showcase the efficacy of llms in the raa task. the significance of the lora fine-tuning procedure is evident in the scores presented in tabs. 3 and 4. the fine-tuned models remarkably outperformed their base counterparts. in addition, the lora hybrid model (lora-hy) generally outperforms the lora-sy model when evaluated on both the synthetic and hybrid datasets, indicating that the inclusion of additional real-world ra mention instances improves those models. similar findings were observed in the model tests on the gars dataset, as shown in tab. 5. notably, in the gars ## qualitative analysis in this section, we delve into an analysis of some representative examples of our models' predictions, aiming to demonstrate the impact of the lora finetuning process on the model's understanding of this specific task. we initially explore two typical instances of ra mentions, the first for a dataset and the second for a software (refer to figs. 5 and 6 , app. e), to highlight the improvements brought about by the fine-tuning of the flat-t5 model. considering the dataset mention illustrated in fig. 5 (app. e), it is evident that the correct answer is an unnamed dataset represented by "n/a". both of our lora fine-tuned models yielded the correct result, in contrast to the base models. more specifically, the flan-t5 base model erroneously returned "100,000 reviews", while the flan-t5 xl model inaccurately produced the general term "costumer reviews" as a dataset name. the same error is observed in the software mention depicted in fig. 6 (app. e), where both base models failed to give correct answers. the distinction in performance can be traced back to our fine-tuning process, as it allowed our models to understand that not every name-like term in the snippet necessarily represents a specific dataset. an examination of different scenarios reveals that the lora-hy model exhibits superior performance to the lora-sy model in more complex cases, like the provenance and usage question instances in figs. 7 and 8 ## discussion & conclusions in this work, we have made significant steps towards knowledge discovery from scholarly literature and raa by advancing the identification and extraction of dataset and software mentions within scientific literature, thereby addressing pressing challenges in reproducibility and reusability of ras. more specifically, by leveraging the capabilities of chatgpt in conjunction with meticulous human curation, we streamlined the extraction of dataset and software mentions. this innovative approach made it possible to transform raa from a ner task to an instruction-based qa task. furthermore, we investigated how llms could be effectively employed for this task and how the lora fine-tuning method can enhance such models when trained on ra datasets. through comprehensive examples and analysis, we demonstrated the efficacy of both the ra datasets and their associated models. our results not only confirm the feasibility of this specific approach to the raa task but also indicate its potential as a powerful tool in future applications. in future work, our ra datasets can be further refined and expanded by including more representative snippets drawn from a broader and more diverse assortment of scientific publications. this will facilitate the creation of new and more generalized ra datasets, helping to mitigate potential biases and incorporate knowledge from various scientific disciplines.