Annotating Research Infrastructure in Scientific Papers: An NLP -driven Approach (2023)

Abstract: In this work, we present a natural language processing (NLP) pipeline for the identification, extraction and linking of Research Infrastructure (RI) used in scientific publications. Links between scientific equipment and publications where the equipment was used can support multiple use cases, such as evaluating the impact of RI investment, and supporting Open Science and research reproducibility. These links can also be used to establish a profile of the RI portfolio of each institution and associate each equipment with scientific output. The system we are describing here is already in production, and has been used to address real business use cases, some of which we discuss in this paper. The computational pipeline at the heart of the system comprises both supervised and unsupervised modules to detect the usage of research equipment by processing the full text of the articles. Additionally, we have created a knowledge graph of RI, which is utilized to annotate the articles with metadata. Finally, examples of the business value of the insights made possible by this NLP pipeline are illustrated.

## introduction according to the definition adopted by the european commission @xcite , research infrastructure (ri) refers to "facilities, resources and related services that are used by the scientific community to conduct top-level research in their respective fields and foster innovation". a similar concept, which is more commonly used in the united states, is "research core" @xcite . foot_0 ri plays a crucial role in conducting high-quality research, with significant financial resources invested every year; for example, european countries have invested over 10 billion eur every year in the period 2014 -2020 @xcite , while uk research and innovation (ukri), the main public research funding agency in the united kingdom, has announced in its corporate plan for the years 2022 -2025 to increase the ri investments by at least £200 million every year, to reach over £1.1 billion in 2024 to 2025 (ukri, 2022). it is, therefore, extremely important for all stakeholders in the research landscape to assess the impact of such investments. various frameworks for impact evaluation have been proposed in the past @xcite and they all include scientific outputs, particularly publications in peer-reviewed journals, as an important facet of impact. there are several challenges in tracking research outputs enabled by ri, such as the lack of a standard approach to recognize contributions of facility managers and staff scientists @xcite , or the fact that sometimes it is not even considered appropriate to include them as co-authors @xcite . another important issue is that the contribution of ri to the research project is mostly found in the full text of publications, usually in sections named "materials and methods", "experimental setup", or similar. this means that abstract and indexing databases such as pubmed, scopus or web of science, which don't index the full text of records, are of limited help in this scenario. other approaches, such as assigning persistent identifiers to scientific instruments and reference them in the manuscript @xcite , while in principle effective for new publications, require widespread adoption among publishers, as well as time and effort to create a database of equipment records. for all these reasons, the identification of links between publications and ri remains largely a manual and inefficient task @xcite . in this work we present a solution to the problem of identifying and linking ri in the text of scientific publications, introducing a pipeline designed to connect scientific publications with ri utilized in the respective works. to the best of our knowledge, this is the first comprehensive solution to tackle this challenge end-end, and to be brought in a production environment. the pipeline of the solution utilizes state-of-the-art few shot learning algorithms to train our machine learning models using a limited labeled dataset. by employing these cutting-edge techniques, we were able to achieve very insightful results for research stakeholders, despite the constraints of a small training set. the remaining of the paper is organized as follows; the key user problems that this solution is addressing are discussed in section 2. section 3 provides an overview of the solution's architecture, followed by a detailed description of each module in section 4. section 5 presents evaluation results and analysis. section 6 highlights concrete business impact of the proposed system and in section 7 we conclude and provide pointers to future work. ## description of user problems working with academic institutions and funding agencies with a specific interest in ri, we have identified several use cases where being able to link ri (as an input to research) with research outputs (in particular scientific publications) can provide valuable insights to a broad range of stakeholders such as policy makers, academic leaders, researchers and technical staff, as well as the general public. here are some representative use cases: (1) supporting decision making processes and investment planning about ri with a quantitative, evidencebased approach that complements qualitative insights based on expert opinion; (2) showcasing ri to attract top talents. institutions can promote themselves as a destination for the best researchers in various fields by showcasing state-of-the-art instruments available at their research facilities; (3) promoting collaboration at local, national and regional level, as well as across disciplines and sectors (for example academic-corporate collaborations); (4) supporting open science and big science, by promoting transparency and accountability, particularly for large ris; (5) improving the reproducibility of research, by providing useful information about the equipment used in research works. ## overview of the proposed solution this section provides an overview of the proposed solution, including the input requirements and desired output of the pipeline. we also discuss the dataset collected for training the nlp components. an overview of the solution's pipeline is illustrated in figure 1 . the first part of the pipeline (unsupervised provess) is composed of fully automated modules, while the latter (supervised process) requires supervision by a subject matter expert (sme) to guarantee high-quality results. ## modules of proposed solution this section provides the details of each of the components in the pipeline illustrated as blue boxes in figure 1 . ## business impact we have completed several projects with institutions active in the science, technology, engineering and mathematics (stem) domains. we focused primarily on the first use case of those listed in section 2: we helped institutions evaluate the impact of their ri investments by providing quantitative evidence based on a scientometric analysis of publications enabled by institutional ri. those insights include: (1) the contribution of ri to the scientific output in a certain topic; (2) the scientific impact of publications enabled by ri, compared to the institutional average; (3) the scientific impact of a specific facility or lab inside the institution; ## conclusions and future work in this paper we have presented a novel system that can detect, extract and link the research infrastructure (ri) used and mentioned in scientific publications. the system comprises several advanced nlp components that can annotate and classify sentences, as well as detect ri entities and link them to a knowledge graph (kg) that has been created for the purpose of this business application. we have discussed the performance of the key individual components of our system, the use cases that the proposed solution can address, and we have demonstrated the insights and knowledge that any research facility or institution can obtain around the impact and return of investment of their equipment in research conducted by its personnel. our future work will focus on expanding and releasing the kg in public, as well as optimizing the paralelization and scaling of the existing pipeline.