A ffil G ood: Building reliable institution name disambiguation tools to improve scientific literature analysis (2024)

Abstract: The accurate attribution of scientific works to research organizations is hindered by the lack of openly available manually annotated data–in particular when multilingual and complex affiliation strings are considered. The AffilGood framework introduced in this paper addresses this gap. We identify three sub-tasks relevant for institution name disambiguation and make available annotated datasets and tools aimed at each of them, including i) a dataset annotated with affiliation spans in noisy automatically-extracted strings; ii) a dataset annotated with named entities for the identification of organizations and their locations; iii) seven datasets annotated with the Research Organization Registry (ROR) identifiers for the evaluation of entity-linking systems. In addition, we describe, evaluate and make available newly developed tools that use these datasets to provide solutions for each of the identified sub-tasks. Our results confirm the value of the developed resources and methods in addressing key challenges in institution name disambiguation.

## introduction the availability and access to research outcomes are gradually becoming less of an issue in an open data and open science context @xcite . this is due, in part, to the emergence and expansion of open scholarly knowledge graphs @xcite @xcite , open research information providers @xcite , and the increasing trend of governments and public agencies to release their research and innovation (r&i) policy data @xcite . however, effective metadata curation of large open databases remains a significant challenge. a persistent issue is how to uniquely identify institutions involved in research from unstructured affiliation strings in scientific publications, where organizations are mentioned in non-standard formats-even when authors are en-couraged to use official institution name signatures when publishing @xcite . @xcite the task of automatically identifying organizations in author-provided affiliation strings and linking them to unique identifiers from global registriessuch as the research organization registry (ror)2 or wikidata-is known as institution name disambiguation or affiliation normalization. linking scientific works to a regularly-updated human-curated registries of organizations is crucial for addressing organization changes over time, including institutional mergers and splits, as well as evolving naming conventions @xcite . accurate normalization of institutions is vital for research evaluation @xcite and essential for analyzing scientific production trends, particularly within an open science context (l' @xcite . furthermore, research assessment may be affected by wrong attribution of publications to institutions @xcite . to precisely attribute publications to institutions can be challenging due to the fact that organizations are frequently mentioned in diverse and unstructured manners, employing various patterns, languages,3 and abbreviations. in addition, automatically extracted affiliation strings often include noise, irrelevant information, or typographical errors. affiliations can also refer to different institutional levels, such as departments and collaborative institutions, adding complexity to the task. illustrative examples of these challenges are presented in despite several tools and methods having been proposed to tackle different subtasks, as described in 1in openalex @xcite , for example, there are more than 8 million different raw affiliation strings in publications produced in 2023, with 72% of those not present in publications from the previous five years. our contributions include: i) providing training and evaluation datasets for affiliation span identification and named entity recognition (ner) subtasks, along with seven entity-linking validation datasets with varying difficulty levels (including multiple institutions, diverse formats, and several languages), and the disambiguated institutions with their ror identifiers; ii) fine-tuning and evaluating new ner models based on roberta and xlm-roberta for identifying entities in affiliation strings (organizations and their locations); iii) using the entities predicted by our ner models as input for a two-step entity linking module that first retrieves candidate ror identifiers via an information retrieval system, i.e. elasticsearch, and then re-ranking them using a quantized generative model; iv) using our datasets to compare our modules' and the full pipeline's performance against existing methods; v) making datasets, code, and models available to the research community.4 to our knowledge, this is the first effort to provide open, manually annotated data for institution name disambiguation in scientific papers and r&i projects. ## related work various approaches have been proposed to tackle the disambiguation of institution names, includ-4our code, models, and datasets are available at https: //github.com/sirisacademic/affilgood ing knowledge-and rule-based approaches, name matching and search techniques, supervised learning, and clustering. rule-based methods @xcite require great effort to develop and maintain rules and often depend on external sources such as thesauri containing organization names and variants. although attempts have been made to automate the creation of rules @xcite it is still challenging to extend rule-based methods to a global scale and to adapt them to growing and changing data. clustering methods have been proposed in combination of rules to normalise and group name variants @xcite . however, additional manual cleaning steps are often necessary to obtain reliable results. l @xcite introduce affiliation-matcher, a tool aimed at linking affiliations to different registries of organizations, including ror and sirene,5 using the elasticsearch search engine6 to match organization names and locations in affiliations and the destination registries. this approach, however, presents some limitations when information available in affiliation strings is noisy or with significant variations with respect to the data indexed in the registries. the openalex institution parsing7 tool also links affiliations strings to ror institutions, and is the affiliation disambiguation system used in the ope-nalex database @xcite . the authors of this system model the normalization task as an extreme multi-label text classification problem: they propose to use two distilbert sequence classification models to predict ror identifiers trained on synthetic affiliation strings obtained from the openalex and based on different affiliation templates (including, for instance, organization names, aliases, acronyms, cities, regions, and countries). affro,8 the affiliation matching algorithm used in openaire @xcite , pre-processes affiliation strings in order to identify relevant segments in them and then employs cosine similarity for matching organization names with ror entries, considering different similarity thresholds for universities and non-university organizations. s2aff,9 the affiliations linker module used in semantic scholar (kinney et al., 2023) implements a three-steps method: a ner model first parses raw affiliation strings to extract main and child organizations as well as address components. the identified entities are used to retrieve a set of candidates from a ror dump, which are then re-ranked by means of a parwise feature-based model, as described in §4.3.1. the main limitation of the s2aff linker is that it has not been trained to deal with non-english affiliations. @xcite explore the use of a deep learning model to normalize organization names according to closed scholarly knowledge graphs: they classify parts of the affiliation strings as institutional data (first level and second level institutions) and non-institutional data (for address components) and then apply an institution matching and merging model that uses word embeddings and a set of manually formulated rules for data transformation and processing. the system also includes a relation extraction model to identify hierarchical institutional relationships. ## task definition and datasets in line with previous work in this area @xcite , we propose to approach affiliation normalization as a three-step pipeline that includes the following sub-tasks: (1) raw affiliation span identification ( §3.1), (2) named entity recognition ( §3.2), and (3) entity linking ( §3.3). we tackle the obstacle posed by the limited availability of annotated data for these tasks -in particular, for complex and/or multilingual cases -by compiling (creating/or curating and refining) new datasets with which to train and/or evaluate our modules individually, as well as the whole pipeline. addressing the unique identification of organizations in affiliation strings as three related subtasks has the additional advantage of allowing more flexibility, as each of the modules can be used independently in different downstream applications.10 ## methods as mentioned in §3 we divided the task of linking affiliation strings to ror identifiers in three subtasks: affiliation span identification, named-entity recognition, and entity linking. in this section we describe the implementation of tools aimed at each of these tasks. ## results and discussion in this section we report the results obtained by each of the three modules (span identification, ner, and entity linking) as well as the integration of the ner and entity linking components, and we compare them to baselines and other existing systems. we have not included the span identification task as part of the pipeline, because as our evaluation datasets contain few examples in which affiliation spans do not cover the whole raw affiliation string, we would obtain limited information to properly conduct an extrinsic evaluation of this modules' performance. ## conclusions in this work, we have introduced the affilgood institution name disambiguation framework, including two different datasets for information extraction from raw affiliation strings, and a collection of seven entity linking datasets connecting organizations mentioned in affiliation strings and/or research projects to ror identifiers. we benchmark our entity linking datasets with openly available institution name disambiguation systems. finally, we propose a flexible and multilingual multistep pipeline based on a named-entity recognition model and an entity linking module. the obtained results confirm the quality of the contributed datasets and the validity of the proposed systems to address some of the most difficult challenges in the institution name disambiguation task, including noisy and incomplete input data, affiliations in languages other than english and/or with mixed languages, and complex affiliations including diverse types of institutionscompanies, universities, hospital, research centers -in different hierarchical levels. to facilitate reproducibility and promote future research in this area we make available all the data and systems developed in the context of this work.