Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction (2023)

Abstract: The Zero-Shot Learning (ZSL) task pertains to the identification of entities or relations in texts that were not seen during training. ZSL has emerged as a critical research area due to the scarcity of labeled data in specific domains, and its applications have grown significantly in recent years. With the advent of large pretrained language models, several novel methods have been proposed, resulting in substantial improvements in ZSL performance. There is a growing demand, both in the research community and industry, for a comprehensive ZSL framework that facilitates the development and accessibility of the latest methods and pretrained models. In this study, we propose a novel ZSL framework called Zshot that aims to address the aforementioned challenges. Our primary objective is to provide a platform that allows researchers to compare different state-of-the-art ZSL methods with standard benchmark datasets. Additionally, we have designed our framework to support the industry with readily available APIs for production under the standard SpaCy NLP pipeline. Our API is extendible and evaluable, moreover, we include numerous enhancements such as boosting the accuracy with pipeline ensembling and visualization utilities available as a SpaCy extension.

## introduction zero-shot learning (zsl) is a machine learning field focused on the study of models able to classify objects or perform tasks that they have not experienced during training. this is achieved by leveraging additional information about the output classes, such as their attributes or descriptions. zsl has a wide range of potential applications, since it allows a model to generalize and adapt to new situations without requiring retraining or large amounts of labeled data. this can be particularly useful in real world applications where new classes or categories may be constantly emerging and it would be infeasible to retrain the model every time. zsl can also be used to classify and predict rare or minority classes that may not have a significant amount of labeled data available for training. if we consider named entity recognition (ner) -including classification and linking (nel) -and relation extraction (re) problems, recent zsl methods @xcite ; @xcite ; @xcite leverage textual descriptions of entities or relations as additional information to perform their tasks. this additional input allows models to recognize previously unseen entities (or relations) in text, based on the provided descriptions. @xcite and @xcite provide examples of the effectiveness of descriptions in the zero-shot ner task. the same mechanism can also be applied to the re task @xcite by providing descriptions of the relation between entity pairs. figure 1 shows the input and output of a zero-shot ner and classification model such as smxm @xcite . leveraging entity descriptions, the model is able to disambiguate between mentions of the term "apple" -which could indicate a mention of the homonymous company, or the fruit -despite having been trained only on ontonotes @xcite classes. by projecting each token in the latent space of a pretrained language model (lm), the semantic distance between the label/description of a class and each token in a sentence can be used as a method assign tokens to a certain entity. there are several variations to zsl approaches, both for ner and re. for example, de @xcite show how it is possible to frame the zeroshot ner task as an end-to-end autoregressive generation problem that exploits the probability distribution learned by the lm, while @xcite uses a combination of a bi-encoder and a cross-encoder for entity linking. in addition to the different configurations of ner and re models -i.e. model architectures, training strategies and datasets -these components often depend on other nlp tools to process texts and on each other, e.g. re models often are not end-to-end and require entity spans in a text as part of their input. ## motivations and contributions the hidden complexity in running ner or re experiments in the zsl context and the lack of a shared framework and api to describe the inputs and outputs of such models hampers the reproducibility of many approaches, as well as the ability of the research community to evaluate new models against strong baselines under the same experimental conditions. more specifically, we identified the following major shortcomings for what concerns the ability to evaluate existing ner and re approaches under the same experimental conditions. different assumptions about the task. some ner and entity linking methods assume that the mentions to link to are given, while others are endto-end. similar considerations apply also to re, where some approaches tackle the task as an endto-end problem and others as a classification one. furthermore, some approaches may be limited in terms of performance or scalability and might therefore provide an evaluation only on a restricted subset of a dataset or on a simplified version of their task. for example, entity linking models are often unable to link entities to a dataset the size of wikipedia or wikidata in the same way as many re approaches cannot scale to hundreds of relation classes. these differences in the input-output configuration of ner and re models impact the ability of researchers to evaluate new approaches against existing ones, either for the lack of clarity when describing their evaluation strategy or for the investment required to make the appropriate changes to the models configurations, in order to make them comparable under the same experimental conditions. lack of standard datasets. as the performance of zsl approaches increases, better and more specific datasets are developed to help researchers evaluate increasingly challenging problems. the release of the fewrel and fewrel 2.0 datasets is an example of this refinement process @xcite . in the first version of fewrel, a re model was expected to assign to each entity pair in a sentence a relation from a list. in other words, there was always a correct class for the model to pick to describe the relation between an entity pair. however, the most frequently occurring scenario in the real world is when two entities are not related or when the correct relation between the entities is missing among the options given by the model. this prompted the release of an updated version of the dataset, which includes the no relation class as an option for re models to choose from. similar considerations can be made for the evaluation of ner models and domain-specific ones. another aspect which is often overlooked in many research papers is the lack of a shared implementation of a training, validation and test split for a dataset. this is especially important in the zsl setting where entities or relation classes should not overlap between training/validation and test dataset splits. often, zero-shot datasets are obtained from a random split of existing datasets, which were not originally designed to be used for the evaluation of zero-shot approaches and contain overlapping output classes in the respective training/validation and test set. in the process of transforming these datasets the exact split of classes is often not reported and this hampers the reproducibility of the evaluation results presented in a research work. ## design and implementation figure 2 shows a high-level overview of the zshot pipeline composed by three main modules i.e., mention detection, entity linking and relation extraction. the pipeline accepts a configuration object that allows to select the output classes of mentions, entities and relationships of interest and the models to use to perform each task (see listing 1). then, each of the library modules adds annotations that can be used at a later stage, extending a spacy nlp pipeline. components like the entity linker and the relation extractor can also be end-to-end, in which case the pipeline automatically skips the unnecessary previous steps. zshot also automatically manages batching, parallelization and the device on which to perform the computation. all options are configurable and modules are customizable. listing 1 contains a sample pipeline configuration for ner, the same used to produce the example shown in figure 1 . the configuration defines the two entity classes to be identified, providing a title and a textual description for each of them. it is also specified to use the smxm end-to-end linker (see section 3.2 for more details). in the remainder of this section, we will dive deeper into the details of each of the building blocks of zshot. ## visualization ner and re models are broadly used in multiple and diverse nlp pipelines, from knowledge graph generation and population to information extraction systems. however, for development or research, ner and re models are sometimes the final output of the process. for these reasons, an appealing visualization of the entities and relations extracted by a model is important. visualization helps users to rapidly assess what entities and relations have been extracted, and allows them make changes to improve their models. in zshot, we extend displacy, a spacy tool for visualization. when using displacy for ner with custom entities, the visualization shows the entities, but all of them have the same color, so it's not easy for the user to see the entities detected. we expanded the capabilities of the library to support distinct colors for different entities (see figure 3 ). at the time of writing, we could not find a visualization tool for re in displacy, foot_1 as re is not supported in spacy. in zshot, we extend displacy to support the visualization of edges between entity pairs in a sentence as shown in figure 4 . ## evaluation evaluation is key to improve the performance of a system. to assess the performance of ner and re models, zshot provides an evaluation module. this module includes an interface to different datasets i.e., ontonoteszs (pradhan et al., 2013), med-mentionszs (mohan and li, 2019) for ner and fewrel @xcite for re. all the datasets included have been preprocessed to assure that they can be used in the zsl setting i.e., the entities/relations in the training or validation splits of these datasets are not overlapping with each other and are not included in the respective test sets to ensure a correct evaluation. these datasets are managed using the huggingface datasets library, foot_2 which makes it easier to add new datasets using the huggingface hub foot_3 . in zshot, we use the evaluate package foot_4 , implementing evaluators for the mentions extraction, entity linking and re tasks. on top of the evaluators, we added a function that receives a spacy nlp model with a zshot configuration and the splits of the predefined datasets, and it evaluates the model on all datasets configurations. this function returns the evaluation results both as a table with the results as a string object or as a python dictionary. as a default option, we use the seqeval package @xcite to compute the evaluation metrics, including accuracy, precision, recall, f1-score macro and f1-score micro. however, using the evaluate package, it is also easy to define and use custom metrics extending the same package. ## conclusion and future work in this paper, we described zshot, a framework for zero-shot ner and re in nlp. zsl is a growing area of research in nlp. the increasing number of research works published every year and the difficulties associated to the lack of standardization metric ontonotes-test overall_precision_macro 20.96% overall_recall_macro 48.15% overall_f1_macro 29.12% our work aims at standardizing experimental zero-shot pipelines to improve the reproducibility of existing approaches and facilitate the comparison with new ones. we defined a customizable interface based on the popular spacy library. this allows developers and researchers already familiar with this popular python library to quickly try out zero-shot nlp approaches, while more experienced users can extend it to include new models. we also provide an evaluation package to aid the evaluation and comparison of ner and re models on different datasets through standard evaluation metrics, which can be further customized and expanded by users. finally, we extended the displacy library to support visualizations of recognized entities and the relations between them as edges between entity spans. we open source all our code, models, metrics and datasets. in the future, we plan to increase the number of supported models and datasets for ner and re. we also aim to increase the efficiency of these models so that they could be employed by nlp practitioners in real world applications.