Description Boosting for Zero-Shot Entity and Relation Classification (2024)

Abstract: Zero-shot entity and relation classification models leverage available external information of unseen classes – e.g., textual descriptions – to annotate input text data. Thanks to the minimum data requirement, Zero-Shot Learning (ZSL) methods have high value in practice, especially in applications where labeled data is scarce. Even though recent research in ZSL has demonstrated significant results, our analysis reveals that those methods are sensitive to provided textual descriptions of entities (or relations). Even a minor modification of descriptions can lead to a change in the decision boundary between entity (or relation) classes. In this paper, we formally define the problem of identifying effective descriptions for zero shot inference. We propose a strategy for generating variations of an initial description, a heuristic for ranking them and an ensemble method capable of boosting the predictions of zero-shot models through description enhancement. Empirical results on four different entity and relation classification datasets show that our proposed method outperform existing approaches and achieve new SOTA results on these datasets under the ZSL settings. The source code of the proposed solutions and the evaluation framework are open-sourced.

## introduction named entity recognition (ner) and relation extraction (re) allow for the extraction and categorization of structured data from unstructured text, which in turn enables not only more accurate entity recognition and relationship extraction, but also getting data from several unstructured sources, helping to build knowledge graphs and the semantic web. however, these methods usually rely on labeled data (usually human-annotated data) for a good performance, usually requiring domain experts for data acquisition and labeling, which may incur in high costs. thus, it is not surprisingly that there is often a lack of labeled data for new domains, limiting the performance of these methods. zero-shot learning (zsl) is a classification task in machine learning where -at inference time -samples are classified into one of several classes which were not observed during training. in zsl, the sets of training and test entity classes are disjoint. therefore, the strategy employed by zero-shot models is to rely on prior general knowledge that could be transferred to unseen instances at inference time. having a classifier that can generalize to new unseen classes is important for a variety of practical reasons. first, zsl methods can be used to learn models that are more robust to labeled data shortages and distributional shifts. moreover, they can be used to extend the reach of models to new domains. zsl approaches in the natural language processing (nlp) domain have seen significant improvements in recent years thanks to the availability of large pre-trained language models (lms). for example, it has been shown that models such as gpt-3 @xcite , opt @xcite and flan (dbl) achieve strong performances on many nlp tasks, including translation, question-answering, and cloze tests without any gradient updates or fine-tuning. for entity recognition -including classification and linking -and relation classification problems, recent zsl methods @xcite @xcite rely on textual descriptions of entities or relations. descriptions provide the required information about the semantics of entities (or relations), which help the models to identify entity mentions in texts without observing them during training. works such as @xcite and @xcite show how effective it is to use textual descriptions to perform entity recognition tasks in the zero-shot context. however, the quality of the descriptions has an impact on how effective the transfer of knowledge from observed to unseen entities @xcite . the same mechanism can also be applied in other contexts, such as relation classification @xcite . from now on, we refer to entity as both named entities and named relations. an example of named entity classification with zsl is demonstrated in figure 1 . at inference time, a zero-shot model is given short textual descriptions of new entity classes such as company or fruits, it then identifies and annotates mentions of those entity classes in an input sentence. although state-of-the-art zsl methods such as smxm @xcite have demonstrated significant results in recent research works, this toy example shows how the quality of the provided descriptions influences the accuracy of these models. for example, in figure 1 even with a small modification of the company entity class description, the smxm model changes its entity prediction. in practice, the sensitivity to entity descriptions is problematic because, for non-expert users, it is not a trivial task to choose a proper description for black-box zero-shot models, in particular in an unfamiliar domain. in this paper, we study different methods for improving the descriptions in an unsupervised way. specifically, we propose udebo (for unsupervised description boosting), the first unsupervised method capable of automatically modifying/generating descriptions to improve entity predictions in the zero-shot settings. we present several strategies to alter descriptions, such as using a generative model, paraphrasing, and summarization combined with description ranking/ensemble methods to reduce model uncertainty and increase overall performance. we empirically evaluate the performance of udebo on 4 existing standard zero-shot datasets, spanning two tasks: (i) named entity classification and (ii) relation classification. we organize the paper as follows. in section 2 we formally define the problem we aim to solve in this paper, i.e. how to enhance entity or relation descriptions to improve the performance of zero-shot models. in section 3 we describe the proposed approaches for description boosting while in section 4 we describe our experimental setup and results. we provide a literature review, a discussion about large language models (llms) and draw the conclusions of our work in sections 5, 6 and 7, respectively. ## problem definition given a set of entity classes e of interest with their textual descriptions d and a corpus of sentences s to annotate as input, we can define the problem of description enhancement as follows: problem 1 (description enhancement) denote ϕ(d, s) as the function estimating the accuracy of zsl models when using a given entity description d for annotating an input text corpus s. our goal is to generate a set of descriptions d * such that: in section 3.1 we describe different strategies to generate new entity descriptions d ′ for the input set e, intending to improve the accuracy of the predictions by the zsl models over that corpus. if the labeled data is known, it is possible to select the best descriptions via a brute force search across different description reformulations by measuring the accuracy as a function of d and s. however, given the absence of labeled data in the zero-shot context, an unsupervised approach is needed for ranking the descriptions d that yield the highest classification accuracy. in section 3.2 and section 3.3, we will discuss methods for ranking or combining predictions from different description variations to achieve better results. ## methods the udebo approach comprises 2 steps. first, the descriptions are generated or improved. finally, the descriptions are ranked in order to select the best ones. as an optional step, we analyze the ensembling of descriptions for boosting performance. ## experiments and results this section discusses experimental settings, baseline methods, and empirical results for both entity and relation classification tasks. ## discussion about large language models in the era of large language models, all kinds of problems are being solved with llms, that achieve outstanding results in different tasks. however, llms also raise some concerns, being one of the most important, the green footprint of these models. serving a single 175 billion llm requires at least 350 gb gpu memory using specialized infrastructure, @xcite . this makes it unfeasible for a lot of users to use llms, and even if it's possible to use, there is a lot of concern about using them, specially for tasks that could be solved with smaller models. with udebo we try to push the research in a direction that improves the performance of small lms to achieve results comparable to llms. however, to compare the performance of udebo against llms, we select 3 open-source llms available to the community and evaluate them. the results, discussion, and settings can be found in appendix d. ## conclusion and future work in this paper, we formally defined the problem of selecting descriptions to make predictions about unseen classes in the zsl context. to the best of our knowledge, this is the first time for entity/relation zsl problems in which the impact of description variations on prediction performance is studied, and different methods for automatic creation of descriptions are considered. we empirically evaluated the sensitivity of two zsl methods to description changes, and proposed 4 different strategies to enhance them using the implicit knowledge of pretrained language models. we also studied in detail the efficacy of the proposed entropy-based heuristic to rank different description formulations, analyzing its correlation with the performance (in terms of macro f1 score) of the model. we observed a negative correlation between the proposed heuristic and macro f1 score on two out of four of the considered datasets (ontonoteszs and fewrel). the same assumption however was not valid for the other datasets (medmentionszs and wikizs), thus motivating the need to develop more effective heuristics in the future. finally, we described the udebo method, which combines the predictions obtained by the same model using different automatically generated variants of entity and relation descriptions. our experimental results, on 4 different datasets, spanning across two different nlp tasks (entity classification and relation classification) showed how udebo outperforms the baselines by a significant margin and achieves new state-of-the-art results on these benchmarks under the zero-shot setting. existing ensemble methods focus on ensembling different models trained on different data or models with different structures. our work is orthogonal to these approaches, we proposed methods that consider entity/relation description variations as the hyper-parameters that need to vary. most importantly, the description variations are not provided by the users but were generated from the initially provided descriptions. therefore, this is a new way of creating ensembles, at least in the context of zero-shot entity/relation extraction this is the first time a method for descriptions generation to diversify the pipelines and make an ensemble that improve the quality of the results is proposed. limitations while our proposed method of boosting for zeroshot entity and relation classification shows promising results, there are several limitations that need to be acknowledged. firstly, ensembling methods can be computationally intensive, which can limit their applicability to large-scale datasets. our current implementation combining multiple model predictions, by varying the descriptions, which requires a significant amount of computational resources. therefore, future research should explore alternative methods for ensembling that are more computationally efficient. secondly, while we used entropy as a metric to identify helpful descriptions, it may not always be the most effective metric. entropy measures the uncertainty or randomness of a distribution, but it may not necessarily capture the semantic relevance of a description. therefore, there is a need for further research to develop better metrics or heuristics for identifying helpful descriptions. finally, we have only experimented with a limited set of generation techniques. future research should explore ways to improve the quality and coverage of the descriptions. artetxe, moya chen, shuohui chen, christopher dewan, mona diab, xian li, xi victoria lin, et al. 2022. opt: open pre-trained transformer language models. arxiv preprint arxiv:2205.01068. lianmin zheng, zhuohan li, hao zhang, yonghao zhuang, zhifeng chen, yanping huang, yida wang, yuanzhong xu, danyang zhuo, joseph e. gonzalez, and ion stoica. 2022. alpa: automating interand intra-operator parallelism for distributed deep learning. corr, abs/2201.12023.