Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing (2023)

Abstract: Ultra-fine entity typing plays a crucial role in information extraction by predicting fine-grained semantic types for entity mentions in text. However, this task poses significant challenges due to the massive number of entity types in the output space. The current state-of-the-art approaches, based on standard multi-label classifiers or cross-encoder models, suffer from poor generalization performance or inefficient inference speed. In this paper, we present CASENT, a seq2seq model designed for ultra-fine entity typing that predicts ultra-fine types with calibrated confidence scores. Our model takes an entity mention as input and employs constrained beam search to generate multiple types autoregressively. The raw sequence probabilities associated with the predicted types are then transformed into confidence scores using a novel calibration method. We conduct extensive experiments on the UFET dataset which contains over 10k types. Our method outperforms the previous state-of-the-art in terms of F1 score and calibration error, while achieving an inference speedup of over 50 times. Additionally, we demonstrate the generalization capabilities of our model by evaluating it in zero-shot and few-shot settings on five specialized domain entity typing datasets that are unseen during training. Remarkably, our model outperforms large language models with 10 times more parameters in the zero-shot setting, and when fine-tuned on 50 examples, it significantly outperforms ChatGPT on all datasets.

## introduction classifying entities mentioned in text into types, commonly known as entity typing, is a fundamental problem in information extraction. earlier research on entity typing focused on relatively small input: in addition, greer said, scientists do not know if chemically treated oil will degrade as quickly as oil that 's dispersed through wind and wave, and if it 's more toxic. box4types @xcite : lite @xcite ): casent (ours): object (0.83) petroleum (0.99) oil (0.99) object (0.99) substance (0.99) liquid (0.99) material (0.98) oil (0.68) liquid (0.59) substance (0.39) object (0.39) petroleum (0.36) fluid (0.32) figure 1: comparison of predicted labels and confidence scores for a ufet test example using box4types @xcite , lite @xcite , and our approach, casent. predictions are sorted in descending order based on confidence. box4types fails to generalize to rare and unseen types, while lite does not predict calibrated confidence scores and exhibits slow inference speed. ultra-fine entity typing can be viewed as a multilabel classification problem over an extensive label space. a standard approach to this task employs multi-label classifiers that map contextual representations of the input entity mention to scores using a linear transformation @xcite @xcite . while this approach offers superior inference speeds, it ignores the type semantics by treating all types as integer indices and thus fails to generalize to unseen types. the current state-of-the-art approach @xcite reformulated entity typing as a textual entailment task. they presented a cross-encoder model that computes an entailment score between the entity mention and a candidate type. despite its strong generalization capabilities, this approach is inefficient given the need to enumerate all 10k types in the ufet dataset. black-box large language models, such as gpt-3 and chatgpt, have demonstrated impressive zeroshot and few-shot capabilities in a wide range of generation and understanding tasks @xcite . yet, applying them to ultra-fine entity typing poses challenges due to the extensive label space and the context length limit of these models. for instance, @xcite reported that gpt-3 with few-shot prompting does not perform well on a classification task with thousands of classes. similar observations have been made in our experiments conducted on ufet. in this work, we propose casent, a calibrated seq2seq model for entity typing. casent predicts ultra-fine entity types with calibrated confidence scores using a seq2seq model (t5-large @xcite ). our approach offers several advantages compared to previous methods: (1) standard maximum likelihood training without the need for negative sampling or sophisticated loss functions (2) efficient inference through a single autoregressive decoding pass (3) calibrated confidence scores that align with the expected accuracy of the predictions (4) strong generalization performance to unseen domains and types. an illustration of our approach is provided in figure 2 . while seq2seq formulation has been successfully applied to nlp tasks such as entity linking @xcite , its application to ultra-fine entity typing remains non-trivial due to the multi-label prediction requirement. a simple adaptation would employ beam search to decode multiple types and use a probability threshold to select types. however, we show that this approach fails to achieve optimal performance as the raw conditional probabilities do not align with the true likelihood of the corresponding types. in this work, we propose to transform the raw probabilities into calibrated confidence scores that reflect the true likelihood of the decoded types. to this end, we extend platt scaling @xcite , a standard technique for calibrating binary classifiers, to the multi-label setting. to mitigate the label sparsity issue in ultrafine entity typing, we propose novel weight sharing and efficient approximation strategies. the ability to predict calibrated confidence scores not only impacts task performance but also provides a flexible means of adjusting the trade-off between precision and recall in real-world scenarios. for instance, in applications requiring high precision, predictions with lower confidence scores can be discarded. we carry out extensive experiments on the ufet dataset and show that filtering decoded types based on calibrated confidence scores leads to state-ofthe-art performance. our method surpasses the previous methods in terms of both f1 score and calibration error while achieving an inference speedup of more than 50 times compared to cross-encoder methods. furthermore, we evaluate the zero-shot and few-shot performance of our model on five specialized domains. our model outperforms flan-t5-xxl @xcite , an instruction-tuned large language model with 11 billion parameters in the zero-shot setting, and surpasses chatgpt when fine-tuned on 50 examples. ## methodology in this section, we present casent, a calibrated seq2seq model designed for ultra-fine entity typing. we start with the task description ( §3.1) followed by an overview of the casent architecture ( §3.2). while the focus of this paper is on the task of entity typing, our model can be easily adapted to other multi-label classification tasks. ## conclusion engineering decisions often involve a tradeoff between efficiency and accuracy. casent simultaneously improves upon the state-of-the-art in both dimensions while also being conceptually elegant. the heart of this innovation is a constrained beam search with a novel probability calibration method designed for seq2seq models in the multi-label classification setting. not only does this method outperform previous methods-including chatgpt and the existing fully-supervised methods-on ultrafine entity typing, but it also exhibits strong generalization capabilities to unseen domains.