E - NER : Evidential Deep Learning for Trustworthy Named Entity Recognition (2023)

Abstract: Most named entity recognition (NER) systems focus on improving model performance, ignoring the need to quantify model uncertainty, which is critical to the reliability of NER systems in open environments. Evidential deep learning (EDL) has recently been proposed as a promising solution to explicitly model predictive uncertainty for classification tasks. However, directly applying EDL to NER applications faces two challenges, i.e., the problems of sparse entities and OOV/OOD entities in NER tasks. To address these challenges, we propose a trustworthy NER framework named E-NER by introducing two uncertainty-guided loss terms to the conventional EDL, along with a series of uncertainty-guided training strategies. Experiments show that E-NER can be applied to multiple NER paradigms to obtain accurate uncertainty estimation. Furthermore, compared to state-of-the-art baselines, the proposed method achieves a better OOV/OOD detection performance and better generalization ability on OOV entities.

## introduction named entity recognition (ner) aims to locate and classify entities in unstructured text, such as extracting location information "new york" from the sentence "how far is new york from me". thanks to the development of deep neural network (dnn), current ner methods have achieved remarkable performance on a wide range of benchmarks @xcite @xcite . despite this progress, current ner-related research typically focuses on improving the model performance, such as recognition accuracy and f1 scores @xcite . however, seldom works focus on investigating the model's reliability. the critical aspect of the model reliability is the uncertainty estimation of the predictive results, which can characterize the probability that the model prediction will be wrong. one natural way to construct the predictive uncertainty is based on the maximum value of the softmax output @xcite @xcite (the smaller this value, the larger the uncertainty). however, previous empirical studies show that probabilistic predictions produced by dnn models (e.g., transformer and cnn) are often inaccurate @xcite @xcite . therefore, this natural way may over/under-estimate the predictive uncertainty, hindering the model's reliability. high-quality uncertainty estimation helps to improve the model's reliability in an open environment and to find valuable samples to improve training sample efficiency, thus reducing the cost of manual labeling. on the one hand, for the reliability aspect, accurate uncertainty estimation can equip the ner model with the ability to express "i do not know" to both the out-of-domain (ood) or out-of-vocabulary (oov) samples @xcite . a desired uncertainty estimation is conceptually shown in figure 1 , wherein misclassified oov/ood entities are assigned with significantly higher uncertainty than the in-domain (id) entities. besides, the estimated uncertainty can be further absorbed into the training process to improve the model robustness against oov/ood samples. on the other hand, for the sample efficiency aspect, prior work shows that high-quality uncertainty estimation can also be used for selecting more "informative" samples and thus can reduce the number of labeled samples required for training the ner model. to attain high-quality uncertainty estimation, evidential deep learning (edl) @xcite provides a promising solution. edl is superior to existing bayesian learning-based methods @xcite @xcite in that model uncertainty can be efficiently estimated in a single forward pass that avoids inexact posterior approximation @xcite or time/storage-consuming monte carlo sampling @xcite . however, directly applying conventional edl to ner applications still faces two critical challenges: (1) sparse entities: in text corpus, entities only take a minority. for example, only 16.8% of the words in the commonly used conll2003 dataset belong to entities. the remaining non-entity types are labeled into the "others" (o) class. the imbalance between entity and non-entity words can cause over-fitting and poor performance on the entity types. (2) oov/ood entity discrimination: in the open environment, ner training/test data typically comes with oov/ood entities. however, the optimization objective of current edl methods lacks explicit modeling of such types of information. to address these two issues, we present a trustworthy ner framework named e-ner with a series of uncertainty-guided training strategies. for the issue of sparse entities, we propose to use an uncertainty-guided importance weighted (iw) loss, wherein samples with higher predictive uncertainties are assigned larger weights. this loss helps the model training to pay more attention to entities of interest (e.g., person and location). to solve the issue of unknown entities, we present an additional regularization term to penalize the case where labels are more prone to errors by assigning higher uncertainties to corresponding samples. we empirically show these two uncertainty-guided loss terms can improve both the quality of estimated confidence and the robustness against oov samples. our contributions are summarized as follows: ## preliminary this section introduces a commonly-used edl implementation based on the dirichlet-based model (dbm) @xcite . we then describe how the dbm computes the uncertainty in a closed form. ## e-ner architecture in this section, we describe the three core modules of e-ner and provide an overview of the system architecture in figure 2 . additionally, we revise the learning strategy of edl by incorporating importance weights (iw) to address the sparse entities problem and uncertainty mass optimization (unm) to model the uncertainty of mispredicted entities. ## conclusion in this work, we study the problem of trustworthy ner by leveraging evidential deep learning. to address the issues of sparse entities and oov/ood entities, we propose e-ner with two uncertaintyguided loss terms. extensive experimental results demonstrate that the proposed method can be effectively applied to various ner paradigms. the uncertainty estimation quality of e-ner is improved without harming performance. additionally, the well-qualified uncertainties contribute to detecting oov/ood, generalization, and sample selection. these results validate the superiority of e-ner on real-world problems.