Uncertainty Estimation on Sequential Labeling via Uncertainty Transmission (2024)

Abstract: Sequential labeling is a task predicting labels for each token in a sequence, such as Named Entity Recognition (NER). NER tasks aim to extract entities and predict their labels given a text, which is important in information extraction. Although previous works have shown great progress in improving NER performance, uncertainty estimation on NER (UE-NER) is still underexplored but essential. This work focuses on UE-NER, which aims to estimate uncertainty scores for the NER predictions. Previous uncertainty estimation models often overlook two unique characteristics of NER: the connection between entities (i.e., one entity embedding is learned based on the other ones) and wrong span cases in the entity extraction subtask. Therefore, we propose a Sequential Labeling Posterior Network (SLPN) to estimate uncertainty scores for the extracted entities, considering uncertainty transmitted from other tokens. Moreover, we have defined an evaluation strategy to address the specificity of wrong-span cases. Our SLPN has achieved significant improvements on three datasets, such as a 5.54-point improvement in AUPR on the MIT-Restaurant dataset. Our code is available at https://github.com/he159ok/UncSeqLabeling_SLPN .

## introduction named entity recognition (ner) is a popular task in the information extraction domain @xcite , which involves two steps, detecting entity spans and predicting the entity labels. in many information extraction scenarios, there are significant consequences for relying on inaccurate ner predictions. for example, extracting an inaccurate time can lead to erroneous policy analysis, or misclassifying a person's name for a time can result in a privacy breach. therefore, it is crucial to determine whether we can trust the ner predictions or not. as a result, our goal is to enhance uncer-tainty estimation in ner (ue-ner), which aims to quantify prediction confidence in ner tasks. the ner task differs from general classification (e.g., text classification @xcite in two key ways, making previous uncertainty estimation models suboptimal for ue-ner. first, the predicted entity labels in the ner task are directly dependent on the token embeddings, and uncertainty transmission between token embeddings is unique in ner. concretely, given an example text "barack obama was born in honolulu, hawaii," the entity label "person" applies to "barack obama." the embedding of the token "obama" is obtained by accumulating its own embedding and embeddings from other tokens in @xcite and transformer @xcite . consequently, if a token embedding has higher uncertainty, the other token embedding will have more transmitted uncertainty from the token. since token embeddings directly affect token labels and further affect entity labels, high uncertainty in a token embedding will result in a predicted entity label with high uncertainty. therefore, in the context of ue-ner, a token uncertainty in ue-ner consists the individual token uncertainty and the uncertainty transmitted from other tokens. however, the current uncertainty estimation methods ignore the uncertainty transmission between tokens. especially, current uncertainty estimation methods can be classified into two main categories: parameter-distribution-based methods, such as bayesian neural networks (bnn) @xcite , which learns a distribution over the model parameters; and sampledistribution-based methods, which calculate uncertainty scores based on the distribution of training samples @xcite @xcite . these methods primarily focus on image or text classification, where correlations between different images or texts are weak or lim-the samsung galaxys21 was featured in ces 2022. ## ue-ner task setting before we introduce ue-ner, we first introduce ner tasks, which is a representative sequential labeling task. given a text @xmath0, x 2 , ..., x n ] with n tokens, where x @xmath1, ner task aims at learning a ner model predicting their token labels. then, the entities are extracted by the token labels based on the bioes mechanism @xcite (e.g., "brack" with b-per label, and "obama" with i-per label). moreover, the extracted entities are classified by merging the entity tokens. for example, "brack obama" is categorized as a person because these two tokens are categorized as the beginning and intermediate of the person label. for the ue-ner task, we aim to learn a ue-ner model φ to predict the confidence of each predicted token label. we apply φ for ood detection, which is a common task to evaluate uncertainty estimation @xcite . concretely, the training data and validation data for φ are the in-domain (id) text without ood entities. the testing data of φ includes both id text and ood text, where ood text has both id and ood entities. a better φ should detect more ood entities in the testing set and have better ner performance. ## preliminary: posterior network the parameter-distribution-based uncertainty estimation method is usually implemented via ensemble sampling @xcite and thus requires multiple forward passes to estimate uncertainty, which is time-consuming. in contrast, evidential deep learning (edl) @xcite is a representative sample-distribution-based uncertainty estimation method and is implemented via a deterministic model, thus requiring only one forward pass to estimate uncertainty. due to its efficiency, we choose edl. in edl, considering the classification task and given the input vector x, the class prediction @xmath2.the categorical distribution naturally follows a dirichlet distribution, i.e. y ∼ cat(p), p ∼ dir(α) as a representative model of edl, posterior network (pn) @xcite is originally designed for image classification and involves two main steps. first, a feature encoder maps the raw features into a low-dimensional latent space. second, a normalizing flow such as radial @xcite is used to estimate class-wise density on the latent space, which is proportional to the class-wise evidence. essentially, a greater density for a particular class implies stronger evidence belonging to this class for the given example. pn is trained with the sum of two loss l uce and l er for n training samples as below, ## model we choose pn as it does not require ood data during training. in contrast, prior network (malinin and gales, 2018), another representative edl method, necessitates ood data in training. furthermore, even if ood data is available, it may not cover all possible ood scenarios. thus, we opt for uncertainty transmission based on pn. ## conclusion incorrect ner predictions incur significant penalties. we primarily focus on ue-ner, which differs from prior uncertainty estimation methods that focus on sample-level labeling. ue-ner centers on token-level sequential labeling, addressing the overlooked transmitted uncertainty from contextual tokens. we introduce slpn to calculate uncertainty from both the token itself and contextual tokens, enhancing ood detection in ner. additionally, for ood detection in ner, ws entities are not applicable. thus, we divide the entities into two distinct subsets-one for ood detection and the other for ws detection. our experiments validate slpn's effectiveness and the importance of considering uncertainty propagation in ue-ner.