Uncertainty-Aware Label Refinement for Sequence Labeling (2020)

Abstract: Conditional random fields (CRF) for label decoding has become ubiquitous in sequence labeling tasks. However, the local label dependencies and inefficient Viterbi decoding have always been a problem to be solved. In this work, we introduce a novel two-stage label decoding framework to model long-term label dependencies, while being much more computationally efficient. A base model first predicts draft labels, and then a novel two-stream self-attention model makes refinements on these draft predictions based on long-range label dependencies, which can achieve parallel decoding for a faster prediction. In addition, in order to mitigate the side effects of incorrect draft labels, Bayesian neural networks are used to indicate the labels with a high probability of being wrong, which can greatly assist in preventing error propagation. The experimental results on three sequence labeling benchmarks demonstrated that the proposed method not only outperformed the CRF-based methods but also greatly accelerated the inference process.

## introduction linguistic sequence labeling is one of the fundamental tasks in natural language processing. it has the goal of predicting a linguistic label for each word, including part-of-speech (pos) tagging, text chunking, and named entity recognition (ner). benefiting from representation learning, neural network-based approaches can achieve state-ofthe-art performance without massive handcrafted feature engineering @xcite @xcite @xcite . although the use of representation learning to obtain better text representation is very successful, figure 1 : schematic of label refinement framework @xcite . the goal is refining the label of "arab" using contextual labels and words, while the refinement of other correct labels may be negatively impacted by incorrect draft labels. creating better models for label dependencies has always been the focus of sequence labeling tasks @xcite @xcite . among them, the crf layer integrated with neural encoders to capture label transition patterns @xcite has become ubiquitous in sequence labeling tasks. however, crf only captures the neighboring label dependencies and must rely on inefficient viterbi decoding. many of the recent methods try to introduce label embeddings to manage longer ranges of dependencies, such as two-stage label refinement @xcite and seq2seq @xcite frameworks. in particular, @xcite introduced a hierarchically-refined representation of marginal label distributions, which predicts a sequence of draft labels in advance and then uses the word-label interactions to refine them. although these methods can model longer label dependencies, they are vulnerable to error propagation: if a label is mistakenly predicted during inference, the error will be propagated and the other labels conditioned on this one will be impacted @xcite . as shown in figure 1 , the label attention network (lan) @xcite draft uncertainty refinement #tokens 0.018 ¥ 39 0.524 ¥ 54 zhang, 2019) would negatively impact the correct predictions in the refinement stage. there are 39 correct tokens that have been incorrectly modified ( in this work, we propose a novel two-stage uncertainty-aware label refinement network (uanet). at the first stage, the bayesian neural networks take a sentence as input and yield all of the draft labels together with corresponding uncertainties. at the second stage, a two-stream self-attention model performs attention over label embeddings to explicitly model the label dependencies, as well as context vectors to model the context representations. all of these features are fused to refine the potentially incorrect draft labels. the above label refinement operations can be processed in parallel, which can avoid the use of viterbi decoding of the crf for a faster prediction. experimental results on three sequence labeling benchmarks demonstrated that the proposed method not only outperformed the crf-based methods but also significantly accelerated the inference process. the main contributions of this paper can be summarized as follows: 1) we propose the use of bayesian neural networks to estimate the uncertainty of predictions and indicate the potentially incorrect labels that should be refined; 2) we propose a novel two-stream self-attention refining framework to better model different ranges of label dependencies and word-label interactions; 3) the proposed parallel decoding process can greatly speed up the inference process; and 4) the experimental results across three sequence labeling datasets indicate that the proposed method outperforms the other label decoding methods. ## uncertainty-aware label refinement in this work, we propose a novel sequence labeling framework, which incorporates bayesian neural networks to estimate the epistemic uncertainty of the draft labels. the uncertain labels that have a high probability of being wrong can be refined by a two-stream self-attention model using long-term label dependencies and word-label interactions. the proposed model is shown in figure 2 . ## experimental setup in this section, we describe the datasets across different sequence labeling tasks, including two english ner datasets and one pos tagging dataset. we also detail the baseline models for comparison. finally, we clarify the hyperparameters configuration of our uncertainty-aware refinement network. ## results and analysis in this section, we present the experimental results of the proposed and baseline models. we show that the proposed method not only achieves better performance but also has a significant speed advantage. since our contribution is mainly focused on the label decoding layer, the proposed model can also be combined with the latest pretrained model to further improve performance. ## conclusions in this work, we introduce a novel sequence labeling framework that incorporates bayesian neural networks to estimate model uncertainty. we find that the model uncertainty can effectively indicate the labels with a high probability of being wrong. the proposed method can selectively refine the uncertain labels to avoid the side effects of the refinement on correct labels. in addition, the proposed model can capture different ranges of label dependencies and word-label interactions in parallel, which can avoid the use of viterbi