Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation? (2024)

Abstract: This work investigates the impact of data augmentation on confidence calibration and uncertainty estimation in Named Entity Recognition (NER) tasks. For the future advance of NER in safety-critical fields like healthcare and finance, it is essential to achieve accurate predictions with calibrated confidence when applying Deep Neural Networks (DNNs), including Pre-trained Language Models (PLMs), as a real-world application. However, DNNs are prone to miscalibration, which limits their applicability. Moreover, existing methods for calibration and uncertainty estimation are computational expensive. Our investigation in NER found that data augmentation improves calibration and uncertainty in cross-genre and cross-lingual setting, especially in-domain setting. Furthermore, we showed that the calibration for NER tends to be more effective when the perplexity of the sentences generated by data augmentation is lower, and that increasing the size of the augmentation further improves calibration and uncertainty.

## introduction named entity recognition (ner) is a one of the fundamental tasks in natural language processing (nlp) to find mentions of named entities and classify them into predefined categories. the predicted information by ner is essential for downstream tasks like event detection @xcite , information retrieval @xcite , and masking of personal user information @xcite . due to the demand, ner is the underlying technology for information extraction from text and documents. based on the recent advances in deep neural networks (dnns), ner's performance is also improved like other nlp fields. in recent years, pretrained language models (plms) based architectures, such as bert @xcite and de-berta @xcite , have been strong baselines in many nlp tasks, including ner. in general, however, dnns are prone to miscalibration @xcite , including plms @xcite ; calibration means the predicted confidence of the model aligns with the accuracy. 1 the problem causes dnns to make incorrect predictions with high confidence, which limits the applicability of dnns on the number of domains where the cost of errors is high, e.g., healthcare and finance. therefore, dnns need to provide high prediction performance with appropriately calibrated confidence at the same time. confidence calibration and uncertainty estimation methods are ways to solve the miscalibration of dnns, and have been applied in nlp tasks such as text classification @xcite , structured prediction @xcite , question answering @xcite , and machine translation @xcite . however, many methods for confidence calibration and uncertainty estimation, typically monte-carlo dropout (mc dropout) @xcite , are computationally expensive due to multiple stochastic inferences, making them difficult for real-world application. data augmentation has also been applied for ner @xcite , though, it was focusing on the generalization ability on low-resource data. in computer vision (cv) areas, data augmentation makes the model more robust to the input and leads to confidence calibrations @xcite , in which the same labels are trained on different representations of the input than the original data. based on the findings of these previous studies, there is a possibility that data augmentation in ner can improve confidence calibration without increasing inference time, in contrast to the conventional confidence calibration and uncertainty estimation methods. in this study, we conducted comprehensive experiments to analyze the impact of data augmentation methods for ner @xcite on the confidence calibration and uncertainty in the cross-genre and cross-lingual settings on ontonotes 5.0 @xcite and multiconer @xcite , respectively. our experiments yield several findings. first, some data augmentation methods in ner lead to improved confidence calibration and uncertainty estimation, especially in-domain. in particular, entityprediction-based data augmentation @xcite and entity replacement from the same entity type @xcite show good performance. on the other hand, common confidence calibration methods, mc dropout or ts @xcite have worse confidence calibration and uncertainty estimation performance than the data augmentation methods in ner, even though the data augmentation methods do not aim to improve confidence calibration and uncertainty estimation. moreover, increasing the augmentation size improves performance in confidence calibration and uncertainty estimation. the improvement tends to be better the lower the perplexity of the sentences generated by the data augmentation. our code is available on https://github.com/ wataruhashimoto52/ner_da_uncertainty . ## methods in this section, we describe the popular baseline methods for confidence calibration and data augmentation methods for ner. details about existing calibration methods are described in appendix b. ## evaluation metrics we use expected calibration error (ece), maximum calibration error (mce), and area under precision-recall curve (auprc) to evaluate confidence calibration and uncertainty estimation. ## results and discussion we present the performance of cross-genre and cross-lingual confidence calibration and uncertainty estimation as the main results. the crossgenre evaluations are quantified by learning on a training set in one genre and evaluating calibration and uncertainty on a test in another genre. similarly, in the cross-lingual evaluations, we train the model in one language (in this research, we use english; en) and evaluate the calibration and uncertainty on a test set in another language. ## conclusion in this paper, we investigated the impact of data augmentation on the confidence calibration and uncertainty estimation in ner in terms of genre and language, using several metrics. first, we find that melm, mr, and sr lead to better calibration and uncertainty performance in the id setting consistently. on the other hand, in the ood setting, uncertainty estimation by data augmentation is less effective, especially when the target domain is far from the source domain. second, our results suggest that the lower the perplexity of the augmented data, as in mr, the further better the calibration and uncertainty performance as the augmentation size is increased. data augmentation methods for ner do not require changes to the model structure and only require more data to improve entity-level calibration and performance without the need to change the model structure. our findings indicate the effectiveness of uncertainty estimation through data augmentation for ner, and will be expected to stimulate future research based on their limitations.