NAT : Noise-Aware Training for Robust Neural Sequence Labeling (2020)

Abstract: Sequence labeling systems should perform reliably not only under ideal conditions but also with corrupted inputs—as these systems often process user-generated text or follow an error-prone upstream component. To this end, we formulate the noisy sequence labeling problem, where the input may undergo an unknown noising process and propose two Noise-Aware Training (NAT) objectives that improve robustness of sequence labeling performed on perturbed input: Our data augmentation method trains a neural model using a mixture of clean and noisy samples, whereas our stability training algorithm encourages the model to create a noise-invariant latent representation. We employ a vanilla noise model at training time. For evaluation, we use both the original data and its variants perturbed with real OCR errors and misspellings. Extensive experiments on English and German named entity recognition benchmarks confirmed that NAT consistently improved robustness of popular sequence labeling models, preserving accuracy on the original input. We make our code and data publicly available for the research community.

## introduction sequence labeling systems are generally trained on clean text, although in real-world scenarios, they often follow an error-prone upstream component, such as optical character recognition (ocr; @xcite or automatic speech recognition (asr; @xcite . sequence labeling is also often performed on user-generated text, which may contain spelling mistakes or typos @xcite . errors introduced in an upstream task are propagated downstream, diminishing the performance of the end-to-end system @xcite . while humans can easily cope with typos, misspellings, and the complete omission of letters when reading @xcite reference text: singapore sees prestige in hosting wto . training loss ̃ ( ) ( ) ̃ figure 1 : an example of a labeling error on a slightly perturbed sentence. our noise-aware methods correctly predicted the location (loc) label for the first word, as opposed to the standard approach, which misclassified it as an organization (org). we complement the example with a high-level idea of our noise-aware training, where the original sentence and its noisy variant are passed together through the system. the final loss is computed based on both sets of features, which improves robustness to the input perturbations. in this work, we are concerned with the performance difference of sequence labeling performed on clean and noisy input. is it possible to narrow the gap between these two domains and design an approach that is transferable to different noise distributions at test time? inspired by recent research in computer vision @xcite , neural machine translation (nmt; @xcite , and asr @xcite , we propose two noise-aware training (nat) objectives that improve the accuracy of sequence labeling performed on noisy input without reducing efficiency on the original data. figure 1 illustrates the problem and our approach. our contributions are as follows: ## noise-aware training 3.1 noise model to model the noise, we use the character-level noisy channel matrix γ, which we will refer to as the character confusion matrix ( §2.2). ## related work improving robustness has been receiving increasing attention in the nlp community. the most relevant research was conducted in the nmt domain. noise-additive data augmentation a natural strategy to improve robustness to noise is to augment the training data with samples perturbed using a similar noise model. heigold et al. (2018) demonstrated that the noisy input substantially degrades the accuracy of models trained on clean data. they used word scrambling, as well as character flips and swaps as their noise model, and achieved the best results under matched training and test noise conditions. belinkov and bisk (2018) reported significant degradation in the performance of nmt systems on noisy input. they built a look-up table of possible lexical replacements from wikipedia edit histories and used it as a natural source of the noise. robustness to noise was only achieved by training with the same distribution-at the expense of performance degradation on other types of noise. in contrast, our method performed well on natural noise at test time by using a simplified synthetic noise model during training. karpukhin et al. (2019) pointed out that existing nmt approaches are very sensitive to spelling mistakes and proposed to augment training samples with random character deletions, insertions, substitutions, and swaps. they showed improved robustness to natural noise, represented by frequent corrections in wikipedia edit logs, without diminishing performance on the original data. however, not every word in the vocabulary has a corresponding misspelling. therefore, even when noise is applied @xmath0. 0 5 10 15 20 error rate [%] augm stabil 0 (a) divided by the edit distance value. o per loc org misc named entity class. 0 5 10 15 error rate [%] at the maximum rate, only a subset of tokens is perturbed (20-50%, depending on the language). in contrast, we used a confusion matrix, which is better suited to model statistical error distribution and can be applied to all tokens, not only those present in the corresponding look-up tables. ## conclusions in this paper, we investigated the difference in accuracy between sequence labeling performed on clean and noisy text ( §2.3). we formulated the noisy sequence labeling problem ( §2.2) and introduced a model that can be used to estimate the real noise distribution ( §3.1). we developed the noise induction procedure that simulates the real noisy input ( §3.2). we proposed two noise-aware training methods that boost sequence labeling accuracy on the perturbed text: (i) our data augmentation approach uses a mixture of clean and noisy examples during training to make the model resistant to erroneous input ( §3.3). (ii) our stability training algorithm encourages output similarity for the original and the perturbed input, which helps the model to build a noise invariant latent representation ( §3.4). our experiments confirmed that nat consistently improved efficiency of popular sequence labeling models on data perturbed with different error distributions, preserving accuracy on the original input ( §4). moreover, we avoided expensive re-training of embeddings on noisy data sources by employing existing text representations. we conclude that nat makes existing models applicable beyond the idealized scenarios. it may support an automatic correction method that uses recognized entity types to narrow the list of feasible correction candidates. another application is data anonymization @xcite . future work will involve improvements in the proposed noise model to study the importance of fidelity to real-world error patterns. moreover, we plan to evaluate nat on other real noise distributions (e.g., from asr) and other sequence labeling tasks to support our claims further.