Understanding Model Robustness to User-generated Noisy Texts (2021)

Abstract: Sensitivity of deep-neural models to input noise is known to be a challenging problem. In NLP, model performance often deteriorates with naturally occurring noise, such as spelling errors. To mitigate this issue, models may leverage artificially noised data. However, the amount and type of generated noise has so far been determined arbitrarily. We therefore propose to model the errors statistically from grammatical-error-correction corpora. We present a thorough evaluation of several state-of-the-art NLP systems’ robustness in multiple languages, with tasks including morpho-syntactic analysis, named entity recognition, neural machine translation, a subset of the GLUE benchmark and reading comprehension. We also compare two approaches to address the performance drop: a) training the NLP models with noised data generated by our framework; and b) reducing the input noise with external system for natural language correction. The code is released at https://github.com/ufal/kazitext .

## introduction although there has recently been an amazing progress in variety of nlp tasks @xcite with some models even reaching performance comparable to humans on certain domains @xcite , it has been shown that the models are very sensitive to noise in data @xcite . multiple areas of nlp have been studied to evaluate the effect of noise in data @xcite @xcite and a framework for text corruption to test nlp models robustness is also available @xcite . however, all these systems introduce noise in a custom-defined, arbitrary level and typically for a single language. we suggest modeling natural noise statistically from corpora and we propose a framework with the following distinctive features: furthermore, we also present a thorough evaluation of several current state-of-the-art nlp systems' with varying level of data noisiness and a selection of error aspects in multiple languages. the nlp tasks include morpho-syntactic analysis, named entity recognition, neural machine translation, a subset of glue benchmark and reading comprehension. we conclude that: we also compare two approaches for increasing models robustness to noise: training with noise and external grammatical-error-correction (gec) preprocessing. our findings suggest that training with noise is beneficial for models with large capacity and large training data (neural machine translation), while the preprocessing with grammatical-errorcorrection is more suitable for limited-data classification tasks, such as morpho-syntactic analysis. finally, we also offer an evaluation on authentic noise: we assembled a new dataset with authentic czech noisy sentences translated into english and we evaluate the noise-mitigating strategies in the neural machine translation task on this dataset. ## related work many empirical findings have shown the fact that data with natural noise deteriorate nlp systems performance. @xcite found that natural noise such as misspellings and typos cause significant drops in bleu scores of characterlevel machine translation models. to increase the model's robustness, they trained the model on a mixture of original and noisy input and found out that it learnt to address certain amount of errors. similar findings were observed by @xcite who tested machine translation and morphological tagging under three types of word-level errors. ribeiro et al. ( 2018 ) defined a set of substitution rules that produce semantically equivalent text variants. they used them to test systems in machine comprehension, visual question answering and sentiment analysis. @xcite created a new test set for natural language inference and showed that current systems do not generalize well even for a single-word replacements by synonyms and antonyms. @xcite implemented a framework for introducing multiple noise types into text such as removing or swapping articles, rewriting digit numbers into words or introducing errors in spelling. they found out in four nlp tasks that even recent state-of-the-art systems based on contextualized word embeddings are not completely robust against such natural noise. they also retrained the systems on noisy data and observed improvements for certain error types. similarly to @xcite , we also developed a general framework that allows to test a variety of nlp tasks. the difference is that we estimate the probabilities of individual error types from real-world error corpora. this makes the generated sentences more similar to what humans would do. moreover, since we defined the individual error types with no language-specific rules, we can apply it to multiple languages with an available annotated grammatical-error corpus. grammatical-error corpora are typically used as training data for estimating error statistics in gec systems. in a setting similar to ours, @xcite ; @xcite also estimated error statistics and used them to generate additional training data for gec systems. however, compared to our approach, they defined only a small set of predefined error categories and used it specifically for training gec systems whereas we also use it to asses model performance in noisy scenarios. ## modeling natural noise from corpora robustness of nlp models to natural noise would ideally be evaluated on texts with authentic noise, with error corrections annotated by humans. (we present such authentic data evaluation in section 7.) this perfect-world setting, however, requires an immense annotation effort, as multiple target domains have to be covered by well-educated human annotators for multiple nlp tasks in a range of languages. to ease the annotation burden, we propose a new framework, named kazitext, for introducing natural-like errors in a text. the core of kazitext is a set of several common error type classes, aspects (following naming convention of @xcite . the aspects are composable (can be combined) and the probability of the aspect manifestation as well as the aspect's internal probabilities are estimated from grammatical-error-correction corpora. ## robustness to noise we evaluated the models robustness both to the amount of noise (figure 2 ) and to error types (figures 3 and 4). a unifying trend can be observed in models performance with respect to increasing percentage of token edits. solid lines in figure 2 display the morpho-syntactic mlas, ner f1 and nmt bleu on texts with up to 30% of token edits. the relative performance decreases roughly linearly with the amount of token edits, in accordance with previous findings @xcite . the tendency is consistent across tasks, languages and profiles: for example, compare the czech and english second learners profiles in morpho-syntactic analysis (figure 2a ) or czech native speakers and czech second learners profiles in the nmt clean model (figure 2c ), which exhibit similar behaviour despite their differing distributions of aspects (figure 1 ). this consistency implies that it is the sheer amount of noise rather than the distribution of aspects, that contributes to the model performance deterioration. more results are available in supplementary material (figures s2 and s3 ). estimating the amount of noise is important, as the corpus error level differs greatly across languages and profiles. for example, compare the second learners profile in english (11.3% token edits) and czech (27.1%) in figure 2a , or in czech, see native speakers (6.4%) and second learners from a qualitative point of view, spelling and affixes make for the major performance drop in morpho-syntactic analysis (figure 3 ), ner (figure 4a ) and nmt (figure 4b ). some tasks are more sensitive to certain aspects: casing is a crucial aspect for ner. this is clearly shown in the czech natives informal profile, which contains text scraped from the internet discussions and contains nontrivial amount of casing errors (figure 4a ). we further elaborate the casing aspect effect on ner in section s2 in supplementary material. in nmt and reading comprehension, errors in punctuation seem to decrease the model performance consistently across all profiles (figures 4b and 4c , respectively). for czech as a language with diacritic marks, diacritics is an interesting aspect. we can see that when it is introduced at a corpus error level, the czech model's performance on lemmas drops by circa 7 percent. figure s1 in supplementary material further illustrates that performance significantly deteriorates when all diacritics is stripped, which is quite common in informal web texts. similarly, to emphasize the effect of the diacritization aspect on nmt, we created a new profile natives informal w/o diacritics from the natives informal profile by stripping all diacritization. figure 4b shows that not using diacritics at all results in a performance drop of ca. 10 bleu points. some tasks are more sensitive to noise than others. lemmatization is the most sensitive to errors (20 times more errors when processing czech second learners texts with a clean model, see figure 3 ), which is understandable, given that all lemma characters must be generated correctly from a corrupted surface token. the effect on pos tagging is the least pronounced (figure 3 ), although 8 times as many errors in czech (when processing noisy texts with a clean model) makes the pos tags much less reliable. ## noise-coping strategies we implemented and evaluated two strategies to alleviate the performance drop on noisy inputs: external and internal correction. in the external correction approach, we use a separately trained grammatical-error-correction model to denoise texts before inputting them to the model itself. in the internal correction approach, we instead directly train the model on a combination of noisy and authentic texts. we hypothesise that the external approach may be better in scenarios with small amount of annotated data. in such cases, only few iterations over training data are typically performed to prevent overfitting, and we suppose that learning the task itself and denoising at the same time would harm its performance a lot. contrarily, with enough data and appropriate model capacity, learning the denoising and the task jointly may reduce the amount of potential false positives that might be otherwise proposed by the external language corrector. ## evaluating on authentic user text we assembled a new dataset for mt evaluation consisting of 2223 authentic czech noisy input sentences translated into english, which we release at http://hdl.handle.net/11234/1-3775 . the sentences originate from the project faust foot_3 where they were collected from various users of reverso.net. the advantage of this dataset is that in addition to the original czech noisy sentences, there are manually corrected czech sentences and manual translations to english. on this dataset, we evaluate our neural machine translation models from section 4.3 and section 6, specifically the clean model trained on clean data, noise-trained model trained on a mixture of authentic and noised data and their combination with external grammatical-error-correction system. the results of these systems on authentic noisy texts are presented in ## conclusions we estimated natural error probabilities statistically from real-world grammatical-error-correction corpora in order to model and generate noisy inputs for machine learning tasks. we extensively evaluated several state-of-the-art nlp downstream systems with respect to their robustness to input noise, both in increasing level of text noisiness and in variations of error types. we confirmed that the noise hurts the model performance substantially and we compared two coping strategies: training with noise and preprocessing with gec, concluding that each strategy is beneficial in different scenarios. finally, we also presented authentic noisy data evaluation using a newly assembled dataset for machine translation with authentic czech noisy sentences translated to english. we release both the new framework (under mpl 2.0) at https: //github.com/ufal/kazitext and the newly assembled dataset (under cc by-nc-sa license) at http://hdl.handle.net/11234/1-3775 .