On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise (2021)

Abstract: Intent Classification (IC) and Slot Labeling (SL) models, which form the basis of dialogue systems, often encounter noisy data in real-word environments. In this work, we investigate how robust IC/SL models are to noisy data. We collect and publicly release a test-suite for seven common noise types found in production human-to-bot conversations (abbreviations, casing, misspellings, morphological variants, paraphrases, punctuation and synonyms). On this test-suite, we show that common noise types substantially degrade the IC accuracy and SL F1 performance of state-of-the-art BERT-based IC/SL models. By leveraging cross-noise robustness transfer, i.e. training on one noise type to improve robustness on another noise type, we design aggregate data-augmentation approaches that increase the model performance across all seven noise types by +10.8\% for IC accuracy and +15 points for SL F1 on average. To the best of our knowledge, this is the first work to present a single IC/SL model that is robust to a wide range of noise phenomena.

## introduction intent classification (ic) and slot labeling (sl) tasks (shown in ## related work prior works demonstrate the brittleness of neural systems in nlp to different noise types. adversarial nlp focuses on construction of examples that impair a model's performance @xcite @xcite . a parallel thread of research investigates the behavior of models towards real-word noise types. machine translation (mt) systems perform poorly in the presence of synthetic and natural character-level changes (eg. character swaps, typographical errors, misspellings) @xcite ; similar conclusions hold for casing noise @xcite . other works investigate robustness of nlp models to paraphrases @xcite , misspellings @xcite , and morphological variants @xcite . the most common approach to improving model robustness is data augmentation, either at pre-training or training time. for example, in @xcite , augmentation with adversarial examples and a single epoch of pretraining can improve robustness of the model for question answering (qa) and mt tasks; a comprehensive summary of da techniques can be found in @xcite . recent works present frameworks for comprehensive model-agnostic testing of nlp systems to grammatical and noise variants @xcite . all of these prior works focus on either (1) improving robustness to individual noise types or (2) simply evaluating model performance on synthetically-generated noise variants. in contrast, we construct a single model that is robust to all seven noise types seen in beyond training-data augmentation, pre-training robust word-embeddings is a popular defense against adversarial examples @xcite , misspellings @xcite , and character ordering @xcite . in @xcite , authors investigate the impact of different subword tokenization techniques on model robustness. effective regularization has also been shown to improve the robustness of ic/sl models to paraphrases @xcite . yet many of the techniques to improve robustness are not comparable due to evaluation on different test sets. hence, we provide high-quality noisy test data to serve as a common test bed. ## noise categories we consider seven types of noise that are prevalent in the traffic of a task oriented dialogue service -namely, misspellings, casing, synonyms, paraphrases, punctuation, morphological variants, and abbreviations (see ## approach in this section, we highlight the various approaches we consider to construct a single model robust to various noise types seen in live traffic. ## experiments we first evaluate the trade-off between use of data augmentation during pre-training vs. fine-tuning time. we focus the majority of our analysis on fine-tuning time augmentation as we show it outperforms pre-training augmentation. ## conclusion in this paper, we show that sota bert based ic/sl models are not robust to many real world noise types found in production. we further demonstrate that cross-noise robustness transfer -training on one noise type to improve model robustness on another noise type -yields gains for a number of noise type pairs. through aggregate data augmentation (a-da), we leverage cross-noise robustness transfer to improve the model's average performance by +10.8% on ic and +15 points on sl f1. despite the gains we obtain, our benchmark provides substantial headroom to improve model performance on the abbreviation, synonym, and paraphrase noise types. we hope that our benchmark will support future research in this direction and enable the design of robust ic/sl models for goal-oriented dialogue systems.