Balanced Joint Adversarial Training for Robust Intent Detection and Slot Filling (2020)

Abstract: Joint intent detection and slot filling has recently achieved tremendous success in advancing the performance of utterance understanding. However, many joint models still suffer from the robustness problem, especially on noisy inputs or rare/unseen events. To address this issue, we propose a Joint Adversarial Training (JAT) model to improve the robustness of joint intent detection and slot filling, which consists of two parts: (1) automatically generating joint adversarial examples to attack the joint model, and (2) training the model to defend against the joint adversarial examples so as to robustify the model on small perturbations. As the generated joint adversarial examples have different impacts on the intent detection and slot filling loss, we further propose a Balanced Joint Adversarial Training (BJAT) model that applies a balance factor as a regularization term to the final loss function, which yields a stable training procedure. Extensive experiments and analyses on the lightweight models show that our proposed methods achieve significantly higher scores and substantially improve the robustness of both intent detection and slot filling. In addition, the combination of our BJAT with BERT-large achieves state-of-the-art results on two datasets.

## introduction intent detection and slot filling are two critical components in dialogue systems. although these two tasks are normally considered as parallel tasks, they may inherently correlate with each other as one is helpful in defining the other. some existing works @xcite @xcite indicate that sharing parameters in an encoder module to simultaneously detect the intent and fill the slots of an utterance can utilize such correlation between the two tasks. the joint modeling of them has therefore achieved tremendous success @xcite @xcite and further improved the performance of spoken language understanding (slu) systems. however, recent outstanding joint models still show insufficient robustness. for example, some small changes of inputs can mislead the models to give wrong prediction. as shown in figure 1 (a), misspelled words "sanfranciso" (missing a blank) and "pitsburgh" (missing "t") are easily to fool the neural models to produce wrong slot predictions. one might think that such wrong predictions are related to the input word embeddings and using character embeddings can avoid these prediction errors. however, we find in our preliminary experiments that character embeddings cannot fully eliminate such errors as it is not easy to reconstruct the meaning of corresponding slot words based on character embeddings. in figure 1 @xcite , word-level perturbations such as inserted word "video" or substituted word "movie" can lead to incorrect prediction in either slot filling or intent detection. in real world, spelling errors are easily found and word variances are pervasive as utterances are from different people with different text styles. this is challenging for many joint models of intent detection and slot filling. in this paper, we resort to adversarial training for solving this problem. adversarial training provides a way to inject perturbed inputs (adversarial examples) into training data to regularize models for robustness. specifically, we propose a joint adversarial training (jat) model to improve the robustness of the joint intent detection (id) and slot filling (sf). in jat, we first add tiny noises to the original training data on word (character) embeddings. the perturbed data created in this way are referred to as joint adversarial examples. we then train the joint id and sf model to defend against these adversarial examples so as to make the model robust to small perturbations. by adding perturbations to word/character embeddings, we enable the joint model to learn from adversarial examples previously unseen to deal with variances and perturbations in utterances. in doing so, we hope that small changes in utterances cannot fool the new joint model trained with adversarial examples. furthermore, as the loss function of the joint model is the combination of the id loss and sf loss, the two losses are sensitive to adversarial examples in different degrees, which may cause the joint training to be unstable. to overcome this, we further propose a balanced joint adversarial training (bjat) model. particularly, we add a balanced factor to the joint loss function as a regularization term to obtain the impact equilibrium that adversarial examples impose on the id and sf loss. bjat shows a more stable training process and achieves better performance than jat. to summarize, the contributions of this work are threefold as follows: ## methodology in this section, we elaborate the proposed joint adversarial training model for the joint id and sf and its balanced variant. we first introduce the joint model that we use as our baseline for intent detection and slot filling. ## experiments we conducted intensive experiments on two benchmark datasets to validate the effectiveness of the proposed two adversarial training models for joint intent detection and slot filling. ## analysis in this section, we further take a deep look into data to investigate how the proposed adversarial training models improve intent detection and slot filling. ## conclusion we have presented a joint adversarial training framework to improve the robustness of the joint intent detection and slot filling. to balance the impacts of the joint adversarial examples on the intent detection loss and slot filling loss, we further propose a balanced joint adversarial training model with a balance factor as a regularization term. experiment results on the atis and snips datasets demonstrate the capability of our approaches in improving both the performance and robustness. our in-depth analyses further disclose that both jat and bjat can (1) boost slot filling accuracy for rare/unseen words, (2) deal with noisy inputs such as spelling errors to some extent, (3) perform well on complex sentences with multiple slots, and (4) successfully handle the problem of unbalanced intent distribution. our experiments and analyses also suggest that the balanced jat is better than jat in both the performance and training stability.