Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding (2021)

Abstract: Lack of training data presents a grand challenge to scaling out spoken language understanding (SLU) to low-resource languages. Although various data augmentation approaches have been proposed to synthesize training data in low-resource target languages, the augmented data sets are often noisy, and thus impede the performance of SLU models. In this paper we focus on mitigating noise in augmented data. We develop a denoising training approach. Multiple models are trained with data produced by various augmented methods. Those models provide supervision signals to each other. The experimental results show that our method outperforms the existing state of the art by 3.05 and 4.24 percentage points on two benchmark datasets, respectively. The code will be made open sourced on github.

## introduction spoken language understanding (slu) is a key component in task-oriented dialogue systems. slu consists of two subtasks: intent detection and slot tagging @xcite . although promising progress has been achieved on slu in english @xcite @xcite , those methods need large amounts of training data, and thus cannot be applied to low-resource languages where zero or few training data is available. in this paper, we target at the extreme setting for cross-lingual slu where no labeled data in target languages is assumed, which is critical for industry practice, since annotating a large slu dataset with high quality for every language is simply infeasible. existing cross-lingual transfer learning methods mainly build on pre-trained cross-lingual word embeddings @xcite or contextual models @xcite @xcite , which represent texts with similar meaning in different languages close to each other in a shared vector space. those approaches often show good performance on intent detection. the results on slot tagging, however, are often unsatisfactory, especially for distant languages, which are dramatically different from english in scripts, morphology, or syntax @xcite @xcite . several studies @xcite @xcite show that adding translated data into the fine-tuning process of pretrained models can improve the results of cross-lingual slu substantially when no golden-labeled training data in target languages is available. for example, machine translation can be employed to translate the training data in english into target languages, and some alignment methods, such as attention weights @xcite , fastalign @xcite or giza++ @xcite , can be further applied to label the translated data. another approach to alleviate the problem of data scarcity is to automatically generate training data. recently, some methods for monolingual slu @xcite @xcite automatically label domain-specific data or use pre-trained language models to generate additional data. however, the synthesized training data derived from both the translation approach and the data generation approach may be quite noisy and may contain errors in label. for the translation approach, both the translation process and the alignment process may generate errors @xcite . for the data generation approach, it is often hard to control a right tradeoff between generating correct but less diverse data and generalizing more diverse data but with more noise. moreover, generating synthetic training data across languages further adds challenges to the robustness of the generation methods. to filter out noise in the synthesized training data, a few methods are proposed, such as the mask mechanism @xcite , the soft alignment method @xcite , the unsupervised adaptation method @xcite , the rule-based filtering method @xcite , the classifier based filtering method @xcite and the language model score based filtering method @xcite . these methods rely on either adhoc rules or extra models. although they have shown promising results, each of them considers only a single source for data augmentation. it is still challenging to differentiate noisy instances from useful ones, since all those instances are sampled from the same distribution generated by the same method. in this paper, we regard both the translation approach and the generation approach as data augmentation methods. we tackle the problem of reducing the impact of noise in augmented data sets. we develop a principled method to learn from multiple noisy augmented data sets for cross-lingual slu, where no golden-labeled target language data exists. our major technical contribution consists of a series of denoising methods including instance relabeling, co-training and instance re-weighting. first, motivated by the self-learning method @xcite , we design a model-ensemble-based instance relabeling approach to correct the noisy labels of augmented training data in low resource languages. as teacher models can not always generate correct labels, the original self-learning method tends to suffer from accumulated errors caused by model predictions. to alleviate the problem of accumulated errors, in our instance relabeling approach, we use crowd-intelligence from multiple models to derive the more reliable labels of pseudo training instances. besides, we filter out noisy instances based on co-training and instance re-weighting strategies to reduce the impact of incorrect predictions on subsequent training. our training strategy does not follow a traditional teacher-student manner. instead, we use model predictions in the last epoch as pseudo labels in the current epoch to compute loss which saves training time. in order to filter out noisy instances, we adopt a co-training mechanism, which uses selected instances from the other models to train the current model. different from the co-teaching method @xcite where two models are trained with the same data, we propose multiple models should be trained with multiple different noisy augmented data where noise may be largely independent. it is because deep neural networks have a high capacity to fit noisy labels. when two models are trained with the same data, we tend to obtain two similar models. the co-teaching method gradually becomes a naive selection method with two models ensembled. in our co-training method, by employing very different data generation methods, we hope to attain that the noise from different sources may be largely independent and models can learn different knowledge from them. therefore, the instances that pass the screening process of the other models can serve as the supervision signals to the current model which alleviates the problem of accumulated errors caused by selection bias. last, we further propose an instance reweighting technique to adjust the weights of training instances adaptively. as we do not have real training data in target languages, we can use the consistency among the soft labels predicted by different models to predict the reliability of the instances. intuitively, if the predictions of different models are highly inconsistent on an instance, the instance may contain much noise. the larger deviation, the more uncertainty, and the less weight. this idea further increases the robustness of the selected training instances. we conduct extensive experiments on two public datasets. the experiment results clearly indicate that, by consciously considering multiple noisy data sources derived from very different augmentation methods, our approach is more effective than using any single source. our methods improve the state of the art (sota) by 3.05 and 4.24 percentage points on the two benchmark datasets, respectively. ## related work the cross-lingual spoken language understanding methods can be divided into two main categories: the model transfer methods and the data transfer methods. the model transfer methods build on pre-trained cross-lingual models to learn language agnostic representations, such as muse @xcite , @xcite , mbert @xcite , xlm @xcite , unicoder @xcite , and xlm-r @xcite . the english training data is applied to fine-tune the pre-trained models and then the fine-tuned models are directly applied to target languages @xcite @xcite @xcite . to better align embeddings between source and target languages, @xcite use domain-related word pairs and employ a latent variable model to cope with the variance of similar sentences across different languages. @xcite al. (2020b) and qin et al. (2020) use parallel word pairs to construct code-switching data for fine-tuning. their methods encourage the model to align similar words in different languages into the same space and attend to keywords. @xcite propose a regularization approach to align word-level and sentence-level representations across languages without any external resource. the data transfer methods construct pseudolabeled data in target languages. these methods usually employ machine translators to translate training instances in a source language into target languages and then apply alignment methods, such as attention weights @xcite , fastalign @xcite , or giza++ @xcite , to project slot labels to the target language side. the derived training instances are combined with the training data in the source language to fine-tune the pre-trained cross-lingual models. previous studies @xcite @xcite show that adding translated training data can significantly improve the model performance, especially on languages which are distant from the source language. the data generation approaches can also construct additional training data. some methods @xcite @xcite ) make slight changes to the original training instances through word replacement or paraphrases. more sophisticated methods generate training data through large-scale neural networks, such as generative adversarial networks @xcite , variational autoencoders @xcite @xcite , and pre-trained language models @xcite @xcite . ## method in this section, we define the problem and then propose our method. ## experiments in this section, we report our experiments on two benchmark datasets. ## case study we conduct case analysis of the instance relabeling results on the mtop dataset to examine the capability of our approach. we statistically analyze the differences between the original labels and the modified labels after relabeling stage. we find that our instance relabeling method effectively corrects wrong labels of the synthesized data, including intent label and slot label as shown in from the case study, we can see that the synthesized data indeed contains much noise and our relabeling strategy is able to greatly reduce the negative impact of the noise by correcting different types of label errors. ## conclusions in this paper, we propose a denoising training approach where multiple models trained from various augmented methods provide supervision signals to each other. extensive experimental results show that the proposed method outperforms the previous approaches, and can certainly alleviate the noisy label problem. our proposed method is independent of the backbone network (e.g., xlm-r model) and the task. as future work, we plan to investigate the performance of our method on different cross-lingual tasks.