Efficient Semi-supervised Consistency Training for Natural Language Understanding (2022)

Abstract: Manually labeled training data is expensive, noisy, and often scarce, such as when developing new features or localizing existing features for a new region. In cases where labeled data is limited but unlabeled data is abundant, semi-supervised learning methods such as consistency training can be used to improve model performance, by training models to output consistent predictions between original and augmented versions of unlabeled data. In this work, we explore different data augmentation methods for consistency training (CT) on Natural Language Understanding (NLU) domain classification (DC) in the limited labeled data regime. We explore three types of augmentation techniques (human paraphrasing, back-translation, and dropout) for unlabeled data and train DC models to jointly minimize both the supervised loss and the consistency loss on unlabeled data. Our results demonstrate that DC models trained with CT methods and dropout based augmentation on only 0.1% (2,998 instances) of labeled data with the remainder as unlabeled can achieve a top-1 relative accuracy reduction of 12.25% compared to fully supervised model trained with 100% of labeled data, outperforming fully supervised models trained on 10x that amount of labeled data. The dropout-based augmentation achieves similar performance compare to back-translation based augmentation with much less computational resources. This paves the way for applications of using large scale unlabeled data for semi-supervised learning in production NLU systems.

## introduction deep learning, especially transformer-based language models @xcite , have achieved state-of-the-art performance in many tasks and are widely used in nlu systems. a challenge in deep learning is that it often requires large amounts of labeled training data in order to reach a desirable performance level. this is especially a problem for nlu systems in commercial production as the cost of labeling data scales with the expanding number of supported features and languages. recent research in semi-supervised learning (ssl) demonstrated that it is possible to combine a small amount of labeled data and a large amount of unlabeled data to match or even outperform purely supervised learning @xcite . one of the most promising approaches in ssl is called consistency training @xcite @xcite . in short, consistency training is a technique that regularizes model predictions to be invariant to augmentations of unlabeled data. examples of augmentations include applying noise to input features @xcite or hidden states @xcite . in this paper, we experimented with consistency training in a major nlu task: domain classification (dc). we tested three different types of data augmentations: paraphrasing by user feedback, back-translation, and dropout. as a testbed for our approach, we applied our experiments to bert @xcite -based models using a real-world dataset collected from portuguese users of a voice-controlled assistant. we found that all three types of augmentations can be effectively used alongside consistency training to improve model performance compared to a baseline model trained without consistency training. for the scenario where labeled data was limited to only 0.1% of all available labeled data, the best top-1 accuracy, which was -9.14% compared to fully supervised model trained with 100% labeled data, was achieved by consistency training on data augmented using back-translation. if we use dropout-only augmentation, the relative top-1 accuracy change was -12.25%. lastly, we observed a relationship between the amount of labeled data used for training and the size of ct benefits, with larger benefits for smaller sets of labeled data. our results demonstrate the possibility of using consistency training to drastically reduce the amount of labeled data needed for an nlu system while retaining a reasonable accuracy. this can be done on large unlabeled datasets without using computationally expensive back-translation or financially costly human-authored augmentation. ## experiment we designed our experiments to explore the performance impact of incorporating consistency training using each type of data augmentation. we also investigated how performance changes as the amount of labeled data or unlabeled data used for training is varied. ## results here we present the results of our consistencytraining experiments and illustrate how model performance changed as we varied the underlying training data. ## conclusion with the aim of developing a strategy to efficiently leverage large amounts of unlabeled data in deployed nlu systems, we examined three different augmentation techniques for consistency training using real-world data. back-translation performed the best, dropout was slightly behind and paraphrase by human users was the worst-performing technique. from an operations perspective dropout is more favorable because it doesn't require any extra system resources and is quick to compute. paraphrasing by back-translation requires a machinetranslation model that can translate to an intermediate language and back. this adds extra cost and processing time for unlabeled data which scales linearly with the amount of unlabeled data. for industry-scale nlu applications with massive amounts of data, dropout-based consistency training can provide performance gains over purely supervised methods with minimal additional resource overhead. top-1 relative change dropout ct* -11.84% confidence @xmath0.6 -11.01% confidence @xmath1.3 -11.42% confidence @xmath2.37% tsa @xmath3.70% tsa @xmath4.69% tsa @xmath5.22% softmax @xmath6.7 -13.70% softmax @xmath7.9 -12.87% softmax @xmath8.94%