T ask M ix: Data Augmentation for Meta-Learning of Spoken Intent Understanding (2022)

Abstract: Meta-Learning has emerged as a research direction to better transfer knowledge from related tasks to unseen but related tasks. However, Meta-Learning requires many training tasks to learn representations that transfer well to unseen tasks; otherwise, it leads to overfitting, and the performance degenerates to worse than Multi-task Learning. We show that a state-of-the-art data augmentation method worsens this problem of overfitting when the task diversity is low. We propose a simple method, TaskMix, which synthesizes new tasks by linearly interpolating existing tasks. We compare TaskMix against many baselines on an in-house multilingual intent classification dataset of N-Best ASR hypotheses derived from real-life human-machine telephony utterances and two datasets derived from MTOP. We show that TaskMix outperforms baselines, alleviates overfitting when task diversity is low, and does not degrade performance even when it is high.

## introduction deep learning has seen a meteoric rise in speech and language related applications, leading to largescale applications of voice-bots, voice assistants, chatbots, etc., which aim to automate mundane tasks such as answering users' queries either in spoken or textual modality. in many applications, users tend to code-switch or use borrowed words from other languages. a model trained for a particular language will not understand these borrowed words, and hence language-specific models are undesirable in such scenarios. on the other hand, a multilingual model can understand and reason what the user is speaking. due to the scale of the applications, data captured from various sources have different distributions or have different use-cases. recently, meta-learning has emerged as a novel research direction that aims to leverage knowledge from diverse sets of data to learn a transferable initialization so that a low amount of training data is required to adapt to new datasets or tasks. however, meta-learning requires a large number of training tasks, or else the model would overfit to the training tasks and would not generalize well to new tasks @xcite . in this work, we propose a novel data augmentation method, taskmix for meta-learning problems, inspired by mixup @xcite . we investigate our proposed method against baselines such as metamix @xcite , multitask-learning, and vanilla transfer learning for multi-domain multi-lingual spoken intent classification. ## preliminaries in this section we describe the problem formulation and the prior work which we built upon. ## experiments this section presents empirical results on two multilingual and multi-domain datasets. for choice of hyperparamters and other experimental details, please refer to the appendix. ## conclusion in this paper, we propose a novel dataaugmentation method, taskmix, to alleviate the problem of overfitting in meta-learning datasets when the task diversity is too low. through experiments on two multilingual, multi-domain intent classification datasets, metamix could worsen the overfitting problem when the task diversity is low, whereas taskmix is beneficial in such cases.