M eta M ix S peech: Meta Task Augmentation for Low-Resource Speech Recognition (2025)

Abstract: Meta-learning has proven to be a powerful paradigm for effectively improving the performance of low-resource speech recognition by learning generalizable knowledge across multiple tasks. However, multilingual meta learning also faces challenges such as task overfitting and learner overfitting, thereby reducing its ability to generalize to new tasks. To address these issues, we augment the meta-training task with “more data” during both training and evaluation phases. Concretely, we propose an interpolation-based task augmentation method called MetaMixSpeech, which includes both support augmentation and query augmentation. MetaMixSpeech enhances task diversity by linearly combining perturbed features from the support and query sets and performing the same linear interpolation on their corresponding losses. Experimental results on the FLEURS and Common Voice datasets demonstrate that MetaMixSpeech achieves a 6.35 % improvement in Word Error Rate (WER) compared to meta-learning approaches, effectively mitigating the overfitting problem and showcasing superior generalization across diverse datasets and language families.

## introduction automatic speech recognition (asr) has revolutionized various aspects of people's lives, delivering remarkable success in several widely spoken languages @xcite . however, there are more than 7,000 languages in the world, and it is estimated that 94% of them are spoken by fewer than 1 million people @xcite . these languages are categorized as low-resource languages due to limited labeled data availability. compared with commonly spoken languages, low-resource languages lack the necessary transcribed speech data, pronunciation dictionaries, and language scripts, making it challenging to build * corresponding author a usable speech recognition system. despite these challenges, it is imperative to research efforts on low-resource languages and preserve their cultural heritage. a lot of progress has been made in low-resource speech recognition, which includes efforts like data augmentation @xcite , transfer learning @xcite and multilingual learning @xcite . recently, a novel paradigm known as meta-learning has been investigated for its potential in enhancing low-resource speech recognition @xcite @xcite . among these, gradient-based metalearning algorithms @xcite have gained widespread adoption in the speech domain due to their flexibility and generalization capabilities. gradient-based meta-learning algorithms train models at two levels. in the inner loop, the model undergoes a gradient descent on a small amount of training data (the support set), enabling the task learner to adapt to the task. in the outer loop, the adapted model is evaluated on the query set of the task, and the evaluation loss is optimized to im-prove the model's generalization capability across the meta-training tasks. the key to meta-learning lies in optimizing the model initialization to maximize its generalizability, measured by the adapted model's performance on the query set. however, when data is limited, meta learning face the risks of memory overfitting @xcite and learner overfitting @xcite , which have been analyzed in computer vision applications. unlike computer vision, multilingual meta-learning does not encounter the issue of memory overfitting due to several reasons. first, the labels of sequence generation tasks are not fixed, ensuring tasks' exclusivity. second, the number of languages per episode is fixed, preventing different categories' learning within an episode. however, we analyze that multilingual metalearning still faces two types of overfitting risks, as shown in figure .1: (1) task-overfitting, where repeated learning of the same languages in each episode causes the task learner ϕ to overfit the support set and fails to generalize to the query set. improving meta-generalization is challenging. in the field of image processing, methods such as meta-regularizers @xcite and constrained search spaces @xcite , as well as active data augmentation @xcite , have been employed to improve meta-generalization. compared to regularizers that restrict the flexibility of the inner loop, task augmentation methods @xcite are more flexible and effective. for instance, yao et al. @xcite proposed a task augmentation method that improves reliance on the support set by mixing it with the query set, thereby mitigating memory overfitting. however, this approach is not suitable for multilingual meta-learning for not suffering from the problem of memory overfitting. to address the risks of task overfitting and learner overfitting in multilingual meta-learning, we propose a flexible and effective task augmentation method called metamixspeech. as shown in figure .1, this method includes support augmentation and query augmentation, which conduct feature space manifold expansion by linearly combining perturbed features or hidden representations from the support and query sets and performing the same linear interpolation on their corresponding losses. it can effectively mitigate the two types of meta-overfitting and enhance the generalization capability of meta-learning methods. the main contributions of this paper are: ## conclusion in this paper, we first analyze two overfitting issues in multilingual meta-learning: task overfitting and learner overfitting. therefore, we introduce metamixspeech, an innovative interpolation-based task augmentation method that enhances task diversity through support augmentation and query augmentation to overcome these problem. experimental results on the fleurs and common voice datasets demonstrate that metamixspeech is highly effective, achieving superior performance and generalization across multiple language families.