Revisiting Interpolation Augmentation for Speech-to-Text Generation (2024)

Abstract: Speech-to-text (S2T) generation systems frequently face challenges in low-resource scenarios, primarily due to the lack of extensive labeled datasets. One emerging solution is constructing virtual training samples by interpolating inputs and labels, which has notably enhanced system generalization in other domains. Despite its potential, this technique’s application in S2T tasks has remained under-explored. In this paper, we delve into the utility of interpolation augmentation, guided by several pivotal questions. Our findings reveal that employing an appropriate strategy in interpolation augmentation significantly enhances performance across diverse tasks, architectures, and data scales, offering a promising avenue for more robust S2T systems in resource-constrained settings.

## introduction recently, neural network-based end-to-end systems have achieved impressive improvements and become the de facto modeling method for speechto-text (s2t) generation tasks, such as automatic speech recognition (asr) @xcite and automatic speech translation (ast) @xcite . these deep learning models typically comprise millions or even billions of parameters and require vast amounts of training data to achieve state-of-the-art performance @xcite . for example, leading asr models demand thousands of hours of training data @xcite . however, the labeling of such extensive datasets leads to significant costs, and models trained on limited data are prone to overfitting, resulting in suboptimal generalization to unseen samples @xcite . to enhance generalization capabilities, data augmentation has become a key strategy @xcite . existing approaches in s2t can be broadly classified into two categories: online and offline augmentation. online methods, such as specaugment @xcite , enhance regularization by transforming the input representation during training. by introducing random noise into input features, these techniques have become standard in s2t tasks. offline methods, on the other hand, boost data diversity by creating large amounts of pseudo-data through original audio distortion @xcite or synthesis @xcite . though effective, these offline techniques are separate from the training process, often requiring additional steps and computational resources. this creates a demand for more efficient solutions. we resort to interpolation augmentation (ipa), also known as mixup, a notable method first introduced in image classification @xcite . ipa mitigates overfitting by constructing virtual samples through linear interpolation of both input features and labels from two randomly selected samples. this approach has achieved impressive success across diverse domains, including speech processing @xcite @xcite , natural language processing @xcite @xcite , and computer vision @xcite . in the specialized field of speech processing, preliminary studies have explored ipa in speech separation @xcite and classification tasks @xcite . however, its application in s2t tasks remains limited and largely unexplored @xcite @xcite . the existing work has not yet established clear guidelines on when and how ipa can be optimally leveraged in s2t tasks, leaving a substantial gap in our understanding and application of this promising technique. in this paper, we examine this question more closely, conducting a series of experiments to answer the following questions: q1 what is the appropriate interpolation strategy, and what distinctions arise between interpolating speech features and text embeddings? ( q2 how can ipa create an effective combination with existing augmentation techniques, such as the well-established method specaugment? ( §4) q3 are there specific issues in applying ipa to s2t tasks, and how can they be addressed? q4 how does ipa perform across various scenarios? ( §6) by probing these questions, we develop an effective ipa method that achieves consistent improvements across two s2t tasks (including asr and ast), various architectures (including encoderdecoder and encoder-ctc), and diverse data scales (ranging from librispeech 10h to 960h). ## experimental settings data augmentation methods typically demonstrate greater potential in low-resource scenarios. in light of this, we conduct analyses using the librispeech 100h asr dataset and subsequently apply our findings to various scenarios. we report results mainly on the test-clean and test-other sets. the average word error rate (wer) is calculated on the concatenation of all four subsets. various existing data augmentation techniques, such as specaugment and speed perturbation, have achieved excellent results. specaugment @xcite , the most widely employed method in s2t tasks, introduces random noise to the input features through time warping, frequency masking, and time masking. speed perturbation @xcite , on the other hand, commonly expands the dataset by generating three variations of raw audio with speed factors of 0.9, 1.0, and 1.1, facilitating its integration. in our work, the goal of ipa is to not only lead to isolate improvements but to also work orthogonally with these methods. therefore, we first examine scenarios without other augmentations and then explore the effects of their combination. in the field of s2t, common architectures encompass both encoder-decoder (enc-dec) and encoder-ctc (enc-ctc) designs. the enc-dec model consists of an encoder with 12 conformer layers and a decoder with 6 transformer layers, each containing 256 hidden units, 4 attention heads, and 2048 feed-forward sizes. connectionist temporal classification (ctc, @xcite multitask learning is applied on top of the encoder, introducing an additional loss with a weight of 0.3. the enc-ctc model can be viewed as a variant of the enc-dec model, containing only an 18-layer conformer encoder for comparable parameters of about 30m. it predicts the text purely through ctc, where the weight of the ctc loss is 1. we initially investigate the effects of ipa on the enc-dec model before extending the method to the enc-ctc model. more details about the datasets and model settings are described in appendix a. ## q1: choice of interpolation strategy in this section, we begin with an overview of the basic implementation of ipa. subsequently, we investigate the appropriate interpolation strategy tailored specifically for the field of s2t generation. ## q2: combination of augmentation techniques although the straightforward application of the ipa method has yielded noticeable improvements, our exploration seeks to combine it with existing data augmentation techniques. ## q3: resolution of specific issues while the current method does achieve stable effects, the enhancements are relatively modest. this section delves into further optimization by addressing the specific issues when employing aipa in s2t tasks. in the standard implementation, interpolated samples are given the dual responsibility of predicting two corresponding text sequences in both ctc and ce losses. however, this strategy might introduce a risk of ambiguity in the decision boundaries, potentially leading to an over-smoothed model. this risk is notably amplified during ctc learning, where the likelihood of a particular transcript x given the hidden representations h is obtained by summing over the probabilities of all feasible alignment paths φ(x) between the speech s and x: this implies that each representation is required to cater to a multiplicity of labels, which substantially complicates the ideal predicted distribution, making it challenging to converge and somewhat counter-intuitive. therefore, the design of appropriate training objectives for interpolated samples is pivotal. we propose constraint objective space (cos), which facilitates ctc learning by replacing the complex traversal with deterministic labels. rather than computing the best alignment by the model @xcite , we take the predicted distribution of the original samples as the objective of the interpolated samples for efficiency. specifically, we calculate the cos loss as follows: in this framework, the original samples act as a teacher, guiding the more accessible learning process of the interpolated student @xcite . this distribution offers detailed information across the entire vocabulary, and importantly, the training objective becomes more deterministic, thereby simplifying the learning process. the final design of aipa with cos is depicted in figure 3 . similarly, this strategy can be extended to the cross-entropy (ce) loss, denoted by l cos ce . the final training objective thus takes the form: we present the results in ## q4: effect under various scenarios we have obtained numerous valuable insights from the ablation studies conducted on the librispeech 100h dataset. we now extend the application of the aforementioned settings to a broader array of scenarios. ## conclusion in this paper, we develop a comprehensive exploration of the interpolation augmentation (ipa) method's application in s2t generation. our findings provide actionable insights for the effective application of ipa in s2t: (1) utilizing ipa alone may not surpass the effectiveness of specaugment; a careful combination of both lies in mitigating distribution shift and preserving the learning of original data distribution. (2) defining an appropriate training objective for interpolated samples is of paramount importance. (3) ipa demonstrates particular compatibility with the enc-ctc model. (4) the appropriate ipa strategy significantly enhances performance across diverse scenarios.