Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data (2023)

Abstract: Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient parallel data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing in this way results in additional gains on top of the vanilla pseudo-labeling setup providing a total improvement of up to 0.4% absolute WER and 2.1 BLEU points for En–De and 0.6% absolute WER and 2.2 BLEU points for En–Zh.

## introduction semi-supervised learning methods have been a cornerstone in addressing annotated data scarcity by taking advantage of and incorporating the relatively larger amounts of unlabeled 1 data in the training process. self-training is a relatively early instance of such methods @xcite . conceptually, self-training is simple: first, a base model is trained using limited labeled data. the base model is then used to predict labels for the unlabeled data. the generated labels are termed "pseudo-labels" (pls) to signify their predicted nature, as opposed to gold supervised data. finally, the pseudo-labels are combined with the initial seed supervised data to train a new model, and this process is repeated until no further improvement in performance is observed. self-training, or pseudo-labeling interchangeably, has been shown to be effective to improve upon fully supervised baselines in low-resource settings for several sequence-to-sequence (seq2seq) tasks, such as machine translation (mt) @xcite @xcite , endto-end speech recognition (asr) @xcite @xcite , end-to-end speech translation (st) @xcite , and more recently speech-tospeech translation @xcite . in this work, we study pseudo-labeling for a recently proposed new setup, joint speech transcription and translation (stt) @xcite : a setup that is of interest in use cases where both the transcript and translation of a speech signal are returned to the user. as we describe in detail later in §2.1, the fully supervised data for modeling end-to-end joint transcription and translation is triples of form (s, tc, tl) where s is the speech signal, tc is the transcript, and tl is the translation. as that is especially costly to come by, stt also seems to have the potential to benefit from pseudo-labeling. our investigations show that while pseudolabeling (pl) is indeed helpful, the quality of pseudo-labels that bring about the benefits is subpar. upon inspecting the supervised and unsupervised sets, that proves to be not surprising: with limited amounts of supervised data, it is likely that the supervised and unsupervised sets differ in domain, impacting the quality of pseudo-labels. specifically, in our case, we identify two causes leading to domain mismatch with out-of-distribution unlabeled data: difference between the sequence length ranges and vocabulary sets of the supervised and unsupervised sets. in this work, we ask if we can specifically counteract the domain mismatch to reach a set of pseudo-labels of higher quality, and if that higher quality, in turn, translates into a better overall performance of pseudo-labeling. first, we propose pls filtering based on simple data-centric criteria inspired by @xcite . while pls filtering is a common component of pl algorithms, it is usually based on the model prediction scores @xcite @xcite , which may not directly target the identified domain mismatch aspects, e.g., different sequence length ranges, as our proposed filtering does. second, we propose augmenting the supervised data by concatenating randomly-picked samples to create new ones and adding them to the supervised set. these two are essentially different in nature: while filtering increases the overall quality by removing samples with pls that are likely to be faulty, augmentation does so by extending the supervised set and generating better labels in the first place. our results confirm that indeed this distinction in nature gets reflected in different ways filtering and augmentation improve the performance of pseudo-labeling. the outline of this paper is as follows. we provide some background in §2 and detail the experimental setup in §3. then, in §4, we report and discuss the results from vanilla pseudo-labeling, the observation of domain mismatch, and the gains brought about by filtering and augmentation. our contributions are: 1) we specifically focus on pl in the face of domain mismatch between the supervised and unsupervised sets; 2) we investigate the mitigation of the effect of domain mismatch through two approaches: pls filtering and augmentation by concatenation and demonstrate how they improve pl in different ways. these approaches can be repurposed wherever pl is considered as a solution; 3) we apply pl modified with those approaches specifically to a novel setup, joint speech transcription and translation, and report gains on top of the vanilla pl for stt. ## background our work studies a pseudo-labeling solution for end-to-end joint speech transcription and translation. in this section, we provide the background for these two components involved in the study, namely speech transcription and translation and pseudo-labeling. ## results and discussion we present our results in this section in the following order: §4.1 establishes vanilla pseudo-labeling performance, which leads to our analysis of the domain mismatch between the supervised and unsupervised sets. §4.2 and §4.3 then describe the two categories of remedies we devise to mitigate the effect of domain discrepancies on pseudo-labeling. as mentioned in §2.2, this is all using the best setting we were able to establish during our pilot experiments: at each pseudo-labeling round, we 1) label only the unsupervised data, and 2) finetune the existing checkpoint on the combination of supervised and pseudo-labeled data. we conduct our pilot experiments on en-de. we were able to confirm that the aforementioned setting consistently beats the rest over several rounds of pseudolabeling. figure 1 illustrates the lead of the best setting over others in the last round of our experiments. the same pattern holds across all rounds. ## related work the two paradigms often considered in lowresource data scenarios are self-training and pre-training. self-training, or pseudo-labeling, has long been studied for a variety of seq2seq tasks @xcite @xcite @xcite @xcite . regarding the relationship between pretraining and self-training, @xcite and @xcite show that self-training and unsupervised pretraining are complimentary and can be combined to boost performance on speech recognition and speech translation, respectively. in the case of supervised pretraining, however, @xcite show in the vision domain that as the size of the labeled data available grows, self-training remains helpful, whereas the benefits of supervised pretraining start to diminish. for applying self-training to the unvisited setup of joint speech transcription and translation (sperber et al., 2020), we focus on domain mismatch, a matter which can get overlooked when gains from vanilla pseudo-labeling are observed. as solutions, we study pseudo-label filtering and augmentation by concatenation. in contrast to conventional filtering, which relies on normalized model confidence scores @xcite , or recently, the agreement between several forward passes of the model run with dropout @xcite , we define and use data-centric factors that are attuned to the domain differences we observe and directly target them. concatenation as an effective augmentation method has been studied in the context of machine translation @xcite @xcite and speech-to-text @xcite . in our case, we use it to expose our base model to sequences of higher length to improve the quality of generated pseudo-labels. ## conclusion we study pseudo-labeling for joint speech transcription and translation. we show that while vanilla pseudo-labeling is helpful, additional improvements are obtained by addressing the low quality of generated pseudo-labels due to domain mismatch between the supervised and unsupervised sets. we find that our proposed solutions help in two different ways, as they are in distinct nature: pseudo-label filtering, which discards low-quality labels, is mostly helpful by expediting gains in earlier rounds, especially for transcriptions. augmentation by concatenation, on the other hand, does not discard any of the labels. as a result, it is able to maintain an edge over vanilla pseudo-labeling in the late rounds as well.