Learning Robust and Multilingual Speech Representations (2020)

Abstract: Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages.

## introduction the input representation of machine learning model strongly determines the difficulty faced by the learning algorithm, how much data the learner will require to find a good solution, and whether the learner generalizes out of sample and out of the domain of the training data. representations (or features) that encode relevant information about data enable models to achieve good performance on downstream tasks, while representations that are invariant to factors that are not relevant to downstream tasks can further improve generalization. traditionally, many invariances were hard-coded in feature extraction methods. for example, in image representations, geometric and photometric invariance has been investigated @xcite . for acoustic representations, standard mfcc features are sensitive to additive noise and many modifications have been proposed to overcome those limitations @xcite . recently, unsupervised representation learning algorithms have shown significant improvements at learning representations that correlate well with phonetic structure @xcite and improving downstream speech recognition performance @xcite . most of this work focused on learning representations from read english speech (from the librispeech and librivox datasets) and evaluating the features when used to recognize speech in a rather similar domain (read english text). however, this approach to evaluation fails to test for the invariances that we would like good speech representations to have: robustness to domain shifts and transferability to other languages. in our experiments we learn representations from 8000 hours of diverse and noisy speech, using an extended version of contrastive predictive coding model: bidirectional predictive models with dense residual connections ( §2- §4), and evaluate the robustness and transferability of our representations by estimating how invariant they are to domain and language shifts. to do so, an asr model is trained using our representations on one dataset but evaluated on the test sets of other datasets. in this experiment, we find that the representations derived from the large pretraining dataset lead the asr model to be much more robust to domain shifts, compared to both log filterbank features as well as to pretraining just on librispeech. we also train asr models on 25 languages, including low-resource languages (e.g. amharic, fongbe, swahili, wolof), and show that our representations significantly outperform both standard features and those pretrained only on clean english data in the language transfer setup. in summary, we confirm several increasingly common patterns that may be discerned in the literature on unsupervised representation learning, across a variety of modalities. first, scale matters: good representation learning requires a large amount of data. second, unsupervised representations consistently improve robustness on downstream tasks. and finally, representations learned from multilingual data can transfer across many languages. contrastive predictive coding (van den oord et al., 2018, cpc) is a mutual information maximization method that has been successfully applied to many modalities such as images and speech @xcite . the objective is designed to extract features that allow the model to make long-term predictions about future observations. this is done by maximizing the mutual information of these features with those extracted from future timesteps. the intuition is that the representations capture different levels of structure dependent on how far ahead the model predicts. for example, if the model only predicts a few steps ahead, the resulting representations can capture local structures. on the other hand, if the model predicts further in the future, the representations will need to infer "slow features" @xcite ; more global structures such as phonemes, words and utterances in speech. the overall unsupervised learning process is visualized in figure 1 . given a raw audio signal of length l (@xmath0, x 2 , . . . , x l , x @xmath1, a function g enc encodes the audio signals into vector representations (@xmath2, z 2 . . . , z m , @xmath3. next, an autoregressive function g ar , such as a recurrent neural network, summarizes the past representations and produces context vectors (@xmath4, c 2 . . . , c m , @xmath5. the representations are learned to maximize mutual information between context vectors (c t ) and future latent representations (z + k) as follows: since the mutual information is not tractable for high dimensional data, it is common to use a lower-bound on the mutual information such as infonce @xcite which is a loss function based on noise contrastive estimation @xcite . given a set @xmath6, . . . z n } which contains one positive sample from p(z t+k |c t ) and n -1 negative samples from a "noise" distribution p(z), the approximated lower-bound is written as: the loss function we maximize is a sum of the infonce loss for each step, l @xmath7. ## methods in this section, we describe our models and objectives for unsupervised representation learning and downstream speech recognition. first, an acoustic feature extractor is trained with a bidirectional variant of contrastive predictive coding on an unlabeled audio dataset. next, the parameters of this model are frozen and its output representations are used as input to train various speech recognition models, potentially on a different or smaller labeled dataset (figure 1 ). ## conclusion we have introduced an unsupervised speech representation learning method that discovers acoustic representations from up to 8000 hours of diverse and noisy speech data. we have shown, for the first time, that such pretrained representations lead speech recognition systems to be robust to domain shifts compared to standard acoustic representations, and compared to representations trained on smaller and more domain-narrow pretraining datasets. these representations were evaluated on a standard speech recognition setup where the models are trained and evaluated on in-domain data and also on transfer tasks where the models are evaluated on out-of-domain data. we obtained consistent improvements on 25 phonetically diverse languages including tonal and low-resource languages. this suggests we are making progress toward models that implicitly discover phonetic structure from large-scale unlabelled audio signals.