Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020)

Abstract: Training objectives based on predictive coding have recently been shown to be very effective at learning meaningful representations from unlabeled speech. One example is Autoregressive Predictive Coding (Chung et al., 2019), which trains an autoregressive RNN to generate an unseen future frame given a context such as recent past frames. The basic hypothesis of these approaches is that hidden states that can accurately predict future frames are a useful representation for many downstream tasks. In this paper we extend this hypothesis and aim to enrich the information encoded in the hidden states by training the model to make more accurate future predictions. We propose an auxiliary objective that serves as a regularization to improve generalization of the future frame prediction task. Experimental results on phonetic classification, speech recognition, and speech translation not only support the hypothesis, but also demonstrate the effectiveness of our approach in learning representations that contain richer phonetic content.

## introduction unsupervised speech representation learning, which aims to learn a function that transforms surface features, such as audio waveforms or spectrograms, to higher-level representations using only unlabeled speech, has received great attention recently @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . a large portion of these approaches leverage self-supervised training, where the learning target is generated from the input itself, and thus can train a model in a supervised manner. @xcite propose a method called autoregressive predictive coding (apc), which trains an rnn to predict a future frame that is n steps ahead of the current position given a context such as the past frames. the training target can be easily generated by right-shifting the input by n steps. their intuition is that the model is required to produce a good summarization of the past and encode such knowledge in the hidden states so as to accomplish the objective. after training, the rnn hidden states are taken as the learned representations, and are shown to contain speech information such as phonetic and speaker content that are useful in a variety of speech tasks @xcite . following their intuition, in this work we aim to improve the generalization of the future frame prediction task by adding an auxiliary objective that serves as a regularization. we empirically demonstrate the effectiveness of our approach in making more accurate future predictions, and confirm such improvement leads to a representation that contains richer phonetic content. the rest of the paper is organized as follows. we start with a brief review of apc in section 2. we then introduce our approach in section 3. experiments and analysis are presented in section 4, followed by our conclusions in section 5. ## autoregressive predictive coding given a context of a speech signal represented as a sequence of acoustic feature vectors (x 1 , x 2 , . . . , x t ), the objective of autoregressive predictive coding (apc) is to use the context to infer a future frame x t+n that is n steps ahead of x t . let @xmath0, x 2 , . . . , x n ) denote a full utterance, where n is the sequence length, apc incorporates an rnn to process each frame x t sequentially and update its hidden state h t accordingly. for @xmath1, . . . , n -n, the rnn produces figure 1 : overview of our method. l f is the original apc objective that aims to predict x t+n given a context (x 1 , x 2 , . . . , x t ) with an autoregressive rnn. our method first samples an anchor position, assuming it is time step t. next, we build an auxiliary loss l r that computes l f of a past sequence (x t-s , x t-s+1 , . . . , x t-s+ℓ-1 ) (see section 3.1 for definitions of s and ℓ), using an auxiliary rnn (dotted line area). in this example, (n, s, ℓ) = (1, 4, 3). in practice, we can sample multiple anchor positions, and averaging over all of them gives us the final l r . an output y @xmath2, where w is an affinity matrix that maps h t back to the dimensionality of x t . the model is trained by minimizing the frame-wise l1 loss between the predicted sequence (y 1 , y 2 , . . . , y n -n ) and the target sequence (x 1+n , x 2+n , . . . , x n ): when @xmath3, one can view apc as an acoustic version of neural lm (nlm) @xcite by thinking of each acoustic frame as a token embedding, as they both use a recurrent encoder and aim to predict information about the future. a major difference between nlm and apc is that nlm infers tokens from a closed set, while apc predicts frames of real values. once an apc model is trained, given an utterance (x 1 , x 2 , . . . , x n ), we follow @xcite and take the output of the last rnn layer (h 1 , h 2 , . . . , h n ) as its extracted features. ## proposed methodology our goal is to make apc's prediction of x t+n given h t more accurate. in section 4 we will show this leads to a representation that contains richer phonetic content. ## experiments we demonstrate the effectiveness of l r in helping optimize l f , and investigate how the improvement is reflected in the learned representations. ## conclusions we improve the generalization of autoregressive predictive coding by multi-target training of fu-1 according to chung and glass (2020), when using a transformer architecture @xcite as the autoregressive model, representations learned with l f can achieve a wer of 13.7 on asr and a bleu score of 14.3 on st.