Recycle Your W av2 V ec2 Codebook: A Speech Perceiver for Keyword Spotting (2022)

Abstract: Speech information in a pretrained wav2vec2.0 model is usually leveraged through its encoder, which has at least 95M parameters, being not so suitable for small footprint Keyword Spotting. In this work, we show an efficient way of profiting from wav2vec2.0’s linguistic knowledge, by recycling the phonetic information encoded in its latent codebook, which has been typically thrown away after pretraining. We do so by transferring the codebook as weights for the latent bottleneck of a Keyword Spotting Perceiver, thus initializing such model with phonetic embeddings already. The Perceiver design relies on cross-attention between these embeddings and input data to generate better representations. Our method delivers accuracy gains compared to random initialization, at no latency costs. Plus, we show that the phonetic embeddings can easily be downsampled with k-means clustering, speeding up inference in 3.5 times at only slight accuracy penalties.

## introduction keyword spotting (kws) has benefited recently from the adoption of the transformer architecture @xcite , as well as from recent advances in self-supervised learning proposals like wav2vec2.0 @xcite . transformers are capable of capturing information from broader contexts, going beyond the locality of convolutional neural networks (cnn) @xcite and avoiding the vanishing/exploding gradients from recurrent neural networks (rnn) @xcite . however, this comes at the quadratic cost of the self-attention mechanism @xcite , which is even more pronounced in high-dimensional modalities like speech or vision. kws models like the keyword spotting transformer (kwt) @xcite and the audio spectrogram transformer (ast) @xcite , minimize such cost by downsampling the spectrogram space into patches, inspired by the vision transformer (vit) @xcite proposal from computer vision. in parallel, approaches like wav2kws @xcite or the model from superb @xcite have successfully applied wav2vec2.0 to kws. during training, wav2vec2.0 learns a latent codebook that codifies phonetic information, using each code as a target for training a feature encoder. this codebook is typically thrown away after training, only keeping the encoder for downstream tasks like automatic speech recognition (asr) or kws. even though the encoder is capable of extracting rich features from raw waveforms, the size of it (at least 95m parameters for the base model) and its added latency time might discourage straightforward use for small footprint kws classifiers. in this short paper, we focus on exploring methods for recycling the phonetic information from the wav2vec2.0 latent codebook, showing that such information kickstarts the accuracy of a kws model at initialization and leads to a better convergence, at virtually no cost in terms of inference time or model size. the cornerstone of our proposal is a natural synergy that we have spotted between wav2vec2.0 and the recently proposed perceiver @xcite model. the latter's design relies on cross-attention between input data and a smaller latent bottleneck array, achieving smaller computational costs than pure self-attention over input data. we find that a pretrained wav2vec2.0 latent codebook can be used as an initialization for the perceiver's latent bottleneck array, which boosts the model's accuracy with respect to random weight initialization. furthermore, since many vectors from the wav2vec2.0 codebook contain similar phonetic information, we apply k-means clustering and average vectors belonging to the same clusters, yielding downsampled latent bottlenecks that provide faster inference at only slight accuracy costs. the focused contribution of this paper provides insight on efficient wav2vec2.0 transfer learning by latent codebook recycling, as well as showing the first application of the perceiver model to a specific speech task like kws, up to our knowledge. the latent array can be initialized randomly (kwp-base), or by transferring the weights from the latent codebook of a pretrained wav2vec2.0 model (kwp-w2v2). in this work, we recycle the latent weights of the wav2vec2.0 base model from the huggingface repo foot_0 . such codebook consists of @xmath0. since the complexity of cross-attention between latent and data arrays is o(m n ), we lose the efficiency gains from it with respect to self-attention over data array o(n 2 ), since o(m n ) = o(100x640) = (6.4x10 4 ) > o(n 2 ) = o(100 2 ) = (10 4 ). to address this, we study three ways to downsample this latent space to lower dimensions @xmath1, 160, 80, 40, 20], by (1) sampling vectors randomly, (2) average pooling contiguous vectors and (3) clustering with k-means method and averaging vectors from the same cluster. according to the wav2vec2.0 paper, most of the codebook latents model specific english phonemes, being some phonemes represented by many latents. for instance, the silence phoneme is represented by 22% of the codebook. being so, we expect k-means clustering to be the best downsampling method from the proposed ones, by clustering latents representing same or similar phonemes. simple average pooling without k-means clustering might conserve phonetic information, although we expect it to be suboptimal given that we cannot guarantee that contiguous vectors in the codebook correspond to similar phonetics, thus potentially mixing information from different phonemes. oppositely, random sampling guarantees that the individual information of each vector in the codebook is preserved, but as n gets lower many information is potentially lost, since most vectors are being dropped out. ## experiments we describe here the evaluations made for our keyword spotting perceiver proposal. first, we assess the effects of transferring the wav2vec2.0 latent codebook to the perceiver bottleneck at initialization. we also try different ways of downsampling this latent space, reporting accuracy comparisons between the baseline kwp-base model and the wav2vec2.0-initialized kwp-w2v2 model. afterwards, we keep the best performing downsampling algorithm for the following round of experiments, where we let the system train until convergence. we report accuracy, model size and inference time metrics for kwp-base and kwp-w2v2 models with different latent number variants @xmath2. training, validation and testing phases are done with the standard partitions from the google speech commands v2 dataset @xcite , obtaining the accuracy metrics from the 35commands task. all timing metrics are obtained by doing inference over 1-second waveforms on cpu, warming up for 10 forward passes and averaging the time for 150 forward passes. we use the adamw optimizer @xcite with a step learning rate scheduler, decreasing the learning each epoch by a gamma factor of 0.98, starting with an initial learning rate of 1e -4 . batch size is set to 32, training for a single epoch in the initialization experiments and for 400 epochs in the convergence experiments. for the latter ones, we pick the top-10 checkpoints with the highest validation accuracy, averaging their weights to obtain the final checkpoint, which we use for test accuracy measurements. we open-source the pytorch code foot_1 used for our experiments to the community. regarding data augmentation, we apply time shifting @xmath3.1 seconds with a probability of 60%. we also do resampling of the waveform signal between a [0.85, 1.15] fraction of the input sampling rate, which is set to 16 khz, with a probability of 100%. background noise is also added in a range of [5.0, 30.0] dbs and specaugment (park et al., 2019) is done with 2 time masks of 25 frame size and 2 frequency masks of 7 frames each. the latter both data augmentation methods are also applied with a 100% probability during training. even though, we relax the augmentation conditions for the shorter initialization experiments of a single epoch, to let the system learn a bit more in the early stages. time shifting and resampling probabilities are lowered to a 30%, specaugment to a 70% and background noise addition to an 80%. ## conclusion in this work, we have shown that phonetic information from the wav2vec2.0 latent codebook can be recycled, by transferring it to the latent bottleneck weights of a keyword spotting perceiver. accuracy gains are consistently significant with respect to random initialization of the latent bottleneck, both at early and late stages of training for the kws task. furthermore, we have studied easy-toapply downsampling techniques for compressing the latent codebook, like averaging k-means clusters, having sped up the inference time of the model up to 3.5 times, at only a 1% accuracy drop. we believe that our work motivates further research on efficient ways of profiting from the information in big self-supervised models like wav2vec2.0, as well as on applications for other tasks like asr, for instance.