Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations (2024)

Abstract: Acoustic word embeddings (AWEs) are vector representations of spoken words. An effective method for obtaining AWEs is the Correspondence Auto-Encoder (CAE). In the past, the CAE method has been associated with traditional MFCC features. Representations obtained from self-supervised learning (SSL)-based speech models such as HuBERT, Wav2vec2, etc., are outperforming MFCC in many downstream tasks. However, they have not been well studied in the context of learning AWEs. This work explores the effectiveness of CAE with SSL-based speech representations to obtain improved AWEs. Additionally, the capabilities of SSL-based speech models are explored in cross-lingual scenarios for obtaining AWEs. Experiments are conducted on five languages: Polish, Portuguese, Spanish, French, and English. HuBERT-based CAE model achieves the best results for word discrimination in all languages, despite HuBERT being pre-trained on English only. Also, the HuBERT-based CAE model works well in cross-lingual settings. It outperforms MFCC-based CAE models trained on the target languages when trained on one source language and tested on target languages.

## introduction self-supervised learning (ssl)-based speech representations are becoming popular in speech processing and producing state-of-the-art results in many downstream tasks such as automatic speech recognition, speaker verification, keyword spotting, voice conversion, etc @xcite . these representations are obtained using self-supervised learning on large amounts of unlabelled speech data. wav2vec2 @xcite , hubert @xcite , and wavlm @xcite are a few examples of such ssl-based speech models. however, representations obtained from these models have not been extensively explored in the context of learning acoustic word embeddings (awes). awes are fixed-dimensional vector representations of spoken words that find applications in various downstream tasks, such as queryby-example search @xcite @xcite , keyword spotting @xcite , providing clues for human lexical processing @xcite , hate speech detection in low resource settings @xcite , etc. recently, the work @xcite proposed extracting awes from ssl-based speech representations using a mean pooling mechanism. the authors suggest that ssl-based speech representations, which are contextualized, can be effectively converted into awes using a straightforward pooling mechanism. on the other hand, correspondence auto-encoder (cae) based training strategies for awes @xcite using mfcc @xcite features are shown to be promising in the literature. correspondence training involves an auto-encoder where a spoken word serves as the input to the encoder, and the target output of the decoder is a different instance of the same spoken word. this approach helps to preserve acoustic-phonetic information while filtering out unnecessary details such as speaker, acoustic environment, and duration, etc. both encoder and decoder are typically recurrent neural networks @xcite . more details about the model will be presented in sec. 2 and sec. 4.2. correspondence training has also been explored in the work @xcite to improve content representations of ssl-based speech models. the work @xcite uses a correspondence transformer encoder (cte) for obtaining robust awes, trained from scratch and a largescale unlabelled speech corpus. in contrast, in this work, pre-trained ssl speech models are coupled with a simple rnn based auto-encoder for correspondence training to obtain robust awes. this work attempts to use the correspondence training of auto-encoder to obtain the awes by leveraging ssl-based speech representations instead of mfcc features as input features to the cae model. further, cross-lingual capabilities are also examined for ssl-based awes trained with cae method. the ssl models (hubert, wav2vec2, and wavlm) used in this work are pre-trained on english data. however, it has been demonstrated that these models work well as feature extractors for the all the languages considered in this study. the performances on the word-discrimination task for all the languages (polish, portuguese, spanish, and french) are as good as on the english language (sec. 5). a detailed analysis is also conducted to assess the importance of contextual information in spoken words by comparing feature extraction with and without context. for this work, we obtain spoken words for all five languages from the subset of multilingual librispeech (mls) @xcite dataset. the derived dataset consists of five languages (polish, portuguese, spanish, french, english) with start and end timestamps of spoken words foot_0 . we chose mls dataset for our experiments as many previous works @xcite @xcite @xcite on awes rely on gobalphone @xcite dataset, which is not freely available. the main contributions of this work are as follows: the rest of the paper is as follows: sec. 2 describes the correspondence auto-encoder methodology to obtain awes; sec. 3 describes the data preparation and data statistics; sec. 4 describes the details of the experiments; sec. 5 describes the results and analysis; sec. 6 concludes the work with possible future directions. sec. 7 describes the limitations of the work. target output: @xmath0,xꞌ 2 ,..,xꞌ n } figure 1 : cae-rnn training setup for extracting awes @xcite . ## methodology correspondence auto-encoder is trained with input as a spoken word and target output as a different instance of the same spoken word. typically, recurrent neural network (rnn) based encoder and decoder are used, hence the model is referred to as cae-rnn. the rationale behind this training method is that cae-rnn will preserve only the acoustic-phonetic information and filter out the other unnecessary information factors such as speaker, duration, acoustic environment, etc @xcite . fig. 1 shows the cae-rnn model setup. the input to the enc is a sequence of acoustic feature vectors (@xmath1, x 2 , ..., x m ) of a spoken word. the target output is the sequence of acoustic feature vectors of the different instance of the same spoken word (x ′ = x ′ 1 , x ′ 2 , ..., x ′ n ). the encoder produces the awe (e) of the spoken word x, which is then fed to the decoder to reconstruct x ′ . the output of the decoder is represented as @xmath2, y 2 , ..., y n . the mean squared loss function for a single training pair (x, x ′ ) can be described as following: ## data preparation the multilingual librispeech (mls) dataset @xcite is utilized to obtain spoken words. five languages, namely polish, portuguese, spanish, french, and english, are selected from mls. for each language, approximately 25,000 utterances are selected for the training set, 500 for the development set, and 500 for the test set. these selected utterances are force-aligned to obtain the spoken word boundaries using the montreal forced aligner toolkit @xcite . only spoken words with a duration of 0.5 seconds or longer are included in the derived dataset, following the standard practice in the literature @xcite . spoken words with a frequency greater than 50 or less than 5 are excluded from the derived dataset. ## experimental setup experiments are conducted on all the five languages with ssl-based speech representations as input features extracted from wav2vec2, hubert, and wavlm. experiments are also conducted with mfcc as input features. first, the feature extraction methods for various ssl-based speech representations and mfccs are described. then, the configuration of the cae-rnn model is explained, along with the mean pooling baseline @xcite and the ae-rnn method (without correspondence training), for comparison. next, the word discrimination task is described, which is used for evaluating the quality of the extracted awes. finally, the training details of the cae-rnn and other models are provided. ## results and analysis ## conclusions and future work it has been demonstrated that ssl-based speech representations with cae-rnn models outperform mean pooling and ae-rnn models across all languages. they also outperform mfcc-based models. among all the ssl models, hubert performs the best when used as input for the cae-rnn model, outperforming models such as wav2vec2 and wavlm. notably, despite being pretrained on english data, the ssl models exhibit excellent performance on other languages, showcasing their cross-lingual generalization capability for awe extraction. furthermore, quantitative analysis reveals that incorporating context information of the spoken word leads to more robust awes. the hubert-based cae-rnn model trained on english language and tested on other target languages outperforms the mean pooling method and the cae-rnn model trained on the target language using mfcc features. this 'zero-shot' method to obtain robust awes for the target language can be useful in applications for low-resource languages @xcite ). an analysis was also conducted to show that the cae-rnn model effectively captures the order of letters in a word. in future work, experiments will be conducted with the "large" variation of ssl models, as well as multilingual pre-trained ssl models such as wav2vec2-xlsr @xcite . additionally, an interesting experiment would involve training a single universal awe model on all languages and comparing its performance with language-specific awe models. further research will focus on measuring the performance gains of ssl-based cae-rnn models on downstream tasks such as query-by-example search @xcite @xcite and keyword spotting @xcite .