Quantifying Language Variation Acoustically with Few Resources (2022)

Abstract: Deep acoustic models represent linguistic information based on massive amounts of data. Unfortunately, for regional languages and dialects such resources are mostly not available. However, deep acoustic models might have learned linguistic information that transfers to low-resource languages. In this study, we evaluate whether this is the case through the task of distinguishing low-resource (Dutch) regional varieties. By extracting embeddings from the hidden layers of various wav2vec 2.0 models (including new models which are pre-trained and/or fine-tuned on Dutch) and using dynamic time warping, we compute pairwise pronunciation differences averaged over 10 words for over 100 individual dialects from four (regional) languages. We then cluster the resulting difference matrix in four groups and compare these to a gold standard, and a partitioning on the basis of comparing phonetic transcriptions. Our results show that acoustic models outperform the (traditional) transcription-based approach without requiring phonetic transcriptions, with the best performance achieved by the multilingual XLSR-53 model fine-tuned on Dutch. On the basis of only six seconds of speech, the resulting clustering closely matches the gold standard.

## introduction deep acoustic models have improved automatic speech recognition (asr) substantially in recent years @xcite @xcite . these models represent linguistic information based on massive amounts of data. while these models are generally evaluated on asr benchmarks, few studies have addressed what kind of linguistic information is represented by them. the work of @xcite examined information represented by the wav2vec 2.0 model @xcite across the various transformer layers. they showed that different layers encode different types of linguistic information. specifically, the initial layers appeared to be most similar to the input speech features, whereas the middle layers mostly encoded contextual information. the final layers again turned out to be similar to the input speech features. however, the representations of the final layers changed when the model was finetuned, likely because task-specific information was learned. in addition, @xcite investigated several deep acoustic models using phonetic probing tasks, and found that representations from these models capture information useful for distinguishing english phones. importantly, these deep acoustic models were better able to distinguish english phones than using conventional mfcc or filterbank features. although they evaluated the transferability of deep acoustic representations across several domains, it remains unclear whether these models learned information that transfers to other languages. this is, however, important when working on more inclusive speech technology. especially when resources for training these models are lacking, such as for regional languages and dialects. in this paper, we therefore investigate if hidden layers of deep acoustic models incorporate fine-grained information, which can be used to represent differences between, and in turn distinguish, regional language varieties. past work on investigating language variation has often been based on computing pronunciation distances that rely on phonetically transcribed speech @xcite @xcite . these (edit) distances have been found to match perceptual judgements of similarity well @xcite . however, transcribing speech phonetically is time-consuming and prone to errors @xcite . while automatic approaches for computing phonetic transcriptions exist (e.g., li et al. 2020), they produce lower quality phonetic transcriptions than human transcribers do. additionally, (discrete) phonetic transcriptions do not capture all (continuous) aspects of human speech @xcite . to mitigate these shortcomings, acoustic approaches have been developed for investigating language variation @xcite @xcite . however, these studies either exclusively focused on the vowels (ignoring differences in the consonants), or were negatively influenced by non-linguistic variation in the speech signal. recently, @xcite found that representations from the hidden layers of pre-trained and fine-tuned wav2vec 2.0 (large) models are suitable to represent language variation. they showed that these representations capture linguistic information that is not represented by phonetic transcriptions, while being less sensitive to non-linguistic variation in the speech signal. furthermore, this approach seems to provide a better match to human perceptual judgements than phonetic transcriptionbased approaches. to investigate if wav2vec 2.0 acoustic models (including newly trained dutch models) learn fine-grained linguistic information that can transfer to regional languages and dialects, we will assess whether or not regional languages and dialects spoken in the netherlands can be distinguished using these models. our code and newly trained models are publicly available. foot_0 ## dataset we use dutch dialect pronunciation recordings from the goeman-taeldeman-van reenen-project @xcite . audio recordings of hundreds of words were obtained (and manually phonetically transcribed) in the 1980s and 1990s and are available for 613 dialect varieties in the netherlands and belgium. unfortunately, the hourlong audio recordings were not segmented, and the metadata with the time stamps we use to extract the audio containing individual word pronunciations were only partially available. in total, therefore, we extract the acoustic recordings (judged to be of sufficient quality) for 10 words (armen: 'arms', deeg: 'dough', draden: 'wires', duiven: 'pigeons', naalden: 'needles', ogen: 'eyes', pijpen: 'pipes', tangen: 'pliers', volk: 'people', vuur: 'fire') pro-nounced in 106 locations in the netherlands. on average, the duration of these 10 words is only 6.3 seconds for each location. some example pronunciations are shown in ## methods we compute embeddings from the hidden transformer layers of three fine-tuned deep acoustic wav2vec 2.0 large models, and subsequently determine pronunciation differences using dynamic time warping (dtw) with these embeddings @xcite . we use fine-tuned acoustic models in this study as their hidden representations were found to show the closest match with human perceptual judgements of pronunciation variation @xcite . for the transcription-based approach, we apply a (phonetically sensitive) levenshtein distance algorithm to the available corresponding phonetic transcriptions of the 10 words in all locations. after averaging the word-based differences, the result of both approaches is a distance matrix representing the aggregate pronunciation difference between every pair of locations. both distance matrices are then clustered in four groups and quantitatively compared to a gold standard clustering of four groups (see figure 1a ). these groups correspond to the three regional languages spoken in the netherlands that are recognised by the european charter for regional or minority languages (frisian: light blue in figure 1a , low saxon: dark blue, limburgish: light green) and standard dutch (dark green). we use the fine-tuned english wav2vec 2.0 large model (abbreviated as w2v2-en) released by @xcite . in addition, we use a new pre-trained dutch wav2vec 2.0 large model that is fine-tuned on dutch labelled data (abbreviated as w2v2-nl), and we use the multilingual xlsr-53 model of @xcite that is fine-tuned on the same dutch labelled data (xlsr-nl). we explicitly use models for dutch because this language is closely related to the different regional languages and dialects spoken in the netherlands (including frisian, low saxon, and limburgish; @xcite . the advantage of having a transformer-based language model that is linguistically closest was shown by @xcite , albeit for a different task (i.e. part-of-speech tagging). it may therefore be the case that a high degree of language similarity is also beneficial for transformer-based models that learn speech representations. acoustic models w2v2-en is pre-trained on 960 hours of english speech from the librispeech dataset @xcite . the model consists of a convolutional encoder, a quantizer, and a 24-layer transformer network. subsequently, the learned representations are fine-tuned on 960 hours of labelled data by adding a randomly initialised linear projection layer on top of the transformer network. this projection layer is used to predict characters from the labelled data using the connectionist temporal classification loss function (ctc; @xcite . xlsr-53 has the same architecture as the other acoustic models, except that the quantizer has learned a single set of discrete speech representations that is shared across the pre-training languages (which includes dutch and german, but not frisian, low saxon or limburgish). this model is pre-trained on 56,000 hours of speech in 53 languages (44,000 hours consists of english speech) obtained from the babel, common voice and multilingual librispeech datasets @xcite @xcite . to obtain xlsr-nl, xlsr-53 is fine-tuned on the same labelled data as w2v2-nl with the same configuration details. obtaining pronunciation differences we compute pronunciation differences between all 106 locations in our dataset using both phonetic transcriptions and acoustic embeddings. for determining the phonetic transcription-based distance, we use a variant of the levenshtein distance (ld) algorithm proposed by @xcite , which includes automatically determined phonetic segment distances. this algorithm matches perception well @xcite and is often used for investigating dialect variation. given a pair of locations, recordings of the same word are compared using ld (phonetic transcriptions) or dtw (acoustic embeddings), which is a frequently-used algorithm for comparing representations of acoustic sequences @xcite . the acoustic embeddings are obtained for each model for each of the 24 layers separately (i.e. to determine the optimal layer). the word-based distances between two locations are averaged to determine the single pronunciation distance between a location pair. this process is repeated for all pairs to create a symmetric distance matrix including all locations. clustering we classify the phonetic transcription distance matrix and the acoustic distance matrices (three models times 24 layers) from the acoustic embeddings using seven clustering techniques, yielding the four different groups. of course, the choice of clustering technique may influence the results, but we determine the optimal clustering algorithm by selecting the one best representing the underlying difference matrix. we use clustering techniques that have previously been applied to distance matrices of dialect pronunciations, namely single link (sl), complete link (cl), group average (ga), weighted average (wa), unweighted centroid (uc), weighted centroid (wc) and minimum variance (mv) clustering @xcite . to select the best clustering algorithm, we calculate the cophenetic correlation coefficient @xcite . this coefficient represents the (pearson) correlation between the original distances and the clustering-based cophenetic distances (i.e. extracted from the dendrogram underlying the clustering). higher values indicate a better (a) xlsr-nl layer 15 (b) ld figure 2: mds maps visualizing pronunciation differences based on dutch dialect pronunciations. similar colours correspond to pronunciations that are also similar. evaluation we compare the layer-based clustering results per model to the gold standard clustering. we do this by computing the cdistance score, which is a clustering comparison measure proposed by @xcite . as opposed to other techniques for comparing clustering partitions, this measure incorporates spatial information in the evaluation (i.e. the coordinates of the locations), which is essential for evaluating spatial (i.e. geographical) clustering. the cdistance scores (for the optimal clustering method per layer) are compared across the layers for each model. the layer with the lowest score per model (i.e. most closely matching the gold standard clustering) is selected for the comparison of the three models. in addition, we create multidimensional scaling (mds) maps @xcite using the best-performing model and compare it to the frequently used ld algorithm to show the (more fine-grained) relationship between the geographical location of the locations and the pronunciation differences. our results show that the xlsr-nl model with output layer 15 and complete link clustering shows the best performance among the fine-tuned models. note that the standard deviation of the performance for the xlsr-nl model across all transformer layers was equal to 0.09, which highlights the strong performance of this model over the other models. importantly, all fine-tuned acoustic models improve over the ld algorithm, which is traditionally used to investigate (dialectal) language variation. perhaps surprisingly, the w2v2-nl model performs similar to the w2v2-en model. we do not have a clear explanation for this pattern, but it may be caused by the dutch model being based on the english model, in combination with a smaller amount of dutch as opposed to english data used for pretraining. in future work we aim to investigate this. the multilingual xlsr-nl model outperforms both monolingual models. the xlsr-nl model is pre-trained on a variety of languages, including dutch, english and german. the regional languages and dialects spoken in the netherlands have clear links to these three languages (i.e. frisian has some overlap with english, low saxon has some overlap with german, and all varieties overlap with dutch, which is also the fine-tuning language). to illustrate, figure 1 visualizes the gold standard together with the fine-tuned acoustic models. the xlsr-nl model clearly classifies pronunciations in the geographical area where limburgish is spoken (i.e. the light green area) most accurately. while the xlsr-nl model does not perfectly distinguish the low saxon pronunciations (i.e. the dark blue area), the other models perform worse in this regard. to evaluate (albeit subjectively) how well more fine-grained differences are captured by the bestperforming model, figure 2 shows the mds maps for the xlsr-nl model, as well as the ld algorithm. both approaches show the relative gradual nature of dialect variation well. however, the xlsr-nl model seems to capture the larger distinctions (e.g., delineating the limburgish area) better than the ld algorithm. based on these evaluations, xlsr-nl appears to be the best model when little data is available. ## conclusion we have found that the xlsr-nl model can be effectively used to distinguish between language groups in the netherlands when only a small amount of data is available. it even outperformed the ld algorithm, which requires time-consuming phonetic transcriptions. our study further shows that multilingual pre-training and fine-tuning on a similar language (compared to the target languages) is beneficial over using a monolingual model.