Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models (2024)

Abstract: In the highly constrained context of low-resource language studies, we explore vector representations of speech from a pretrained model to determine their level of abstraction with regard to the audio signal. We propose a new unsupervised method using ABX tests on audio recordings with carefully curated metadata to shed light on the type of information present in the representations. ABX tests determine whether the representations computed by a multilingual speech model encode a given characteristic. Three experiments are devised: one on room acoustics aspects, one on linguistic genre, and one on phonetic aspects. The results confirm that the representations extracted from recordings with different linguistic/extra-linguistic characteristics differ along the same lines. Embedding more audio signal in one vector better discriminates extra-linguistic characteristics, whereas shorter snippets are better to distinguish segmental information. The method is fully unsupervised, potentially opening new research avenues for comparative work on under-documented languages.

## introduction in recent improvements in speech processing, 1 the amount of data used at pre-training has been instrumental @xcite , which makes it more challenging -if not impossible -to reach similar levels of performance for endangered languages. developing new unsupervised approaches, in addition to being cost-effective @xcite , helps us better understand speech models. speech is highly multifactorial: a recorded voice tells a message and conveys an intention, and the audio also contains information about the surroundings. this study addresses the topic of the nature of the information encoded in the representations produced by a neural network in an unsupervised manner. towards this end, we perform distance measurements over the representations. our goal is to investigate the level of abstraction encapsulated in these representations. our experimental setup relies on tailored datasets to see how specific differences in the input signal are reflected in the output vectors. abx tests are used on audio data in the na language (iso-639-3: nru) and in the naxi language (nxq). three series of experiments are devised to assess differences between recordings. (i) the folk tale series aims to explore an extra-linguistic dimension by comparing seven versions of the same tale by the same speaker. (ii) the song styles series compares different songs interpreted by a single singer. (iii) finally, the phonetics series explores the segmental dimension by comparing sentences (some identical, some different) from different speakers. the results provide an insight into the nature of the information encoded in the representations of a model such as xlsr-53 @xcite . our findings suggest that abx tests can be leveraged to bring out differences in the acoustic setup (room, microphone), in the voice properties, or in the linguistic content. a parametric study shows that processing audio by snippets foot_2 of 10 s is sufficient to bring out differences in the acoustic setup and in voice properties, while 1 s snippets are better for segmental characteristics. this study offers an innovative method to detect confounding factors in corpora intended for unsupervised learning, and provides a means to accelerate the classification of recordings (e.g., by noise level or genre) where such metadata are unavailable. ## method we propose a method based on two components: (i) abx tests to determine -via similarity tests -whether a characteristic of an audio recording is present or not, and (ii) audio corpora with precise metadata. these metadata allow us to build datasets based on one characteristic at a time: language name, speaker id, room acoustics, microphone type, voice properties or segmental content. abx tests to find out, in an unsupervised manner, if a multilingual speech model encodes a characteristic c of the speech signal, we use the abx tests introduced by @xcite and @xcite . the test relies on vector representations built by a pre-trained model for three audio snippets. let a and x denote the snippets that share the characteristic c, while b is the one that does not. the test checks whether the distance d(a, x) is smaller than d(a, b). the metric used in our abx tests is the cosine distance. the abx score corresponds to the proportion of triplets for which the condition d(a, x) < d(a, b) holds true. an abx score close to 50 % (or lower) indicates that, on average, the distance between a and x is close to the distance between a and b, suggesting that c is not encoded in the audio representation. conversely, the closer the score is to 100 %, the more the representation captures the characteristic c. abx tests are interesting for low-resource scenarios because they require no additional training, so they can be directly applied to the representations (unlike linguistic probes: @xcite @xcite . corpora our study relies on recordings in na (iso-639-3 code: nru) and naxi (nxq). na and naxi are spoken in southwest china. na is the mother tongue of approximately 50,000 people. naxi is more widely spoken, as the mother tongue of approximately 200,000 people. both languages are gradually replaced by mandarin, the official language used in schools, administrations and the media @xcite . all recordings come from the pangloss collection, an open-access archive of 'little-documented languages'. each resource's doi is provided in app. e. three series of recordings selected for their characteristics are considered: the first batch studied comprises three versions: v 1 , v 2 and v 3 . v 1 was recorded in a room with perceptible reverberation, while v 2 and v 3 were recorded in a damped room. the second batch is made up of v 6 and v 7 . these two versions were recorded in the same acoustic conditions. the audio was captured simultaneously by two microphones: a headset microphone and a handheld microphone placed on a small stand. the third batch compares v 4 and v 5 to all the other recordings of the folk tale series. v 4 and v 5 have a native listener acting as respondent. these recordings are particularly interesting because some potential confounding factors (typically the topic and the speaker) are controlled, which makes it possible to focus on the influence of certain specific factors (e.g., room acoustics). the five recordings of phonetic elicitations have the same content (apart from the variation inherent to the experimental process in fieldwork conditions: niebuhr and michaud 2015) whereas lexical elicitations are a completely different content. only as participated in both the phonetic and lexical elicitation sessions. tables 1, 2 and 3 in app. a provide a more complete view of the abovementioned metadata. experimental setting in all our experiments, we use the xlsr-53 foot_3 model, a wav2vec2 architecture trained on 56 kh of (raw) audio data in 53 languages @xcite . na is not present in the pre-training data of this model, but it has been shown that the model can be fine-tuned to do asr on na @xcite , and therefore the phonetic module is able to handle the diversity of surface realizations of this language. for the comparisons, we consider audio snippets of length 1 s, 5 s, 10 s and 20 s in order to study the effect of snippet length on our abx test. we use maxpooling to build a single vector representing the snippet. we then build fine-grained heatmaps of abx scores. we use the representations from the 21 st layer, following tests on a validation set. this choice is based on the findings of @xcite and @xcite , who show that the ability of wav2vec2 representations to capture linguistic information declines in the final three layers. ## results using abx tests with carefully selected audio recordings, we investigate whether or not the audio representations computed by wav2vec2 capture specific information from the audio signal. ## discussion and conclusion when one undertakes the task of comparing vector representations of audio, differences are expected, too many of them rather than too few. we adopted an experimental method to submit a given model to different experiments with test variables. in the first two series, the recordings are distinguished according to (i) technical acoustic properties in the folk tale series, or (ii) voice properties in batch v 4 , v 5 of the folk tale series or in the song styles series. a 10 s snippet length seems to best reveal differences in characteristics such as (i) room acoustics or microphone type or (ii) speech rate or genre. our aim in these two series was to explore to what degree extra-linguistic information is present in the representations. being able to detect acoustic differences such as the amount of reverb in a room, or the fact that we are not only capable of measuring differences between narratives and songs but also to distinguish between song styles, gives us reasons to think that our method should be useful to automatically classify recordings based on room acoustics, interview setup, or genre. the prosodic characteristics of a recording also seem to be encoded, which is encouraging for future research on tone using unsupervised methods on audio recordings. in the phonetics series, we focused on 1 s snippet lengths. the recordings of three speakers who participated in a phonetics experiment, quasi-identical to one another, are distinguished from a recording with a different content, but the distinction is not very strong. the snippets from this series are shorter and result in smaller differences on the abx score. this observation suggests that differences are only detected when the segmental content changes, and shows the consistency of our method. using this method on cross-speaker, or cross-linguistic snippets however requires additional investigations to devise a method more suited to phonetic segments. among possible improvements, using segmented corpora would be an interesting avenue of research.