Encoding of lexical tone in self-supervised models of spoken language (2024)

Abstract: Interpretability research has shown that self-supervised Spoken LanguageModels (SLMs) encode a wide variety of features in human speech from theacoustic, phonetic, phonological, syntactic and semantic levels, to speakercharacteristics. The bulk of prior research on representations of phonologyhas focused on segmental features such as phonemes; the encoding ofsuprasegmental phonology (such as tone and stress patterns) in SLMs is not yetwell understood. Tone is a suprasegmental feature that is present in more thanhalf of the world’s languages. This paper aims to analyze the tone encodingcapabilities of SLMs, using Mandarin and Vietnamese as case studies. We showthat SLMs encode lexical tone to a significant degree even when they aretrained on data from non-tonal languages. We further find that SLMs behavesimilarly to native and non-native human participants in tone and consonantperception studies, but they do not follow the same developmental trajectory.

## introduction explaining the inner workings of self-supervised models of written and spoken language has been the focus of much recent work. transformer-based @xcite written language models have been shown to encode many types of linguistic information @xcite . the analysis of self-supervised spoken language models (slms) is also gaining traction: architectures such as wav2vec2 @xcite and hubert @xcite have been shown to encode linguistic information at the phonetic, phonological, syntactic and semantic levels of human speech without labeled data @xcite @xcite @xcite @xcite . the majority of research on representations of phonetic and phonological information in slms focuses on the segmental level. segmental refers to units of speech that do not spread but remain localized. phonemes (e.g. vowels and consonants) are the smallest abstract units of sound that help to distinguish one unit from another (e.g. pat vs bat). suprasegmental, in contrast, refers to features that are not necessarily limited to single units, but can spread across multiple phonemes or phrases. examples include tone, stress patterns, and intonation, which can all entail syllable and phrase level changes @xcite . the representation of suprasegmental information in slms is important to study, as it is one of the main distinguishing features of speech compared to text: spoken utterances use suprasegmental cues to convey information that is sometimes not explicitly marked in a corresponding written sentence. 1 as a first step, in this work, we focus on lexical tone as a highly constrained, relatively well-understood example of a suprasegmental feature. we firstly examine to what extent slms trained on tonal and non-tonal languages encode tone information in their internal representations. we find that slms are capable of capturing tonal information, regardless of whether they are trained on tonal or non-tonal languages. secondly, we investigate the impact of supervised fine-tuning on the automatic speech recognition (asr) task. we find that fine-tuning enhances tone representations for models trained on tonal languages, but reduces them for models trained on non-tonal languages. thirdly, we explore whether slms exhibit the same perceptual patterns as native and non-native human listeners. we find that models show patterns similar to humans in discrimination of mandarin tones and consonants, but find no evidence that they follow a similar developmental trajectory. ## tones estimates suggest that more than 60% of the world's languages use some degree of tonal contrast @xcite . the primary focus of this work is on lexical tone, the process by which lexical items are distinguished from one another primarily by pitch cues @xcite . we focus primarily on pitch cues: while there are suggestions that articulatory cues alone are not sufficient for tonal distinction in languages such as vietnamese, with socio-linguistic register also being important if present in the language in question @xcite , this is still subject to further research in perception and production studies to validate as the body of work remains small, particularly regarding perception. we therefore state here that non-tonal phonemic units (e.g. vowels, consonants) can be defined primarily by non-pitch articulatory cues, such as vowel height, voicing, and duration. in contrast, tonal units make use of pitch cues, with f0 (fundamental frequency) contour usually considered to be the primary cue @xcite . in ambiguous contexts, other pitch cues can be used in combination with non-pitch cues such as amplitude, voice quality (e.g. breathy vs creaky), and spectral tilt @xcite . the tone bearing unit (tbu) is the segment containing tone; this is typically, although not always, found on the syllable level. in the case of the primary language investigated in this paper, mandarin, syllables are morpheme based (as in, mandarin syllables are morphemes). thus in mandarin every morpheme contains obligatory tone, and is in of itself the tbu @xcite . we compare slms trained on non-tonal languages as well as three fully lexical tonal languages: mandarin, cantonese and vietnamese. the models are tested primarily on mandarin data. mandarin demonstrates full tonality, with tone found on each morpheme @xcite , and has been widely studied for tone perception as well as acquisition. secondarily, we also test on data from another lexical tone language, vietnamese, to assess if our results generalize. mandarin chinese is typically described as containing four (lexical) tones and one neutral tone that only occurs in unstressed syllables @xcite . the tones are conventionally assigned the labels 1-4 (t1-4); figure 1 illustrates the four mandarin tones. since one morpheme (one character) corresponds to one tone in mandarin chinese, we can use the pinyin transcription to obtain our tone labels easily (see figure 1 for notations); for example: the tone label corresponds to the tone of the character when it is pronounced in isolation (base form). however, mandarin features tone sandhi, i.e. the tone assigned to individual morphemes can change in pronunciation based on the tone of the adjacent morpheme (sandhi form). one instance of tone sandhi rules in mandarin is t3 sandhi @xcite : if two t3 ('dipping') tones occur next to one another, the first will adjust to t2 ('rising') to avoid two consecutive t3 tones, as can be seen in examples 2 and 3, after @xcite . tone labels obtained from pinyin transcriptions only take the base form into account. the primary pitch cue that distinguishes the individual mandarin tones from each other is f0; however, secondary pitch cues are also present such as voice quality and spectral tilt @xcite . vietnamese also has obligatory tones on every syllable, similar to mandarin's morphemic tbu @xcite . we adhere to the eight tone system described by @xcite in our experiment setup. cantonese is a sinitic language related to mandarin, and also features lexical tone, with six tonal distinctions @xcite as opposed to mandarin's four. ## related work the present paper builds both on works interpreting the inner workings of slms and on experiments on perception of aspects of human speech. ## methodology we use a number of wav2vec2-based @xcite ) models pre-trained and fine-tuned on datasets of different languages for our investigation. as examples of tonal languages, we choose mandarin, vietnamese and cantonese, whereas english and french serve as non-tonal language examples. the models trained in the languages above are then tested on test data from mandarin and vietnamese. to examine the encoding of tone, we train linear probing classifiers on the hidden state activations extracted from the aforementioned models for every morpheme in our testing datasets. foot_1 4.1 datasets training data. we examine slms that were trained on datasets of the following languages: mandarin pre-trained with aishell-2 @xcite and fine-tuned with aishell-1 @xcite . english pre-trained and fine-tuned with librispeech @xcite . vietnamese pre-trained with unlabelled youtube audio and fine-tuned with the vlsp dataset for asr @xcite . cantonese pre-trained on a combined dataset of older cantonese adult speech and youtube audio @xcite . french pre-trained on mls french @xcite . test data. we primarily use the mandarin chinese thchs-30 dataset @xcite for testing models' encoding of mandarin tone. thchs-30 consists of 30 hours of mandarin speech recorded in a laboratory environment. the dataset is transcribed into both chinese characters and mandarin pinyin. we also obtain characterlevel forced alignment with the charsiu aligner @xcite . to test the generalizability of our results, we also use the vietnamese vivos dataset @xcite , which consists of 15 hours of vietnamese read speech recorded in a laboratory environment. the dataset is transcribed into vietnamese orthography. we then convert the transcription into international phonetic alphabet (ipa) with tone labels with vphon @xcite . we use the montreal forced aligner @xcite to obtain a syllable-level forced alignment. pre-training data. for the experiments on slm's learning trajectory and perceptual patterns (see section 5.3), we pre-train slms from scratch on the following datasets: size (hours) training language tonality pre-training fine-tuning speech type english @xcite non-tonal 960 960 read french @xcite non-tonal 1,000 -read mandarin (lu and chen, 2022) tonal 1,000 178 read vietnamese @xcite tonal 13,000 250 youtube audio/read cantonese @xcite tonal 2,800 -spotaneous + read ## results in this section, we present a series of experiments for analyzing the encoding of tone in slms. ## conclusion we analyze the tone encoding capabilities of spoken language models trained on three tonal and two non-tonal languages, using classifier probes with data from two tonal languages: mandarin and vietnamese. we find that slms trained on either tonal or non-tonal languages encode tonal information in mandarin and vietnamese to a significant degree. we also find that fine-tuning for the speech recognition task enhances the tone encoding capabilities of models trained on tonal languages but reduces them for models trained on non-tonal languages. while we see evidence suggesting that the learning trajectories of slms in pre-training do not follow the same developmental trajectories found in human language acquisition, we find that slms show patterns similar to that of human listeners in tone and consonant perception experiments. while this paper focused on investigating the encoding of lexical tones with mandarin and vietnamese as case studies, mandarin and vietnamese are in no way representative of all the diverse tonal languages that exist in the world. the encoding of other suprasegmental features such as stress patterns and intonation is equally important to study in future work. we hope that explaining selfsupervised models of spoken languages provides a unique perspective for us to contribute to a better understanding of how languages work. given the rise of slms trained with multilingual datasets @xcite , it would be interesting to investigate if the multilingual slms encode segmentals and suprasegmentals from different languages more robustly. this paper serves as a starting point in the research of the encoding of suprasegmental cues in spoken language models.