Representation Learning for Discovering Phonemic Tone Contours (2020)

Abstract: Tone is a prosodic feature used to distinguish words in many languages, some of which are endangered and scarcely documented. In this work, we use unsupervised representation learning to identify probable clusters of syllables that share the same phonemic tone. Our method extracts the pitch for each syllable, then trains a convolutional autoencoder to learn a low-dimensional representation for each contour. We then apply the mean shift algorithm to cluster tones in high-density regions of the latent space. Furthermore, by feeding the centers of each cluster into the decoder, we produce a prototypical contour that represents each cluster. We apply this method to spoken multi-syllable words in Mandarin Chinese and Cantonese and evaluate how closely our clusters match the ground truth tone categories. Finally, we discuss some difficulties with our approach, including contextual tone variation and allophony effects.

## introduction tonal languages use pitch to distinguish different words, for example, yi in mandarin may mean 'one', 'to move', 'already', or 'art', depending on the pitch contour. of over 6000 languages in the world, it is estimated that as many as 60-70% are tonal @xcite . a few of these are national languages (e.g., mandarin chinese, vietnamese, and thai), but many tonal languages have a small number of speakers and are scarcely documented. there is a limited availability of trained linguists to perform language documentation before these languages become extinct, hence the need for better tools to assist linguists in these tasks. one of the first tasks during the description of an unfamiliar language is determining its phonemic inventory: what are the consonants, vowels, and tones of the language, and which pairs of phonemes are contrastive? tone presents a unique challenge because unlike consonants and vowels, which can be identified in isolation, tones do not have a fixed pitch, and vary by speaker and situation. since tone data is subject to interpretation, different linguists may produce different descriptions of the tone system of the same language @xcite . in this work, we present a model to automatically infer phonemic tone categories of a tonal language. we use an unsupervised learning approach: a convolutional autoencoder learns a low-dimensional representation of each tone using only a set of spoken syllables in the target language. this is followed by mean shift clustering to identify clusters of syllables that probably have the same tone. we apply our method on mandarin chinese and cantonese datasets, for which the ground truth annotation is used for evaluation. our method does not make any language-specific assumptions, so it may be applied to low-resource languages whose phonemic inventories are not already established. guages in the sinitic family @xcite . mandarin has four lexical tones: high (55), rising (25), low-dipping (214), and falling (51) foot_0 . the third tone sometimes undergoes sandhi, addressed in section 3. we exclude a fifth, neutral tone, which can only occur in word-final positions and has no fixed pitch. ## related work many low-resource languages lack sufficient transcribed data for supervised speech processing, thus unsupervised models for speech processing is an emerging area of research. the zerospeech 2015 and 2017 challenges featured unsupervised learning of contrasting phonemes in english and xitsonga, evaluated by an abx phoneme discrimination task @xcite . one successful approach used denoising and correspondence autoencoders to learn a representation that avoided capturing noise and irrelevant inter-speaker variation @xcite . deep lstms for segmenting and clustering phonemes in speech have also been explored in @xcite and @xcite . in mandarin chinese, deep neural networks have been successful for tone classification in isolated syllables @xcite as well as in continuous speech @xcite . both of these models found that mel-frequency cepstral coefficients (mfccs) outperformed pitch contour features, despite the fact that mfcc features do not contain pitch information. in cantonese, support vector machines (svms) have been applied to classify tones in continuous speech, using pitch contours as input @xcite . unsupervised learning of tones remains largely unexplored. @xcite performed unsupervised and semi-supervised tone clustering in mandarin, using average pitch and slope as features, and k-means and asymmetric k-lines for clustering. graph-based community detection techniques have been applied to group n-grams of contiguous contours into clusters in mandarin @xcite . in recent work concurrent to ours, fry (2020) uses adversarial autoencoders and hierarchical clustering to identify tone inventories, and evaluate their method on mandarin, cantonese, fungwa, and english data. we further explore unsupervised deep neural networks for phonemic tone clustering. it should be noted that our unsupervised model is not given tone labels during training, and the number of tones is assumed to be unknown, so it cannot be directly compared to supervised tone classifiers in the literature. ## data and preprocessing we use data from mandarin chinese and cantonese. for each language, the data consists of a list of spoken words, recorded by the same speaker. the mandarin dataset is from a female speaker and is provided by shtooka 2 , and the cantonese dataset is from a male speaker and is downloaded from forvo foot_2 , an online crowd-sourced pronunciation dictionary. we require all samples within each language to be from the same speaker to avoid the difficulties associated with channel effects and inter-speaker variation. we randomly sample 400 words from each language, which are mostly between 2 and 4 syllables; to reduce the prosody effects with longer utterances, we exclude words longer than 4 syllables. we extract ground-truth tones for evaluation purposes. in mandarin, the tones are extracted from the pinyin transcription; in cantonese, we reference the character entries on wiktionary foot_3 to retrieve the romanized pronunciation and tones. for mandarin, we adjust for third-tone sandhi (a phonological rule where a pair of consecutive third-tones is always realized as a second-tone followed by a third-tone), and use the sandhi tone as the ground truth. we also exclude the neutral tone, which has no fixed pitch and is sometimes thought of as a lack of tone. ## results figure 4 shows the latent space learned by the autoencoders and the clustering output. our model found 4 tone clusters in mandarin, matching the number of phonemic tones ( in cantonese, the 5 clusters a-e correspond to low-falling, mid-level, high-level, mid-rising, and low-rising tones. tone clustering in cantonese is expected to be more difficult than in mandarin because of 6 contrastive tones, rather than 4. the model is more effective at clustering the higher tones (1, 2, 3), and less effective at clustering the lower tones (4, 5, 6), particularly tone 4 (lowfalling) and tone 6 (low-level). this confirms the difficulties in prior work, which reported worse classification accuracy on the lower-pitched tones because the lower region of the cantonese tone space is more crowded than the upper region @xcite . to evaluate how much the clusters match the ground truth, we use normalized mutual information (nmi); this is preferable over accuracy because it does not require the number of detected clusters to be the same as the number of tones. in in all cases, the clusters from the autoencoder model have higher nmi than the k-means model. the improvement is due to the mean shift procedure identifying points that belong to a cluster with high confidence: it only only makes predictions for those points, whereas k-means assigns every point to a cluster. all models perform better on the ## limitations 6.1 contextual effects one limitation of our model is it considers syllables in isolation, but in reality, pitch is affected by context. two types of contextual effects are carryover and declination. a carry-over effect is when the pitch contour of a tone undergoes contextual variation depending on the preceding tone; strong carry-over effects have been observed in mandarin @xcite . prior work @xcite avoided carry-over effects by using only the second half of every syllable, but we do not consider languagespecific heuristics in our model. declination is a phenomenon in which the pitch declines over an utterance @xcite . this is especially a problem in cantonese, which has tones that differ only on pitch level and not contour: for example, a mid-level tone near the end of a phrase may have the same absolute pitch as a low-level tone at the start of a phrase. contextual effects are apparent in our results ( ## conclusion we propose a model for unsupervised clustering and discovery of phonemic tones in tonal languages, using spoken words as input. our model extracts the f0 pitch contour, trains a convolutional autoencoder to learn a low-dimensional representation for each contour, and applies mean shift clustering to the resulting latent space. we obtain promising results with both mandarin chinese and cantonese, using only 400 spoken words from each language. cantonese presents more difficulties because of its larger number of tones, especially at the lower half of the pitch range, and also due to multiple contrastive level tones. still, in both our languages, our method finds clusters of tones that better match the ground truth than the k-means baseline. finally, we discuss the effects of contextual variation and the limitations of unsupervised learning for the tone induction problem.