Self-supervised Semantic-driven Phoneme Discovery for Zero-resource Speech Recognition (2022)

Abstract: Phonemes are defined by their relationship to words: changing a phoneme changes the word. Learning a phoneme inventory with little supervision has been a longstanding challenge with important applications to under-resourced speech technology. In this paper, we bridge the gap between the linguistic and statistical definition of phonemes and propose a novel neural discrete representation learning model for self-supervised learning of phoneme inventory with raw speech and word labels. Under mild assumptions, we prove that the phoneme inventory learned by our approach converges to the true one with an exponentially low error rate. Moreover, in experiments on TIMIT and Mboshi benchmarks, our approach consistently learns a better phoneme-level representation and achieves a lower error rate in a zero-resource phoneme recognition task than previous state-of-the-art self-supervised representation learning algorithms.

## introduction thanks to recent developments in self-supervised speech representation learning @xcite @xcite , there is new hope for the development of speech processing systems without the need for full textual transcriptions. supervised speech processing systems for tasks such as automatic speech recognition (asr) rely on a large amount of textual transcriptions, but self-supervised systems can be applied to under-resourced languages in which such annotation is either scarce or unavailable. a key task of the self-supervised system is to learn a discrete representation. while it is possible to discretize the speech solely on the basis of its acoustic properties, a more desirable discrete representation would serve as a bridge from the continuous acoustic signal toward higher-level linguistic structures such as syntax and semantics. such a rep-resentation would make it possible to repurpose algorithms developed for written languages so that they could be used for unwritten languages in tasks such as speech translation and spoken language understanding. words are the obvious choice for a discrete, semantic-driven speech representation, but a practical speech understanding system needs at least thousands of words; learning them in an unsupervised manner may be challenging. phonemes may be a more learnable representation. according to the standard linguistic definition, phonemes are closely linked to words: definition 1. (linguistic definition of phonemes @xcite ) phonemes are the smallest units in speech such that given a correct native word, the replacement of one or more phonemes by other phonemes (capable of occurring in the same position) results in a native word other than that intended, or a native-like nonsense word. for example, the sentences "he thinks" and "he sinks" differ by exactly one phoneme but have very different meaning. the optimal compactness of a phoneme inventory as specified in the definition leads to three advantages. first, learning phonemes requires lower sample complexity than learning words since the number of distinct phonemes is much smaller than the number of distinct words in a language. second, the phonemes are much more abundant and more balanced in classes than words within a speech corpus, which makes sample complexity less of an issue when learning phonemes. third, phonemes are more generalizable in the sense that knowing the phoneme inventory allows the learner to memorize previously unseen words as sequences of phonemes, and, having memorized them, to begin seeking clues to their meaning. motivated by the semantic-driven definition of phonemes, we formulate the problem of learning a phoneme inventory as a self-supervised learning problem, where a small amount of semantic su-pervision is available. the required supervision specifies which acoustic segments are instances of the same word, and which are instances of different words. such supervision might be acquired in a naturalistic setting by asking native speakers to name objects in a set of standardized images, as is commonly done in primary education classrooms, or by asking for the translations of common words in a second language, a common baseline approach in dialectology and historical linguistics @xcite . our contributions are threefold: (1) we propose a computationally tractable definition of phoneme that is almost equivalent to the linguistic definition. (2) we propose a finite-sample objective function for learning phoneme-level units and prove that when the phoneme segmentation is available and under mild conditions, the empirical risk minimizer (erm) of this objective will find the correct phoneme inventory with exponentially low error rate. (3) we propose a novel neural network called information quantizer to optimize the proposed objective, which achieve state-of-the-art results in the phoneme inventory discovery task on the timit and low-resourced mboshi benchmarks with much less training data than previous approaches. ## related works due to the challenge of learning phonemes, early works on unsupervised speech representation learning @xcite @xcite focus on learning speech segments sharing similar acoustic properties, or phones, without taking into account the meaning of the speech they are part of. there are two main approaches in this direction. one approach is to learn discrete phone-like units without any textual labels by modeling phone labels of the speech segments as latent variables. in particular, @xcite first detect segments with recurring patterns in the speech corpus followed by graph clustering using the similarity graph formed by the segments. @xcite @xcite develop probabilistic graphical models to jointly segment and cluster speech into phone-like segments. an extension to the latent variable approach is to introduce additional latent variables such as speaker identity @xcite or language identity @xcite and develop mechanisms to disentangle these variables. with the advance of deep learning, neural network models have also been proposed to learn unsupervised phone-level representation either by first learning a continuous representation @xcite @xcite followed by off-line clustering, or by learning a discrete representation end-to-end with gumbel softmax @xcite or vector-quantized variational autoencoder (vq-vae) @xcite @xcite . however, codebooks learned by the neural approaches tend to be much larger than the number of phonemes @xcite , leading to low scores in standard phoneme discovery metrics. the second approach utilizes weak supervision such as noisy phone labels predicted by a supervised, multilingual asr system trained on other languages. along this direction, early works @xcite @xcite have showed that phonetic knowledge gained from one language can be leveraged to develop asr systems for another language using an hmm-based or dnn-hmm hybrid approach. instead of using phone labels, @xcite explores the use of articulatory features as supervision for the multilingual asr. recently, @xcite systematically study the performance of zero-shot crosslingual asr on 13 languages trained with international phonetic alphabet (ipa) tokens and found that the system tends to perform poorly on unseen languages. instead, @xcite is able to discover phone-like units by clustering bottleneck features (bnf) from a factorized time-delay neural network (tdnn-f) trained with phone labels predicted by a crosslingual asr @xcite . several works have since shifted focus toward the more challenging phoneme discovery problem by formulating it as a self-supervised learning problem where the semantics of the speech are known, such as from translation, phonemelevel language models or other sensory modalities such as vision. @xcite has studied the use of pairwise word identity labels for training phoneme discovery models based on gaussian mixture models (gmm); @xcite analyzes the hidden layers of a two-branch neural network trained to retrieve spoken captions with semantically related images and finds strong correlation between segment representation and phoneme boundaries. @xcite adds hierarchical vector quantization (vq) layers in the same retrieval network and is able to find a much smaller codebook than the unsupervised neural approach @xcite , and achieve high correlation with the phoneme inventory. @xcite has studied the possibility of learning semantic units using an attention-based speech-to-text translation system, though the units appear to correlate more with words. works on unsupervised speech recognition @xcite attempt to learn to recognize phonemes by leveraging the semantic information from a phoneme language model unpaired with the speech, typically by matching the empirical prior and posterior distributions of phonemes either using cross entropy @xcite or adversarial loss @xcite . such models, however, have a slightly different objective as they assume knowledge about the phoneme inventory of the language and instead tries to find the alignment between the speech and phonemes, rather than induce the phoneme inventory from scratch. 3 semantic-driven phoneme discovery 3.1 notation throughout the paper, we use p{•} to denote probability. we use capital letters to denote random variables and lower-case letters to represent samples of random variables. we use p x := p{@xmath0, depending on whether it is continuous or discrete. further, denote p y |x (y|x) := p{@xmath1. the probability simplex in r d is denoted as ∆ d . ## information quantizer we solve the spd problem using a novel type of neural network called an information quantizer (iq), depicted in figure ## experimental setup datasets we construct four training datasets consisting of spoken words only. the vocabulary set with |y| = 224 is selected from head words of noun phrases from the flickr30kentities dataset @xcite ) that appear at least 500 times. for the flickr audio word dataset, spoken words in the vocabulary are extracted from flickr audio dataset @xcite . for the librispeech and timit word dataset with |y| = 224, spoken words are extracted from librispeech @xcite 460-hour train-clean subset, resulting in a dataset of about 6 hours and 0.1 hours; for librispeech and timit word dataset with |y| = 524 and |y| = 824, we supplement the dataset with the speech for the top 300 frequent words and top 600 frequent words respectively (excluding the visual words) in librispeech, resulting in datasets of about 15 and 21 hours. for mboshi dataset, we found only about 20 actual words occur more than 100 times, so instead we use n-grams with either @xmath2, resulting in a vocabulary size of 161 and 377 respectively. note that the amount of labeled data we need is much lower than previous works @xcite : around 30 hours, @xcite : around 600 hours) and the vocabulary size used is much smaller than the total vocabulary size in the language. more details of the sets can be found in appendix b. we also test our models on two standard phoneme discovery benchmarks, which contain whole-sentence utterances with many words unseen during training. the first dataset is timit @xcite , an english corpus consisting of about 5 hours speech and mboshi @xcite , which contains about 2.4 hours speech from a low-resource language. for both datasets, we follow the split in @xcite , @xcite baselines for phoneme discovery from segmented words, we compare our model (iq) to four baselines. the first two baselines use continuous representation: the cpc+k-means model performs k-means clustering on the segment-level cpc features, and the k-means model performs k-means clustering after the model is trained on the word recognition task. the last two baselines use discrete representations: the gumbel variational information bottleneck @xcite (gumbel vib) is a neural model with a gumbel softmax @xcite layer to approximate the codebook assignment function z(•), and we set β = 0.001 and decay the temperature of the gumbel softmax from 1 to 0.1 linearly for the first 300000 steps, keeping it at 0.1 afterwards, which works best in our experiments; the deterministic information bottleneck (dib), a generalization of (strouse and schwab, 2016) for continuous feature variable x, which assumes the same deterministic relation between speech x and codebook unit z as ours, but optimizes the models in a pipeline fashion (first the speech encoder and then the quantizer) by performing clustering on the learned conditional distributions. the cpc features used are trained in a self-supervised fashion on the 960-hour lib-rispeech dataset and released by @xcite . all models share the same speech encoder as iq. for the whole-sentence datasets, we compare our models to three phoneme discovery systems, namely, the unsupervised h-shmm trained with multilingual speech @xcite , the resdavenet-vq @xcite with visual supervision and the tdnn-f system by @xcite trained with multilingual speech. to study how well our model performs in extreme low-resource speech recognition compared to other neural speech representation learning models, we compare our models to wav2vec @xcite , wav2vec 2.0 @xcite ) (small, trained on the 960-hour librispeech), vq-wav2vec with gumbel softmax and k-means as discretiza-tion strategies @xcite , cpc (van den oord et al., 2019) and vq-cpc (van niekerk et al., 2020), using the pretrained models released by the authors. implementation details of the baselines and our models are in appendix c. evaluation metrics standard metrics are used such as nmi and boundary f1 for the quality of codebook and segmentation respectively with the same implementation as in prior works @xcite . in addition, token f1 @xcite is also reported. to examine the benefit of using our discovered phoneme inventory for low-resource speech recognition, we also evaluate using equivalent phone error rate (equiv. per: @xcite . this metric can be viewed as a proxy for phone error rate (per) applicable beyond supervised speech recognizers. ## conclusion motivated by the linguistic definition of phonemes, we propose information quantizer (iq), a new neural network model for self-supervised phoneme discovery that can take advantage of word-level supervision. we demonstrate in two ways that wordlevel supervision is beneficial for phoneme inventory discovery: theoretically, we prove that iq can achieve zero token error rate asymptotically with the help of word labels; empirically, we show that iq out-performs various speech-only algorithms in phoneme discovery tasks under both simulated (english) and realistic (mboshi) low-resource settings. in the future, we would like to apply the discovered phoneme inventory to develop better low-resource speech technologies such speech translation and speech synthesis systems. marcely zanon boito, aline villavicencio, and laurent besacier. 2019. empirical evaluation of sequenceto-sequence models for word discovery in lowresource settings. in proc. annual conference of international speech communication association (in-terspeech). k.-y. chen, c.-p. tsai, d.-r. liu, h.-y. lee, and l. shan lee. 2019. completely unsupervised speech recognition by a generative adversarial network harmonized with iteratively refined hidden markov models. in proc. annual conference of international speech communication association (inter-speech). jan chorowski, ron j. weiss, samy bengio, and aäron van den oord. 2019. unsupervised speech representation learning using wavenet autoencoders. ieee transactions on audio, speech and language processing. yu-an chung, wei-ning hsu, hao tang, and james r. glass. 2019. an unsupervised autoregressive model for speech representation learning. in in proc. annual conference of international speech communication association (interspeech). thomas m. cover and joy a. thomas. 2006. elements of information theory (wiley series in telecommunications and signal processing). wiley-interscience, usa. ewan dunbar, xuan-nga cao, juan benjumea, julien karadayi, mathieu bernard, laurent besacier, xavier anguera, and emmanuel dupoux. 2017. the zero resource speech challenge 2017. corr, abs/1712.04313. ryan eloff, andré nortje, benjamin van niekerk, avashna govender, leanne nortje, arnu pretorius, elan van biljon, ewald van der westhuizen, lisa van staden, and herman kamper. 2019b. unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. in proc. annual conference of international speech communication association (interspeech). siyuan feng, tan lee, and zhiyuan peng. 2019. combining adversarial training and disentangled speech representation for robust zero-resource subword modeling. in proc. annual conference of international speech communication association (in-terspeech). siyuan feng, piotr żelasko, laureano moro-velázquez, ali abavisani, mark hasegawa-johnson, odette scharenborg, and najim dehak. 2021a. how phonotactic affect multilingual and zero-shot asr performance. in ieee international conference on acoustics, speech and signal processing (icassp).