A Theory of Unsupervised Speech Recognition (2023)

Abstract: Unsupervised speech recognition ({pasted macro ‘ASRU’}/) is the problem of learning automatic speech recognition (ASR) systems from unpaired speech-only and text-only corpora. While various algorithms exist to solve this problem, a theoretical framework is missing to study their properties and address such issues as sensitivity to hyperparameters and training instability. In this paper, we proposed a general theoretical framework to study the properties of {pasted macro ‘ASRU’}/ systems based on random matrix theory and the theory of neural tangent kernels. Such a framework allows us to prove various learnability conditions and sample complexity bounds of {pasted macro ‘ASRU’}/. Extensive {pasted macro ‘ASRU’}/ experiments on synthetic languages with three classes of transition graphs provide strong empirical evidence for our theory (code available at https://github.com/cactuswiththoughts/UnsupASRTheory.gitcactuswiththoughts/UnsupASRTheory.git ).

## introduction unsupervised speech recognition (asr-u) is the problem of learning automatic speech recognition (asr) systems from unpaired speech-only and textonly corpora. such a system can not only significantly reduce the amount of annotation resources required for training state-of-the-art asr system, but serve as a bridge between spoken and written language understanding tasks in the low-resource setting. since its first proposal @xcite , it has seen remarkable progress and the current best system @xcite has achieved comparable performance to systems trained with paired data on various languages. however, there are several mysteries surrounding asr-u, which potentially hinder the future development of such systems. in particular, prior experiments have found that training the current state-of-the-art asr-u model, wav2vec-u @xcite , requires careful tuning over the weights of various regularization losses to avoid converging to bad local optima and that even despite extensive regularization weight tuning, wav2vec-u may still fail to converge @xcite . therefore, it remains a mystery whether or when unpaired speech and text data indeed provide sufficient information for learning an asr system. another mystery is whether the success of existing asr-u models based on generative adversarial net (gan) @xcite is sufficiently explained by the gan objective function per se, or also requires other factors, such as randomness in training, quirks in the data used and careful domain-specific hyper-parameter settings, etc. in this paper, we provide a theoretical analysis of asr-u to investigate the mysteries surrounding asr-u. first, we prove learnability conditions and sample complexity bounds that crucially depend on the eigenvalue spacings of the transition probability matrix of the spoken language. random matrix theory shows that such learnability conditions are achievable with high probability. next, we study the gradient flow of gan-based asr-u and provide conditions under which the generator minimizing the gan objective converges to the true generator. finally, to verify our theory empirically, we perform gan-based asr-u experiments on three classes of synthetic languages. not only do we observe phase transition phenomena predicted by our theory, but we achieve stable training with lower test word error rate by several modifications of the existing state-of-the-art asr-u system inspired by our theory. ## problem formulation general formulation the training data comprise a set of sequences of quantized speech vectors, and a set of sequences of phoneme labels. the data are unpaired: there is no label sequence that matches any one of the speech sequences. the data are, however, matched in distribution. let p x i (x) and p y j (y) be the probability mass functions (pmfs) of the i th speech vector in a sequence, @xmath0, and the figure 1 : overview of the asr-u system for our analysis j th phoneme in a sequence, @xmath1, respectively: the requirement that they are matched in distribution is the requirement that there exists some generator function o : (x, y) → {0, 1} such that @xmath2 the problem of asr-u is to find the generator function o. gan-based asr-u eq. ( 1 ) leverages sequence information to remove ambiguity: o must be an optimal generator not only for the positionindependent distributions of x and y , but also for their position-dependent distributions p x i , p y i ∀@xmath3. in reality we cannot observe every possible sequence of speech vectors, or every possible sequence of phonemes, but instead must estimate o from samples. to address this issue, a gan can be used to reproduce the empirical distribution of the training dataset with minimum error, subject to the generator's inductive bias, e.g., subject to the constraint that the function o is a matrix of the form @xmath4, 1} |x|×|y| , where |x| and |y| are the sizes of the alphabets x and y, respectively. as shown in figure 1 , a gan achieves this goal by computing o as the output of a neural network, @xmath5, y; θ), and by requiring g to play a zerosum game with another neural network called the discriminator d with the following general utility function: for the original gan @xcite , a(d) = log(σ(d)) and b(d) =log(1σ(d)), where σ is the sigmoid function. for the wasserstein gan @xcite , d(y ) is a lipschitz-continuous scalar function, and a(d) = b(d) = d. a maximum mean discrepancy (mmd) gan @xcite minimizes the squred norm of eq. ( 2 ), where d(y ) is an embedding into a reproducing kernel hilbert space (rkhs). in this paper we take the rkhs embedding to be the probability mass function of a scalar random variable d(y ), and assume that the discriminator is trained well enough to maintain eq. ( 2 ). in this situation, the mmd gan minimizes eq. ( 2 ) with a(d) = b(d) = y . in practice, eq. ( 2 ) is optimized by alternatively updating the parameters of the discriminator and the generator using gradient descent/ascent: theoretical questions of asr-u the aforementioned formulation of asr-u is ill-posed. intuitively, the function o has finite degrees of freedom (@xmath6, 1} |x|×|y| ), while eq. ( 1 ) must be valid for an infinite number of distributions (p x i and p y i for @xmath7, so there is no guarantee that a solution exists. on the other hand, if the sequence is unimportant (p x @xmath8, @xmath9, then the solution may not be unique. one important question is then: what are the necessary and sufficient conditions for eq. ( 1 ) to have a unique solution? further, it is well-known that gradient-based training of gan can be unstable and prior works on asr-u @xcite have used various regularization losses to stabilize training. therefore, another question of practical significance is: what are the necessary and sufficient conditions for the alternate gradient method as described by eq. ( 3 )-(4) to converge to the true generator for asr-u? in the subsequent sections, we set out to answer these questions. ## experiments synthetic language dataset to allow easy control of the eigenvalue spacings of the transition matrix t and thus observe the phase transition phenomena predicted by our theory, we design six synthetic languages with hmm language models as follows. first, we create the hmm transition graph by treating non-overlapping bigrams as hidden states of the hmm. the hidden state of the hmm will henceforth be referred to as the "speech unit", while the observation emitted by the hmm will be referred to as the "text unit". for the asymptotic asr-u, we control the number of eigenvalues of the markov transition graph by varying the number of disjoint, identical subgraphs. the number of distinct eigenvalues of the whole graph will then be equal to the number of eigenvalues of each subgraph. for the finite sample setting, we instead select only hamiltonian graphs and either gradually decrease the degrees of the original graph to its hamiltonian cycle or interpolate between the graph adjacency matrix and that of its hamiltonian cycle. thus, we can increase σ min (p x ) by increasing w. for both the subgraph in the former case and the hamiltonian graph in the latter, we experiments with circulant, de bruijn graphs @xcite and hypercubes, as illustrated in figure 2 . next, we randomly permute the hidden state symbols to form the true generator mapping from the speech units to text units. to create matched speech-text data, we simply sample matched speech and text unit sequences using a single hmm. for unmatched datasets, we sample the speech and text data independently with two hmms with the same parameters. please refer to appendix b for more details. model architecture for finite-sample asr-u, we use wav2vec-u @xcite with several modifications. in particular, we experiment with various training objectives other than the jensen-shannon (js) gan used in the original wav2vec-u, including the wasserstein gan @xcite and the mmd gan. all additional regularization losses are disabled. moreover, we experimentally manipulate two hyperparameters: phase transition of per vs. eigenvalue gaps: asymptotic case the phoneme error rate (per) as a function of the number of eigenvalues of a for the asymptotic asr-u on the synthetic datasets are shown in figure 3 . for all three graphs, we observe (a) circulant graph (b) de bruijn graph (c) hypercube figure 4: finite-sample asr-u per vs σ min (p x ) for various markov transition graphs figure 5: effect of discriminator resetting at every update clear phase transitions as the number of eigenvalues exceeds the number of speech units, and an increase of the number of distinct, nonzero eigenvalues required for perfect asr-u as the number of speech units increases. phase transition of per vs. eigenvalue gaps: finite-sample case the per as a function of the least singular value σ min (p x ) for the finite-sample asr-u on the synthetic datasets are shown in figure 4. as we can see, the asr-u exhibit the phase transition phenomena in all three graphs, albeit with differences in the critical point and their rate of approaching the perfect asr-u regime. while the per generally decreases as σ min (p x ) gets larger, we found a dip in per in the circulant graph case as σ min (p x ) moves from 10 -31 to 10 -15 . though unexpected, this observation is not contradictory to our theory since our theory does not make explicit predictions about the rate of phase transition for asr-u. across different gan models, we found that jsd generally approaches perfect asr-u at a faster rate than mmd in all three graphs, suggesting the use of nonlinear dynamic may be beneficial. nevertheless, the overall trends for different gans remain in large part homogeneous. between wasserstein and mmd, we observe very little difference in performance, suggesting the regularization effect of ntk is sufficient to control the lipschitz coefficient of the network. finally, for the mmd gan in the matched setting, we found the network is able to achieve perfect asr-u regardless of the spectral properties of the markov transition graphs, which confirms our theory that a symmetric markov random matrix tends to have simple eigenvalue spectrum suitable for asr-u. ## limitations our theory currently assumes that input speech features are quantized into discrete units, as in @xcite , while preserving all the linguistic information in the speech. as a result, our theory does not account for the loss of linguistic information during the quantization process, as often occurred in realistic speech datasets. further, more recent works @xcite have shown that continuous features, with the help of additional regularization losses, can achieve almost perfect asr-u. such phenomena is beyond explanations based on our current theory and require generalizing our theory to continuous speech features. further, our model assumes that sufficiently reliable phoneme boundaries are fed to the asr-u system, and kept fixed during training. it will be interesting to extend our framework to systems with trainable phoneme boundaries, such as the wav2vec-u systems, to better understand its effect on training stability. inverse littlewood-offord theorems and the condition number of random matrices. annual of mathematics, 169(2):595-632. chih-kuan yeh, jianshu chen, chengzhu yu, and dong yu. 2019. unsupervised speech recognition via segmental empirical output distribution matching. in international conference on learning representations.