DP -Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon (2022)

Abstract: Finding word boundaries in continuous speech is challenging as there is little or no equivalent of a ‘space’ delimiter between words. Popular Bayesian non-parametric models for text segmentation (Goldwater et al., 2006, 2009) use a Dirichlet process to jointly segment sentences and build a lexicon of word types. We introduce DP-Parse, which uses similar principles but only relies on an instance lexicon of word tokens, avoiding the clustering errors that arise with a lexicon of word types. On the Zero Resource Speech Benchmark 2017, our model sets a new speech segmentation state-of-the-art in 5 languages. The algorithm monotonically improves with better input representations, achieving yet higher scores when fed with weakly supervised inputs. Despite lacking a type lexicon, DP-Parse can be pipelined to a language model and learn semantic and syntactic representations as assessed by a new spoken word embedding benchmark. 1

## introduction one of the first tasks that infants face is to learn the words of their native language(s). to do so, they must solve a word segmentation problem, since words are rarely uttered in isolation but come up in multi-word utterances @xcite . segmenting utterances into word or subword units is also an important step in nlp applications, where an input string of orthographic symbols is tokenized into words or sentence piece before being fed to a language model. while many writing systems use white spaces that make (basic) tokenization a relatively simple task, others do not (e.g., chinese) and turn tokenization into a challenging machine learning problem @xcite . a similar situation arises for 'textless' language models based on units derived from raw audio @xcite , which, like infants, do not have access to isolated words or word-separating symbols. the most successful approach to unsupervised word segmentation for text inputs is based on nonparametric bayesian models @xcite @xcite @xcite @xcite that jointly segment utterances and build a lexicon of frequent word forms using a dirichlet process. the intuition is that frequent word forms function as anchors in a sentence, enabling the segmentation of novel words (like 'dax' in 'did you see the dax in the street?'). such models are tested on phonemized texts obtained by converting text into a stream of phonemes after removing spaces and punctuation marks. even though phonemized text may seem a reasonable approximation of continuous speech, such models have given disappointing results when applied directly to speech inputs. since the direct comparison between speech-based and text-based models was introduced in 2014 by @xcite , the gap of performance between these two types of inputs as documented in three iterations of the zero resource speech challenge focused on word segmentation has remained large @xcite . we attribute these difficulties to two major challenges posed by speech compared to text inputs: acoustic variability and temporal granularity, which we address in our contribution. the first and most important challenge is acoustic variability. in text, all tokens of a word are represented the same. in speech, each token of a word is different, depending on background noise, intonation, speaker voice, speech rate, and so forth. text-inspired speech segmentation algorithms @xcite apply a clustering step to word tokens in order to get back to word types. we believe that this step is unnecessary and is responsible for the low performance of existing systems, as errors in the clustering step are not recoverable and negatively impact word segmentation. we propose an algorithm that segments speech utterances based on an instance lexicon of word tokens, instead of a lexicon of word types. each speech segment instance of the training set is represented as a distinct memory trace, and these traces are used to estimate word form frequency using a k-nn algorithm without having to refer to a discrete word type but still applying the same dirichlet process logic. as for the representation of these word tokens, we follow recent approaches that use fixed length speech sequence embeddings (sse) @xcite . we use state-of-the-art sses from @xcite that have either been trained in a self-supervised fashion or with weak labels, in order to assess how the segmentation model behaves with inputs of increasing quality. the second challenge is related to the fact that the speech waveform being continuous in time, the number of possible segmentation points grows very large, making the optimization of segmentation of large speech corpora using bayesian techniques intractable. some models reduce the number of segmentation points using phoneme boundaries @xcite , syllable boundaries @xcite , or a constant discretization of the time axis into speech frames, subject to the following trade-off: too coarse frames may miss some short word boundaries, too fine-grained ones may render segmentation intractable on large corpora because of the quadratic increase in number of segmentations with frame rate. here we choose a compromise with 40ms frames, corresponding to half of the mean duration of a phoneme, yielding a 4x theoretical slowdown compared to phoneme-based transcripts. in addition, we introduce an accelerated version of a dirichlet process segmentation algorithm that replaces gibbs sampling of each boundary by sampling entire parse trees using dynamic programming beam search, and updat-ing the lexicon in batches instead of continuously. this achieves a 10-times speed-up compared to @xcite 's implementation, with similar or superior performances. because of our 'no clustering' approach, we cannot evaluate our model using the word typebased metrics of the zero resource speech segmentation challenge @xcite . here, we use the two classes of metrics: boundary-related metrics (token and boundary f-score) that do not refer to types, and high-level word embedding metrics. the idea is use a downstream language model to learn high level word embeddings and evaluate them on semantic and part-of-speech (pos) dimensions. we rely on a semantic similarity task from @xcite and introduce two new metrics. the combination of our contributions yields an overall system, dp-parse, that sets a new stateof-the-art on speech segmentation for five speech corpora by a large margin. by doing various ablation and anti-ablation studies (replacing unsupervised components by weakly supervised ones) we pinpoint the components of the system where there is still margin for improvement. in particular, we show that using better embeddings obtained with weak supervision, it is possible to double the segmentation f-score and substantially improve our new semantic and pos scores. ## method: instance-based dirichlet process parsing this section describes the building blocks and notations used by dp-parse as depicted in figure 1 and the list of all possible subsegments in c that are possible words (between 40ms and 800ms) l 0 = {e w0 , . . . , e w p } a knn index of embeddings from d @xmath2, . . . , e wn } a knn index of all embeddings from the segments in c seg l 0w the frequency of e w in l 0 l w the frequency of e w in l the probability p 0 w (w) that a given speech segment w is a word, and l w the frequency of a given word w in the (segmented) input corpus. they are estimated by building tables of counts of phoneme-n-grams and words given a segmentation, respectively. at step 2, these numbers are combined using equation 3 (section 3.3) and a new segmentation is sampled. our adjustments to the original algorithm are the following: so modified, the algorithm is no longer dependent on input type and can work either with discrete representations (text, represented as sequences of 1-hot vectors of phonemes) or continuous ones (speech, represented as embeddings). ## conclusion and open questions we introduce a new speech segmentation pipeline that sets the state-of-the-art on the zerospeech's datasets at 16.8 token f1. the whole pipeline needs no specific tuning of hyper-parameters, making it ready to use on any new languages. we showed that the problem of speech segmentation can be reduced to the problem of learning discriminative speech representations. indeed, using different level of supervision, our pipeline reaches up to 35 token f1 score. therefore, as long as the field of unsupervised representation learning makes headway, this method should automatically produce higher token f1 scores. a first avenue of improvement of the current work is in the sse component. here, we took the system described in algayres et al. (2022) out of the box, and we showed good performance in speech segmentation compared to the state of the art, but there was still a large margin of improvement compared to text-based system. a recent unpublished paper @xcite came to our attention based on the non-lexical principle and showed similar or slightly better results than ours on a subset of the zr17 language. kamper (2022) also uses a segmentation lattice that resembles ours for inference. yet, as we have shown, our system is monotonous with input embedding quality, and can therefore reach much better performance if the speech sequence embedding component were better trained. further work is needed to improve the sse component based on purely unsupervised methods. despite our computational speedup, our current system would face challenges to scale up the approach to larger datasets than librispeech. while it uses faiss, an optimized search library, it is unclear that storing each instance of a possible segment is necessary. one interesting avenue of research would therefore be to downsample both instance lexicons to alleviate space complexity and/or to trade storage with compute by recalculating the embeddings on-line. another interesting improvement would be to compute bigram probabilities instead of unigram probabilities, as previously explored in @xcite . yet, this possibility looks challenging, as explained in appendix a.2.3. finally, our method showed promising results in terms of semantic and pos encoding when taken as a preprocessor for a language model. such a phenomenon is displayed by our new in-context semantic tasks, but not by a previous without-context semantic task from @xcite . further work is needed to show whether this can translate into high quality generations when used as a component to generative systems @xcite . a appendix a.1 construction of triplets for abx sem and abx p os let's write the series of triplet from our abx tasks: (a i , b i , x i ) 0≤@xmath3. for all i, a i is defined as a tuple (r a i , s a i , e a i , t a i ) where r a i is a recording of a whole sentence and s a i and e a i are the temporal boundaries of a word with phonetic transcription t a i . b i and x i are defined identically. the sentences are extracted from the librispeech dataset, a 960-hour corpus of read english literature. the speech lm is asked to encode the whole sentences (r a i , r b i , r x i another unexpected result is the low impact of the concentration parameter (α 0 ). the value of this parameter should be a crucial as it controls the implicit vocabulary by controlling the amount of rare words in the segmentation. this observation has led us to create the penalty function q from equation 5 that offers a control on word-lengths, which also impacts the implicit vocabulary. this time, we noticed that q's shape strongly impact token-f1 scores. by a careful tuning of the penalty function, we increased the token-f1 from 16.4, with γ = 0 (i.e., no penalty function) to 18, with γ = 1.8 and δ = 4.