XLS - R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words (2023)

Abstract: Due to the absence of explicit word boundaries in the speech stream, the task of segmenting spoken sentences into word units without text supervision is particularly challenging. In this work, we leverage the most recent self-supervised speech models that have proved to quickly adapt to new tasks through fine-tuning, even in low resource conditions. Taking inspiration from semi-supervised learning, we fine-tune an XLS-R model to predict word boundaries themselves produced by top-tier speech segmentation systems: DPDP, VG-HuBERT and DP-Parse. Once XLS-R is fine-tuned, it is used to infer new word boundary labels that are used in turn for another fine-tuning step. Our method consistently improves the performance of each system and set a new state-of-the-art that is, on average 130% higher than the previous one as measured by the F1 score on correctly discovered word tokens on five corpora featuring different languages. Finally, our system can segment speech from languages unseen during fine-tuning in a zero-shot fashion.

## method let us use a speech dataset c, a pre-trained speech ssl model w , and an off-the-shelf speech segmentation system s. on top of w is added a random feed-forward layer with one neuron and a sigmoid activation. here is our method to train w on boundaries produced by s. first, s is used to infer word boundaries for every spoken sentence in c. in addition, spoken sentences are data-augmented with a random quantity of reverb, pitch, time-stretch and time-drop and then encoded into a series of frames by w . for each output frame, we create a label that is either 1 if the frame aligns with a word boundary or 0 if not. because word boundaries are, by nature, not clearly defined in the time domain, we label as 1 the left and right neighbouring frames of every frame that has been already tagged as 1. w is fine-tuned with back-propagation by minimizing the negative cross entropy between w 's output and the labels. in a sentence, most of the frames are labelled as zeros (for 'not a boundary'), and the loss is particularly low on those frames. to force the model to focus on harder sections of the input, we only backpropagate the loss on the top 50% frames with the highest loss. at inference, for a given sentence, w produces at each frame the probability of discovering a boundary. to decide which frames should be labelled as a boundary, we apply a peak-detection method that finds local maxima by comparison of neighbouring probability values. this function has two hyperparameters: maximal height of peaks and minimal distance between two peaks. we fit these hyperparameters on the development set by maximizing the f1 scores between the boundaries produced by s and the new boundaries produced by w . at this stage, w can be used to infer on the dataset c a set of new word boundary labels. w is set back to its initial unfine-tuned state and is fine-tuned again on the new word boundaries. this process is iterated until segmentation performances start to decrease. ## datasets, evaluation and hyperparameters the metric that we use to evaluate performance is the token-f1 score which is the f1 score on correctly discovered tokens. a token is correctly discovered when both its boundaries correspond to the boundaries of a word in the time-aligned transcription. this metric was introduced by the zerospeech challenge 2017 @xcite and is computed with the tdev2 library. the datasets used in this work are the five corpora introduced in the zerospeech challenge 2017 (ref): mandarin (2h), french (20h), english (30h), german (19h30min), wolof (2h43min). the spoken sentences are split into voice activity detections (vad) (i.e. sequences that only contain speech). the datasets are not split into train/test because they are meant for unsupervised learning. yet, in order to fit hyper-parameters, the five corpora are split into two sets: mandarin, french and english are development datasets where hyper-parameters are tuned and german and wolof are test datasets where the model is tested to show generalization to new languages. we performed a grid-search on hyper-parameters on the development datasets to maximise the token-f1. during our fine-tuning step, we use backpropagation on batches that contain 12 spoken sentences of a maximum of 20 seconds each. the xls-r is fine-tuned on a single 32go gpu for a maximum of 2000 updates, after which we keep the model with the lowest loss on the development set. optimization is done with adam optimizer (kingma and ba, 2017), the learning rate is warmed up from 0 to 10 - foot_2 and then decayed back to 0 with a cosine annealing @xcite with period 10 3 . we freeze the convolutional front end and use 10% dropout, 15% layer-drop, and 15% masked frames. data augmentation is done mainly with the wavaugment library. we use the values of parameters advised by the authors kharitonov et al. (2020): for reverb, we sampled the room scale uniformly in [0,100] while keeping the other parameters unchanged and for pitch, we pick a value uniformly in the range @xcite . time-stretch coefficients are uniformly sampled between 0.8 and 1. ## conclusion in this work, we propose an unsupervised speech segmentation system that fine-tunes xls-r on boundaries provided by an external off-the-shelf speech segmentation system. our method increases word segmentation performances by 130% compared to the previous state-of-the-art. our method also shows high performances in the zero-shot setting, which suggest that universal segmentation features exist in the speech signal.regarding interpretation, our results are sometimes hard to explain. even though we proved that high initial f1 scores (obtained with weak supervision) do lead to better performances, more work is needed on the evaluation of the speech segmentation model to understand what characteristics are beneficial to xls-r finetuning but that are not captured by f1 scores.