On the Difficulty of Segmenting Words with Attention (2021)

Abstract: Word segmentation, the problem of finding word boundaries in speech, is of interest for a range of tasks. Previous papers have suggested that for sequence-to-sequence models trained on tasks such as speech translation or speech recognition, attention can be used to locate and segment the words. We show, however, that even on monolingual data this approach is brittle. In our experiments with different input types, data sizes, and segmentation algorithms, only models trained to predict phones from words succeed in the task. Models trained to predict words from either phones or speech (i.e., the opposite direction needed to generalize to new data), yield much worse results, suggesting that attention-based segmentation is only useful in limited scenarios.

## introduction word segmentation is the task of finding word boundaries in speech. the task has a wide range of applications, including documenting underresourced languages @xcite and bootstrapping speech recognizers @xcite . it is often the first step to a variety of unsupervised speech tasks @xcite @xcite and to the nlp pipeline for languages with no whitespace between words. while unsupervised speech segmentation has been studied @xcite , in many cases a parallel data source may be available, such as transcriptions, translations, or images. previous researchers have suggested that it is possible to extract word segments from the attention map created by training an end-toend sequence-to-sequence model on such parallel data @xcite @xcite @xcite . however, palaskar and metze's evaluation was non-standard foot_0 and the others (which we refer to collectively henceforth as boitoea) used models with text translations on the input side-decoding to phones or phone-like units-which means the trained models cannot be applied to segment novel (untranslated) sequences. in addition, the interpretation of attention as alignments in other areas of nlp has been questioned @xcite . prompted by the prior work, we set out to study word segmentation with attention in sequence-tosequence models, aiming to better understand when and how attention can be interpreted as alignments and in what settings it can be used to segment words. our main experiments follow boitoea in performing and evaluating word segmentation on the same data used to train the sequence-tosequence model. however, instead of training translation models as in boitoea, we train models to perform speech recognition: a well-studied task with a simpler (monotonic) alignment structure. this setting is similar to forced alignment, where both the speech and the transcription are given, and the goal is to discover the hidden alignments. aside from this potential use case, this setting is useful for analysis because it abstracts away from the need to generalize to a novel test set. if the attention is not able to provide acceptable word boundaries in this setting, then the approach is unlikely to succeed in other, more difficult, settings. we perform experiments on both the lowresource mboshi dataset from boitoea and a much larger english dataset, @xcite . we study models trained in both directions with a variety of input-output types (e.g., phones, speech frames) and different postprocessing strategies to extract alignments from the atten-tion weights. we find that although the particular configuration used by boitoea works well, in most configurations the word segments extracted from the attention map are poor, even when using a larger dataset or model size. in particular, we did not get good results from any of the configurations that can be applied to a novel test set or to speech frames (rather than phone-like units). we conclude that even in the simple monotonic case studied here, attention typically does not provide clear word-level alignments. ## problem setting we consider an n-sample dataset @xmath0, y 1 , z 1 ), . . . , (x n , y n , z n )}, each of which is a triplet (x, y, z) ∈ x × y × z. as an example, x is the set of sequences of speech frames, y is the set of sequences of words, and z is the set of sequences of segments, where a segment is a triplet (s, t, w) that indicates the start time s, the end time t, and the word w. the goal is to learn a function f : x × @xmath1, y 1 ), . . . , (x n , y n )}, i.e., discovering the alignments without observing them. formally, we aim to find f that minimizes n @xmath2, y i ), z i ) using only s| xy , for some loss function ℓ that evaluates the quality of the segmentation. note that the evaluation of f is on the set s. we could evaluate f on a test set, but, in general, generalization is not involved in this setting. 3 the setting is general, subsuming many tasks. when x is the set of speech utterances and y is the empty set, this is the usual unsupervised word segmentation. the set y can be images or translations, grounding words from other modalities @xcite . in this work, we focus on y being transcriptions, i.e., we have a forced alignment task. ## word segmentation with attention our pipeline, due to boitoea, consists of two steps: generating an attention map from a sequenceto-sequence model, followed by postprocessing to convert the map into an alignment. ## experiments most of our experiments are conducted on the mboshi dataset. it contains 4616 short read-speech utterances for training (3 seconds/6 words on average; 4.5h in total), with a vocabulary of 6638 words, and 514 utterances for development. mboshi is a bantu language with no orthography, and the speech is transcribed at the word level using a phonetic orthography designed by linguists. we regard the basic units in the transcriptions as phones. 4 we do not use the french translations of the utterances. for experiments with speech, we use kaldi to extract speech features, with a 25 ms window shifted by 10 ms. each acoustic frame consists of 40dimensional log mel features and 3-dimensional pitch features. the acoustic feature vectors are used directly, we do no clustering or acoustic unit discovery. we use a 1-layer bidirectional lstm encoder and a 1-layer unidirectional lstm decoder, foot_3 with 0.5 dropout on the encoder and a 256-dimensional hidden layer. 6 further hyperparameter details are in appendix a.2. as noted above, our main questions do not require testing generalization, so except where otherwise noted (to compare to previous work), we evaluate all models on the training set. results on the development set, which do not change the story, are reported in appendix a.3. we report precision, recall, and f-score of the hypothesized word boundaries. when transcribing phones to words, a ## conclusion previous researchers had suggested a connection between attention weights and word alignments in both speech recognition and speech translation. however, we have experimented with several attention-based segmentation methods and demonstrated that these only succeed in the scenario where words are used as the input to the modela scenario with limited application. performance drops considerably for models with phones as input, and is no better than unsupervised segmentation for models using speech as input, even when the amount of training data is increased by two orders and the scheduler applies a decay factor of 0.5 after two consecutive epochs where loss does not decrease. all models are trained until cross-entropy loss on training reaches 0. the implementation of each model has around 3m and 19m of learnable parameters for the 1 and 5 layers encoder model, respectively. they are trained with one nvidia geforce gtx 1080 ti. to reduce computation on segmental assignment, we set the maximum duration of a word to 4 seconds (400 frames for speech or phone frames representation) for mboshi and 10 seconds for must-c. we set them by analyzing their performance on the development set. regarding the unsupervised speech baseline models, we use the unigram public implementation of @xcite 9 with a minimum word segment duration of 250 ms. because its performance is linked to the syllable segmentation method, we select the best configuration by fine-tuning räsänen et al. (2015