Phone Features Improve Speech Translation (2020)

Abstract: End-to-end models for speech translation (ST) more tightly couple speech recognition (ASR) and machine translation (MT) than a traditional cascade of separate ASR and MT models, with simpler model architectures and the potential for reduced error propagation. Their performance is often assumed to be superior, though in many conditions this is not yet the case. We compare cascaded and end-to-end models across high, medium, and low-resource conditions, and show that cascades remain stronger baselines. Further, we introduce two methods to incorporate phone features into ST models. We show that these features improve both architectures, closing the gap between end-to-end models and cascades, and outperforming previous academic work – by up to 9 BLEU on our low-resource setting.

## introduction end-to-end models have become the common approach for speech translation (st), but the performance gap between these models and a cascade of separately trained speech recognition (asr) and machine translation (mt) remains, particularly in low-resource conditions. models for low-resource asr leverage phone foot_0 information, but this information is not typically leveraged by current sequence-to-sequence asr or speech translation models. we propose two methods to incorporate phone features into current neural speech translation models. we explore the existing performance gap between endto-end and cascaded models, and show that incorporating phone features not only closes this gap, but greatly improves the performance and training efficiency of both model architectures, particularly in lower-resource conditions. the sequences of speech features used as input for st are ≈10 times longer than the equivalent sequence of characters in e.g. a text-based mt model. this impacts memory usage, the number of model parameters, and training time. multiple consecutive feature vectors can belong to the same phone, but the exact number depends on the phone and local context. further, these speech features are continuously valued rather than discrete, such that a given phone will have many different instantiations across a corpus. neural models learn to associate ranges of similarly valued feature vectors in a data-driven way, impacting performance in lower-resource conditions. using phoneme-level information provides explicit links about local and global similarities between speech features, allowing models to learn the task at hand more efficiently and yielding greater robustness to lower-resource conditions. we propose two simple heuristics to integrate phoneme-level information into neural speech translation models: (1) as a more robust intermediate representation in a cascade; and (2) as a concatenated embedding factor. we use the common fisher spanish-english dataset to compare with previous work, and simulate high-, mid-, and low-resource conditions to compare model performance across different data conditions. we compare to recent work using phone segmentation for end-to-end speech translation @xcite , and show that our methods outperform this model by up to 20 bleu on our lowest-resource condition. we add higher-level phone features to low-level speech features to improve our models' robustness across data conditions and training efficiency. we propose two methods to incorporate phone information into cascaded and end-to-end models, depicted in figure 1 . our phone cascade uses phone labels as the machine translation input, in place of the output transcription from a speech recognition model. our phone end-to-end model uses (2) a concatenated embedding factor in an end-to-end model. we additionally compare to previous work; model 1: phone cascade. in a cascade, the intermediate representation between asr and mt is the final output of a speech recognition model, e.g. characters, subwords, or words. using separate models for asr and mt means that errors made in asr are likely to propagate through mt. common errors include substitution of phonetically similar words, or misspellings due to irregularities in a language's orthography, the latter of which may be addressed by using phone labels in place of asr output. by not committing to orthographic targets, we believe this model will propagate fewer errors to downstream mt. model 2: phone end-to-end. our final model uses phone-factored embeddings, where trainable embeddings for phone features are concatenated to typical speech feature vector input. because phone durations are variable and typically span more than one filterbank feature (or frame), adjacent filterbank features may have the predicted phone label; in the example shown in figure 1 , /r/ spans three frames or filterbank features. we note that this method maintains the same source sequence length as the original speech feature sequence. this method associates similar feature vectors at the corpus level, because all filterbank features with the same phone alignment (e.g. /oh/) will have the same trainable phone embedding concatenated. in mt and ner, concatenating trainable embeddings for linguistic features to words, such as morphemes and phones, has improved models' ability to generalize @xcite . while these works appended finer-grained information to associate words with similar lower-level structure, we use phone embeddings to associate higher-level structure to similar but unique speech feature vectors globally across a corpus. model 3: phone segmentation. we compare to the method from @xcite as a strong end-to-end baseline. here, phone boundaries are used to segment and compress speech feature vector sequences. within each utterance, the feature vectors of consecutive speech frames with the same phone label are averaged to produce one feature vector for translation from a variable number of frames. this significantly reduces source sequence lengths (by ∼80%), reducing the number of model parameters and memory. rather than having a variable number of feature vectors per phone-like unit, each has one representation, more similar in granularity to character-based mt. the averaged feature vectors remain continuously-valued, and are locally summarized: a given phone across the corpus will still have different representations in each instance. ## data we use the fisher spanish-english corpus, foot_3 which consists of parallel speech, transcripts, and translations, enabling comparisons between cascaded and direct models on the same data and allowing us to generate phone supervision using matched data. the dataset contains 160 hours of spanish telephone speech, split into 138k utterances, which were translated via crowdsourcing by @xcite . we use the standard dev and test sets, each with ∼4k utterances. because we are particularly interested in how our methods will affect training across differently-resourced conditions, we compare results using randomly selected 40 hour and 20 hour subsets of the data. ## generating phone supervision to generate phoneme-level labels for sequences of speech features, we generate frame-level alignments using a trained speech recognizer. specifically, we extract 40-dimensional mel filterbank features with per-speaker mean and variance normalization using kaldi @xcite . we train an hmm/gmm system on the full fisher spanish dataset with the kaldi recipe @xcite , using the spanish callhome lexicon (ldc96l16), and compute per-frame phone alignments with the triphone model (tri3a) with lda+mllt features. this yields 50 phone labels, including silence (<@xmath0, noise, and laughter. producing phone alignments uses supervision from a transcript, which inherently does not exist at inference time. while phones can be extracted from kaldi lattices at inference time, we found that our hmm/gmm model was not our best performing asr model on this dataset -by greater than 10 wer. to leverage our better-performing neural asr models for phone generation, we create essentially a '2-pass' alignment procedure: first, generating a transcript, and second, using this transcript to force align phones. ## model & training procedure as in previous academic work on this corpus @xcite @xcite , we use a sequence-to-sequence architecture inspired by @xcite modified to train within lower resources; specifically, each model converges within ≈5 days on one gpu. we build encoder-decoder models with attention in xnmt @xcite with 512 hidden units. our pyramidal encoder uses 3-layer bilstms with linear network-in-network (nin) projections and batch normalization between layers @xcite . the nin projections are used to downsample by a factor of 2 between layers, resulting in the same total 4× downsampling in time as the additional convolutional layers from @xcite ; @xcite : they give us the benefit of added depth with fewer additional parameters. we use single layer mlp attention @xcite with 128 units and 1 decoder layer as opposed to 3 or 4 in previous workwe did not see consistent benefits from additional depth. in line with previous work on this dataset, all experiments preprocess target text by lowercasing and removing punctuation aside from apostrophes. we use 40-dimensional mel filterbank features as previous work did not see significant difference with higherdimensional features @xcite . we use 1k bpe units for translation text, shown in @xcite to have both better performance and training efficiency than characters @xcite or words @xcite . for both text and phones, we use 64-dimensional embeddings. for the mt component in cascaded speech translation models, we compared using the pyramidal speech architecture above (3 encoder, 1 decoder layers) to the traditional bilstm text model (2 layers each for encoder and decoder). using the pyramidal architecture resulted in the same performance as the bilstm model when translating bpe transcriptions from asr, but gave us consistent improvements of up to 1.5 bleu when instead translating phone sequences; we posit this is because phone sequences are longer than bpe equivalents. accordingly, we use the same model architecture for all our asr, mt, and st models. we use layer dropout with @xmath1.2 and target embedding dropout with @xmath2.1 @xcite . we apply label smoothing with @xmath3.1 @xcite and fix the target embedding norm to 1 @xcite . for inference, we use beam of size 15 and length normalization with exponent 1.5. we set the batch size dynamically depending on the input sequence length with average batch size was 36. we use adam (kingma and ba, 2015) with initial learning rate 0.0003, decayed by 0.5 when validation bleu did not improve for 10 epochs initially and subsequently 5 epochs. we do not use l2 weight decay or gaussian noise, and use a single model replica. we use input feeding @xcite , and exclude utterances longer than 1500 frames in training for memory. ## prior work: cascaded vs end-to-end models on fisher spanish-english the large body of research on the fisher spanish-english dataset, including both cascaded and end-to-end models, makes it a good benchmark to compare these architectures. not all previous work has compared across multiple resource settings or compared to cascaded models, which we address in this section. we summarize best previous results on this dataset on high, medium, and low-resource conditions in best results. the cascade of traditional hmm/dnn asr and joshua mt models from kumar et al. (2014) set a competitive baseline on the full dataset (40.4 test bleu) which no subsequent academic models have been able to match until this work; subsequent exploration of end-to-end models has produced notable relative improvements but the best end-to-end academic number (salesky et al., 2019) remains 1.6 bleu behind this traditional cascade. industry models from weiss et al. ( 2017 ) achieved exceptional performance with very deep end-to-end models on the full dataset (47.3 test bleu), exceeding a cascade for the first time. they additionally show results with an updated cascade using neural models, improving over @xcite . their results have been previously unmet by the rest of the community. this is likely in part due to the computational resources required to fully explore training schedules and hyperparameters with models of their depth. while their asr models took ∼4 days to converge, their st models took another 2 weeks, compared to the lighter-weight models of recent academic work which converged @xmath4. this dataset is challenging: improving asr wer from 35 @xcite to 23 @xcite only resulted in 4 bleu st improvement: see components in ## results using phone features we compare our two ways to leverage phone features to our cascaded and end-to-end baselines across three resource conditions. phone features. @xcite performs most similarly to the end-to-end baseline, but nonetheless represents an average relative improvement of 13% across the three data sizes with a significant reduction in training time. our phone featured models use not just the phone segmentation, but also the phone labels, and perform significantly better. our phone end-to-end model not only shows less of a decrease in performance across cascaded models in orange, end-to-end models in purple. our proposed models yield improvements across all three conditions, with a widening margin under low-resource conditions for the phone cascade. ## model robustness & further analysis in this section, we analyze the robustness of each of our models by varying the quality of our phone features, and further explore the strengths and limitations of each model. ## related work speech translation was initially performed by cascading separately trained asr and mt models, allowing each model to be trained on larger data sources without parallel speech, transcriptions, and translations, but potentially yielding unrecoverable errors between models. linking models through lattices with both phrase-based @xcite and neural mt @xcite reduced many such errors. using one model to directly translate speech was later enabled by attentional encoder-decoder models. direct end-to-end speech translation was first explored as a way to reduce both error propagation, and also the need for high quality intermediate transcriptions (e.g. for unwritten languages). the first such models were investigated in @xcite ; @xcite , but these used, respectively, a small synthetic corpus and evaluated on speech-to-text alignments rather than translation. subsequently @xcite extended these neural attentional models to deep, multitask models with excellent results on fisher spanish-english, exceeding a cascade for the first time. however, efforts from the community have not yet replicated their success @xcite @xcite . end-to-end models have performed inconsistently compared to cascades on other corpora: @xcite perform well on high-resource audiobooks but do not exceed a cascade; @xcite found 'triangle' models performed better than cascades for 2 of 3 very low-resource language pairs; and in the most recent iwslt evaluation campaigns, cascades have remained the highest-performing systems @xcite . similarly-motivated work exists in speech translation. in addition to @xcite al. (2019); sperber et al. (2019) addressed above, preliminary cascades using phone-like units have been explored for low-resource speech translation, motivated by translation of unwritten languages where a traditional cascade would not be possible. to this end, @xcite utilized unsupervised term discovery, and @xcite synthesized speech; but these approaches were only evaluated in terms of precision and recall and were not tested on both 'higher-resource' and natural speech data conditions. ## conclusion we show that phone features significantly improve the performance and data efficiency of neural speech translation models. we study the existing performance gap between cascaded and end-to-end models, and introduce two methods to use phoneme-level features in both architectures. our improvements hold across high, medium, and low-resource conditions. our greatest improvements are seen in our lowest-resource settings (20 hours), where our end-to-end model outperforms a strong baseline cascade by ≈5 bleu, and our cascade outperforms prior work by ≈9 bleu. generating phone features uses the same data as auxiliary speech recognition tasks from prior work; our experiments suggest these features are a more effective use of this data, with our models matching the performance from previous works' performance without additional training data. we hope that these model comparisons and results inform development of more robust end-to-end models, and provide a stronger benchmark for performance on low-resource settings.