Adaptive Feature Selection for End-to-End Speech Translation (2020)

Abstract: Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to ASR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered features from the (frozen) ASR encoder. We take L0DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions. Results on LibriSpeech EnFr and MuST-C benchmarks show that AFS facilitates learning of ST by pruning out ~84% temporal features, yielding an average translation gain of ~1.3-1.6 BLEU and a decoding speedup of ~1.4x. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation).

## introduction end-to-end (e2e) speech translation (st), a paradigm that directly maps audio to a foreign text, has been gaining popularity recently @xcite @xcite @xcite . based on the attentional encoder-decoder framework @xcite , it optimizes model parameters under direct translation supervision. this end-toend paradigm avoids the problem of error propagation that is inherent in cascade models where an automatic speech recognition (asr) model and a machine translation (mt) model are chained together. nonetheless, previous work still reports that e2e st delivers inferior performance compared to cascade methods @xcite . we study one reason for the difficulty of training e2e st models, namely the uneven spread of information in the speech signal, as visualized in figure 1 , and the consequent difficulty of extracting informative features. features corresponding to uninformative signals, such as pauses or noise, increase the input length and bring in unmanageable noise for st. this increases the difficulty of learning @xcite and reduces translation performance. in this paper, we propose adaptive feature selection (afs) for st to explicitly eliminate uninformative features. figure 2 shows the overall architecture. we employ a pretrained asr encoder to induce contextual speech features, followed by an st encoder bridging the gap between speech and translation modalities. afs is inserted in-between them to select a subset of features for st encoding (see red rectangles in figure 1 ). to ensure that the selected features are well-aligned to transcriptions, we pretrain afs on asr. afs estimates the informativeness of each feature through a parameterized gate, and encourages the dropping of features (pushing the gate to 0) that contribute little to asr. an underlying assumption is that features irrelevant for asr are also unimportant for st. we base afs on l 0 drop @xcite , a sparsity-inducing method for encoder-decoder models, and extend it to sparsify speech features. the acoustic input of speech signals involves two dimensions: temporal and feature, where the latter one describes the spectrum extracted from time frames. accordingly, we adapt l 0 drop to sparsify encoder states along temporal and feature dimensions but using different gating networks. in contrast to @xcite , who focus on efficiency and report a trade-off between sparsity and quality for mt and summarization, we find that sparsity also improves translation quality for st. we conduct extensive experiments with transformer @xcite on librispeech en-fr and must-c speech translation tasks, covering 8 different language pairs. results show that afs only retains about 16% of temporal speech features, revealing heavy redundancy in speech encodings and yielding a decoding speedup of ∼1.4×. afs eases model convergence, and improves the translation quality by ∼1.3-1.6 bleu, surpassing several strong baselines. specifically, without data augmentation, afs narrows the performance gap against the cascade approach, and outperforms it on librispeech en-fr by 0.29 bleu, reaching 18.56. we compare against fixed-rate feature selection and a simple cnn, confirming that our adaptive feature selection offers better translation quality. our work demonstrates that e2e st suffers from redundant speech features, with sparsification bringing significant performance improvements. the e2e st task offers new opportunities for follow-up research in sparse models to deliver performance gains, apart from enhancing efficiency and/or interpretability. thus, l 0 drop can schedule hardconcrete via α i to put more probability mass at either 0 (i.e g @xmath0.e. g @xmath1. @xmath2. intuitively, l 0 drop controls the openness of gate g i via α i so as to determine whether to remove (g @xmath3. l 0 drop enforces sparsity by pushing the probability mass of hardconcrete towards 0, according to the following penalty term: by sampling g i with reparameterization (kingma and welling, 2013), l 0 drop is fully differentiable and optimized with an upper bound on the objective: l mle + λl 0 (x), where λ is a hyperparameter affecting the degree of sparsity -a larger λ enforces more gates near 0 -and l mle denotes the maximum likelihood loss. an estimation of the expected value of g i is used during inference. @xcite applied l 0 drop to prune encoder outputs for mt and summarization tasks; we adapt it to e2e st. sparse stochastic gates and l 0 relaxations were also by @xcite to construct interpretable classifiers, i.e. models that can reveal which tokens they rely on when making a prediction. ## adaptive feature selection one difficulty with applying encoder-decoder models to e2e st is deciding how to encode speech signals. in contrast to text where word boundaries can be easily identified, the spectrum features of speech are continuous, varying remarkably across different speakers for the same transcript. in addition, redundant information, like pauses in-between neighbouring words, can be of arbitrary duration at any position as shown in figure 1 , while contributing little to translation. this increases the burden and occupies the capacity of st encoder, leading to inferior performance @xcite . rather than developing complex encoder architectures, we resort to feature selection to explicitly clear out those uninformative speech features. figure 2 gives an overview of our model. we use a pretrained and frozen asr encoder to extract contextual speech features, and collect the informative ones from them via afs before transmission to the st encoder. afs drops pauses, noise and other uninformative features and retains features that are relevant for asr. we speculate that these retained features are also the most relevant for st, and that the sparser representation simplifies the learning problem for st, for example the learning of attention strength between encoder states and target language (sub)words. given a training tuple (audio, source transcription, translation), denoted as (x, y, z) respectively, foot_1 we outline the overall framework below, including three steps: ## conclusion and future work in this paper, we propose adaptive feature selection for e2e st to handle redundant and noisy speech signals. we insert afs in-between the st encoder and a pretrained, frozen asr encoder to filter out uninformative features contributing little to asr. we base afs on l 0 drop @xcite , and extend it to modeling both temporal and feature dimensions. results show that afs improves translation quality and accelerates decoding by ∼1.4× with an average temporal sparsity rate of ∼84%. afs successfully narrows or even closes the performance gap compared to cascading models. while most previous work on sparsity in nlp demonstrates its benefits from efficiency and/or interpretability perspectives @xcite , we show that sparsification in our scenario -e2e st -leads to substantial performance gains. in the future, we will work on adapting afs to simultaneous speech translation.