CTC -based Compression for Direct Speech Translation (2021)

Abstract: Previous studies demonstrated that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). However, they required a dedicated model for phone recognition and did not test this solution for direct ST, in which a single model translates the input audio into the target language without intermediate representations. In this work, we propose the first method able to perform a dynamic compression of the input in direct ST models. In particular, we exploit the Connectionist Temporal Classification (CTC) to compress the input sequence according to its phonetic characteristics. Our experiments demonstrate that our solution brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German), contextually reducing the memory footprint by more than 10%.

## introduction speech translation (st) is the process that converts utterances in one language into text in another language. traditional approaches to st consist of separate modules, each dedicated to an easier subtask, which are eventually integrated in a so-called cascade architecture @xcite . usually, its main components are an automatic speech recognition (asr) model -which generates the transcripts from the audio -and a machine translation (mt) model -which translates the transcripts into the target language. a newer approach is direct st, in which a single model performs the whole task without intermediate representations @xcite . the main advantages of direct st systems are: i) the access to information not present in the text (e.g. prosody, vocal characteristics of the speaker) during the translation phase, ii) a reduced latency, iii) a simpler and easier to manage architec-ture (only one model has to be maintained), which iv) avoids error propagation across components. in both paradigms (cascade and direct), the audio is commonly represented as a sequence of vectors obtained with a mel filter bank. these vectors are collected with a high frequency, typically one every 10 ms. the resulting sequences are much longer than the corresponding textual ones (usually by a factor of ~10). the sequence length is problematic both for rnn @xcite and transformer @xcite architectures. indeed, rnns fail to represent long-range dependencies @xcite and the transformer has a quadratic memory complexity in the input sequence length, which makes training on long sequences prohibitive due to its memory footprint. for this reason, architectures proposed for direct st/asr reduce the input length either with convolutional layers @xcite or by stacking and downsampling consecutive samples @xcite . however, these fixed-length reductions of the input sequence assume that samples carry the same amount of information. this does not necessarily hold true, as phonetic features vary at a different speed in time and frequency in the audio signals. consequently, researchers have studied how to reduce the input length according to dynamic criteria based on the audio content. salesky et al. (2019) demonstrated that a phoneme-based compression of the input frames yields significant gains compared to fixed length reduction. phone-based and linguistically-informed compression also proved to be useful in the context of visually grounded speech @xcite . however, @xcite questioned the approach, claiming that the addition of phone features without segmentation and compression of the input is more effective. none of these works is a direct st solution, as they all require a separate model for phone recog-nition and intermediate representations. so, they: i) are affected by error propagation (salesky and black 2020 show in fact that lower quality in phone recognition significantly degrades final st performance), ii) have higher latency and iii) a more complex architecture. a direct model with phonebased multi-task training was introduced by @xcite for speech-to-speech translation, but they neither compared with a training using transcripts nor investigated dynamic compression. in this paper, we explore the usage of phones and dynamic content-based input compression for direct st (and asr). our goal is an input reduction that, limiting the amount of redundant/useless information, yields better performance and lower memory consumption at the same time. to this aim, we propose to exploit the connectionist temporal classification (ctc) @xcite to add phones prediction in a multi-task training and compress the sequence accordingly. to disentangle the contribution of the introduction of phone recognition and the compression based on it, we compare against similar trainings leveraging transcripts instead of phones. our results show that phone-based multi-task training with sequence compression improves over a strong baseline by up to 1.5 bleu points on two language pairs (english-german and english-italian), with a memory footprint reduction of at least 10%. ## ctc-based sequence compression the ctc algorithm is usually employed for training a model to predict an output sequence of variable length that is shorter than the input one. this is the case of speech/phone recognition, as the input is a long sequence of audio samples, while the output is the sequence of uttered symbols (e.g. phones, sub-words), which is significantly shorter. in particular, for each time step, the ctc produces a probability distribution over the possible target labels augmented with a @xmath0. these distributions are then exploited to compute the probabilities of different sequences, in which consecutive equal predictions are collapsed @xmath1. finally, the resulting sequences are compared with the target sequence. adding an auxiliary ctc loss to the training of direct st and acoustic asr models has been shown to improve performance @xcite . in these works, the ctc loss is computed against the transcripts on the encoder output to favour model convergence. generally, the ctc loss can be added to the output of any encoder layer, as in figure 1 where the hyper-parameter n ctc indicates the number of the layer at which the ctc is computed. formally, the final loss function is: if n ctc is equal to the number of encoder layers (n e ), the ctc input is the encoder output. we consider this solution as our baseline and we also test it with phones as target. as shown in figure 1 , we use as model a transformer, whose encoder layers are preceded by two 2d convolutional layers that reduce the input size by a factor of 4. therefore, the ctc produces a prediction every 4 input time frames. the sequence length reduction is necessary both because it makes possible the training (otherwise out of memory errors would occur) and to have a fair comparison with modern state-of-the-art models. a logarithmic distance penalty @xcite is added to all the transformer encoder layers. our proposed architecture is represented in figure 2 . the difference with the baseline is the introduction of an additional block (collapse same predictions) that exploits the ctc predictions to compress the input elements (vectors). hence, in this case the ctc does not only help model convergence, but it also defines variable-length segments representing the same content. so, dense audio portions can be given more importance, while redundant/uninformative vectors can be compressed. this allows the following encoder layers and the decoder to attend to useful information without being "distracted" by noisy elements. the architecture is a direct st solution as there is a single model whose parameters are optimized together without intermediate representations. at inference time, the only input is the audio and the model produces the translation into the target language (contextually generating the transcripts/phones with the ctc). we compare three techniques to compress the consecutive vectors with the same ctc prediction: ## data we experiment with must-c @xcite , a multilingual st corpus built from ted talks. we focus on the english-italian (465 hours) and english-german (408 hours) sections. for each set (train, validation, test), it contains the audio files, the transcripts, the translations and a yaml file with the start time and duration of the segments. in addition, we extract the phones using gentle. 1 besides aligning the transcripts with the audio, gentle returns the start and end time for each recognized word, together with the corresponding phones. for the words not recognized in the audio, gentle does not provide the phones, so we lookup their phonetic transcription on the voxforge foot_1 dictionary. for each sample in the corpus, we rely on the yaml file and the alignments generated by gentle to get all the words (and phones) belonging to it. the phones have a suffix indicating the position in a word (at the end, at the beginning, in the middle or standalone). we also generated a version without the suffix (we refer to it as ph w/o pos in the rest of the paper). the resulting dictionaries contain respectively 144 and 48 symbols. ## experimental settings our transformer layers have 8 attention heads, 512 features for the attention and 2,048 hidden units in ffn. we set a 0.2 dropout and include specaugment @xcite in our trainings. we optimize label smoothed cross entropy @xcite with 0.1 smoothing factor using adam @xcite (betas (0.9, 0.98)). the learning rate increases linearly from 3e-4 to 5e-3 for 4,000 updates, then decays with the inverse square root. as we train on 8 gpus with minibatches of 8 sentences and we update the model every 8 steps, the resulting batch size is 512. the audio is pre-processed performing speaker normalization and extracting 40-channel mel filter-bank features per frame. the text is tokenized into subwords with 1,000 bpe merge rules @xcite . as having more encoder layers than decoder layers has been shown to be beneficial @xcite , we use 8 transformer encoder layers and 6 decoder layers for asr and 11 encoder and 4 decoder layers for st unless stated otherwise. we train until the model does not improve on the validation set for 5 epochs and we average the last 5 checkpoints. trainings were performed on k80 gpus and lasted ~48 hours (~50 minutes per epoch). our implementation foot_2 is based on fairseq @xcite . we evaluate performance with wer for asr and with bleu @xcite 4 and sacre-bleu @xcite 5 results ## conclusions as researchers' focus is shifting from cascade to direct solutions due to the advantages of the latter, we proposed a technique of dynamic sequence-length reduction for direct st. we showed that averaging the vectors corresponding to the same phone prediction according to the ctc improves the translation quality and reduces the memory footprint, allowing for training deeper models. our best model outperforms a strong baseline, which uses transcripts in a multi-task training, by 1.3 (en-it) and 1.5 (en-de) bleu, reducing memory usage by 10-16%.