Worse WER , but Better BLEU ? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation (2020)

Abstract: Speech translation (ST) aims to learn transformations from speech in the source language to the text in the target language. Previous works show that multitask learning improves the ST performance, in which the recognition decoder generates the text of the source language, and the translation decoder obtains the final translations based on the output of the recognition decoder. Because whether the output of the recognition decoder has the correct semantics is more critical than its accuracy, we propose to improve the multitask ST model by utilizing word embedding as the intermediate.

## introduction speech translation (st) increasingly receives attention from the machine translation (mt) community recently. to learn the transformation between speech in the source language and the text in the target language, conventional models pipeline automatic speech recognition (asr) and text-to-text mt model @xcite . however, such pipeline systems suffer from error propagation. previous works show that deep end-to-end models can outperform conventional pipeline systems with sufficient training data @xcite @xcite . nevertheless, well-annotated bilingual data is expensive and hard to collect @xcite . multitask learning plays an essential role in leveraging a large amount of monolingual data to improve representation in st. multitask st models have two jointly learned decoding parts, namely the recognition and translation part. the recognition part firstly decodes the speech of source language into the text of source language, and then based on the output of the recognition part, the translation part generates the text in the target language. variant multitask models have been explored @xcite , which shows the improvement in low-resource scenario. although applying the text of source language as the intermediate information in multitask end-toend st empirically yielded improvement, we argue whether this is the optimal solution. even though the recognition part does not correctly transcribe the input speech into text, the final translation result would be correct if the output of the recognition part preserves sufficient semantic information for translation. therefore, we explore to leverage word embedding as the intermediate level instead of text. in this paper, we apply pre-trained word embedding as the intermediate level in the multitask st model. we propose to constrain the hidden states of the decoder of the recognition part to be close to the pre-trained word embedding. prior works on word embedding regression show improved results on mt @xcite . experimental results show that the proposed approach obtains improvement to the st model. further analysis also shows that constrained hidden states are approximately isospectral to word embedding space, indicating that the decoder achieves speech-to-semantic mappings. ## multitask end-to-end st model our method is based on the multitask learning for st @xcite , including speech recognition in the source language and translation in the target language, as shown in fig. 1(a) . the input audio feature sequence is first encoded into the encoder hidden state sequence @xmath0, h 2 , . . . , h t with length t by the pyramid encoder @xcite . to present speech recognition in the source language, the attention mechanism and a decoder is employed to produce source decoder sequence ŝ = ŝ1 , ŝ2 , . . . , ŝm , where m is the number of decoding steps in the source language. for each decoding step m, the probability p (ŷ m ) of predicting the token ŷm in the source language vocabulary can be computed based on the corresponding decoder state ŝm . given the ground truth sequence in the source language ŷ = ŷ1 , ŷ2 , . . . , ŷm and the target language @xmath1, y 2 , . . . , y q with length q, multitask st can be trained with maximizing log likelihood in both domains. formally, the objective function of multitask st can be written as: ## proposed methods we propose two ways to help the multitask endto-end st model capture the semantic relation between word tokens by leveraging the source language word embedding as intermediate level. ê = {ê 1 , ê2 , ...ê |v | }, where v is the vocabulary set and êv ∈ r d is the embedding vector with dimension d for any word @xmath2, in the recognition task. we choose the source language decoder state (embedding) ŝ to reinforce since it is later used in the translation task. to be more specific, we argue that the embedding generated by the source language decoder should be more semantically correct in order to benefit the translation task. given the pre-trained source language word embedding ê, we proposed to constrain the source decoder state ŝm at step m to be close to its corresponding word embedding êŷm with the two approaches detailed in the following sections. ## conclusions our proposals showed that utilizing word embedding as intermediate helps with the st task, and it is possible to map speech to the semantic space. we also observed that lower wer in source language recognition not imply higher bleu in target language translation. this work is the first attempt to utilize word embedding in the st task, and further techniques can be applied upon this idea. for example, crosslingual word embedding mapping methods can be considered within the st model to shorten the distance between mt and st tasks.