Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation (2020)

Abstract: We introduce dual-decoder Transformer, a new model architecture that jointly performs automatic speech recognition (ASR) and multilingual speech translation (ST). Our models are based on the original Transformer architecture (Vaswani et al., 2017) but consist of two decoders, each responsible for one task (ASR or ST). Our major contribution lies in how these decoders interact with each other: one decoder can attend to different information sources from the other via a dual-attention mechanism. We propose two variants of these architectures corresponding to two different levels of dependencies between the decoders, called the parallel and cross dual-decoder Transformers, respectively. Extensive experiments on the MuST-C dataset show that our models outperform the previously-reported highest translation performance in the multilingual settings, and outperform as well bilingual one-to-one results. Furthermore, our parallel models demonstrate no trade-off between ASR and ST compared to the vanilla multi-task architecture. Our code and pre-trained models are available at https://github.com/formiel/speech-translation .

## introduction while cascade speech-to-text translation (st) systems operate in two steps: source language automatic speech recognition (asr) and source-to-target text machine translation (mt), recent works have attempted to build end-to-end st without using source language transcription during decoding @xcite @xcite . after two years of extensions to these pioneering works, the last results of the iwslt 2020 shared task on offline speech translation @xcite demonstrate that end-to-end models are now on par (if not better) than their cascade counterparts. such a finding motivates even more strongly the works on multilingual (one-to-many, many-to-one, many-tomany) @xcite @xcite for which end-to-end models are well adapted by design. moreover, of these two approaches: cascade proposes a very loose integration of asr and mt (even if lattices or word confusion networks were used between asr and mt before end-to-end models appeared) while most end-to-end approaches simply ignore asr subtask, trying to directly translate from source speech to target text. we believe that these are two edge design choices and that a tighter coupling of asr and mt is desirable for future end-to-end st applications, in which the display of transcripts alongside translations can be beneficial to the users @xcite . this paper addresses multilingual st and investigates more closely the interactions between speech transcription (asr) and speech translation (st) in a multilingual end-to-end architecture based on transformer. while those interactions were previously investigated as a simple multi-task framework for a bilingual case @xcite , we propose a dual-decoder with an asr decoder tightly coupled with an st decoder and evaluate its effectiveness on one-to-many st. our model is inspired by @xcite , but the interaction between asr and st decoders is much tighter. 1 finally, experiments show that our model outperforms theirs on the must-c benchmark @xcite . our contributions are summarized as follows: (1) a new model architecture for joint asr and multilingual st; (2) an integrated beam search decoding strategy which jointly transcribes and translates, and that is extended to a wait-k strategy where the asr hypothesis is ahead of the st hypothesis by k tokens and vice-versa; and (3) competitive performance on the must-c dataset in both bilingual and multilingual settings and improvements on previous joint asr/st work. joint asr and st joint asr and st decoding was first proposed by @xcite through a multi-task learning framework. @xcite improve multitask st by using word embedding as an intermediate level instead of text. a two-stage model that performs first asr and then passes the decoder states as input to a second st model was also studied previously @xcite . this architecture is closer to cascaded translation while maintaining end-to-end trainability. @xcite introduce the notion of consistency between transcripts and translations and propose metrics to gauge it. they evaluate different model types for the joint asr and st task and conclude that end-to-end models with coupled inference procedure are able to achieve strong consistency. in addition to existing models having coupled architectures, they also investigate a model where the transcripts are concatenated to the translations, and the shared encoder-decoder network learns to predict this concatenated outputs. it should be noted that our models have lower latency compared to this approach since the concatenation of outputs makes the two tasks sequential in nature. our work is closely related to that of @xcite who propose an interactive attention mechanism which enables asr and st to be performed synchronously. both asr and st decoders do not only rely on their previous outputs but also on the outputs predicted in the other task. we highlight three differences between their work and ours: (a) we propose a more general framework in which @xcite ## dual-decoder transformer for joint asr and multilingual st we now present the proposed dual-decoder transformer for jointly performing asr and multilingual st. our models are based on the transformer architecture @xcite but consist of two decoders. each decoder is responsible for one task (asr or st). the intuition is that the problem at hand consists in solving two different tasks with different characteristics and different levels of difficulty (multilingual st is considered more difficult than asr). having different decoders specialized in different tasks may thus produce better results. in addition, since these two tasks can be complementary, it is natural to allow the decoders to help each other. therefore, in our models, we introduce a dual-attention mechanism: in addition to attending to the encoder, the decoders also attend to each other. ## conclusion we introduced a novel dual-decoder transformer architecture for synchronous speech recognition and multilingual speech translation. through a dual-attention mechanism, the decoders in this model are at the same time able to specialize in their tasks while being helpful to each other. the proposed model also generalizes previously proposed approaches using two independent (or weakly tied) decoders or chaining asr and st. it is also flexible enough to experiment with settings where asr is ahead of st which makes it promising for (one-to-many) simultaneous speech translation. experiments on the must-c dataset showed that our model achieved very competitive performance compared to state-of-the-art.