Non-Parametric Domain Adaptation for End-to-End Speech Translation (2022)

Abstract: The end-to-end speech translation (E2E-ST) has received increasing attention due to the potential of its less error propagation, lower latency and fewer parameters. However, the effectiveness of neural-based approaches to this task is severely limited by the available training corpus, especially for domain adaptation where in-domain triplet data is scarce or nonexistent. In this paper, we propose a novel non-parametric method that leverages in-domain text translation corpus to achieve domain adaptation for E2E-ST systems. To this end, we first incorporate an additional encoder into the pre-trained E2E-ST model to realize text translation modeling, based on which the decoder’s output representations for text and speech translation tasks are unified by reducing the correspondent representation mismatch in available triplet training data. During domain adaptation, a k-nearest-neighbor (kNN) classifier is introduced to produce the final translation distribution using the external datastore built by the domain-specific text translation corpus, while the universal output representation is adopted to perform a similarity search. Experiments on the Europarl-ST benchmark demonstrate that when in-domain text translation data is involved only, our proposed approach significantly improves baseline by 12.82 BLEU on average in all translation directions, even outperforming the strong in-domain fine-tuning strategy.

## introduction speech translation (st), the task of automatically translating speech signals in a given language into text in another language, becomes a widely studied topic with the increasing demand for international communications. traditional st systems cascade automatic speech recognition (asr) and machine translation (mt) @xcite @xcite ; iranzo-sánchez * equal contribution † corresponding author @xcite . so far, various large-scale speech-translation datasets have been proposed, e.g., @xcite , @xcite and cov-ost @xcite . with these large-scale annotations, building an end-to-end speech translation (e2e-st) system @xcite @xcite @xcite has become popular, since it has lower latency and less error propagation compared with previous st methods. recent studies have also shown that there is no significant difference between end-to-end models and cascaded systems in translation performance @xcite . in many practical application scenarios, such as political negotiations, business meetings, etc., there is no available in-domain speech-translation dataset to conduct the end-to-end training, which essentially limits the promotion of e2e-st systems. the most common practice is that the e2e-st model learns knowledge well enough in the general domain, and then it is directly used to translate speech input in the target domain. unfortunately, due to the domain shift issue @xcite , the generalization capabilities of current end-to-end models are somehow weak across different scenarios. instead of speechtranslation annotations, parallel text corpus in the target domain is usually abundant and easy to collect. thus, it is essential to explore and extend the capability of e2e-st systems in this scenario, in which a large amount of in-domain bilingual text is utilized. in this paper, we focus on this domain adaptation setting and attempt to replace the domain-specific parameter updating in neural-based e2e-st models with a non-parametric search to make it adaptable and achieve domain adaptation without any speech-translation annotations. actually, the nonparametric approach knn-mt, recently proposed by @xcite , is a promising alter-native to reach this goal. the knn-mt equips the pre-trained neural machine translation (nmt) model with a k-nearest-neighbor (knn) classifier over an external datastore to improve translation accuracy without retraining. however, it requires the in-domain speech-translation corpus to construct an effective datastore when we apply this method in the speech translation setting. to tackle this problem, we propose a novel non-parametric domain adaptation framework based on knn-mt for e2e-st, named as npda-knn-st. its key core is to directly leverage the in-domain text translation corpus to generate the corresponding datastore and encourage it to play a similar role as the real in-domain speech-translation data, through the carefully designed architecture and loss function. specifically, we first incorporate an additional trainable encoder for text modeling into the pretrained e2e-st model. based on that, we make the decoder's output representations for text and speech translation tasks close, through reducing the representation inconsistency of these two tasks in triplet training data and keeping the parameters of the original pre-trained e2e-st model fixed. in this way, the additional encoder module learns the semantic mapping in feature space from the source language text to the speech signal, which enables the construction of an effective in-domain datastore when text translation data is involved only. then we introduce a knn classifier to produce the final translation distribution based on the domainspecific datastore built by the correspondent text translation data. meanwhile, the universal output representation is adopted to perform a similarity search and guides the translation process. we evaluate our approach on the europarl-st benchmark and demonstrate that our method significantly outperforms the strong in-domain finetuning strategy by 3.85 bleu scores on average in all translation directions when only using largescale in-domain text translation data. additional experiments on europarl-st and must-c datasets verify that the in-domain text translation datastore generated by our method could play a similar role with the real in-domain speech-translation data, thanks to the universal output representation. ## conclusion in this paper, we present a novel non-parametric method that leverages in-domain bilingual text to achieve domain adaptation for the e2e-st system. this approach builds the universal output representation for text and speech translation tasks by a carefully designed architecture and loss function. based on that, a knn classifier is introduced to improve translation performance with an external datastore constructed by the in-domain text translation data. experimental results demonstrate that our proposed method obtains significant improvement over pre-trained e2e-st models when using large-scale in-domain bilingual text corpus. in the future, we would like to explore the combination of our method and the large-scale e2e-st model, such as lna-d.