P rompt ST : Abstract Prompt Learning for End-to-End Speech Translation (2023)

Abstract: An end-to-end speech-to-text (S2T) translation model is usually initialized from a pre-trained speech recognition encoder and a pre-trained text-to-text (T2T) translation decoder. Although this straightforward setting has been shown empirically successful, there do not exist clear answers to the research questions: 1) how are speech and text modalities fused in S2T model and 2) how to better fuse the two modalities? In this paper, we take the first step toward understanding the fusion of speech and text features in S2T model. We first design and release a 10GB linguistic probing benchmark, namely Speech-Senteval, to investigate the acoustic and linguistic behaviors of S2T models. Preliminary analysis reveals that the uppermost encoder layers of the S2T model can not learn linguistic knowledge efficiently, which is crucial for accurate translation. Based on the finding, we further propose a simple plug-in prompt-learning strategy on the uppermost encoder layers to broaden the abstract representation power of the encoder of S2T models. We call such a prompt-enhanced S2T model PromptST. Experimental results on four widely-used S2T datasets show that PromptST can deliver significant improvements over a strong baseline by capturing richer linguistic knowledge. Benchmarks, code, and scripts are freely available at https://github.com/ytf-philp/PromptST.

## introduction different from a cascade of separately trained automatic speech recognition (asr, @xcite and machine translation @xcite ) models, end-to-end speech-to-text translation (s2t, @xcite directly translates source-language acoustic speech * work was done when tengfei was interning at jd explore academy. different from the modal fusion in the input, this paper takes the first step toward understanding the fusion of speech and text features in the s2t model. specifically, we design a fine-grained linguistic probing benchmark, namely speech-senteval, for the s2t model following @xcite . considering the information flow transferring from audio to text, the encoder of the s2t model probably learns to extract, align, and fuse acoustic features at the lower layers and then turns to learn the important knowledge, e.g., linguistic properties, to achieve translation at the high-level layers. to fur-ther understand this, we investigate a fine-grained comparison of acoustic and linguistic analysis and leave the vanilla text probing benchmark for the t2t model. the result shows that the pre-trained s2t model fails to effectively learn linguistic information at the high-level layers, which brings a gap between speech and text in end-to-end s2t models. motivated by these findings, we argue that broadening the representation power of the high-level layers for the pre-trained s2t models is at the core of achieving better performance. plenty of works have shown that translation models can largely benefit from the enriched representation @xcite @xcite @xcite @xcite . to this end, we design a strategy, abstract prompt, to augment the high-level layers of the pre-trained s2t model, leveraging prompt-learning methods @xcite . this strategy is named promptst due to its succinct and plugin properties. we experiment with the method in a widely used pre-trained s2t model @xcite on covost-2 data sets, spanning english-german, english-catalan, english-arabic, and english-turkish language pairs. promptst consistently and significantly outperforms a strong baseline by an average of +0.4 bleu. our main contributions are as follows: interpreting the neural network models probing task is often designed to facilitate comparison between different models at a fine-grained level. for example, yang et al. (2019) design a word reordering detection task to evaluate a model's ability to extract word order information. rogers et al. (2021); lin et al. (2019) analyze the hierarchy of linguistic information in transformer encoder's hidden representation. shi et al. (2016) assess the learned representations in machine translation models for syntactic knowledge (wang et al., 2023). conneau et al. (2018) introduce a set of tasks to probe the linguistic knowledge encoded in sentence embeddings. these methods help open the black box of networks. in the realm of speech-to-text, @xcite compare auxiliary text translation tasks, and @xcite define the localness of a word to analyze the model architecture. in this paper, we extend these ideas and construct a 10gb probing benchmark, speech-senteval, to analyze the learned linguistic properties of s2t models. prompt learning prompting refers to adding a snippet of natural language text to unlabeled data during the pre-training phase. in the case of discrete prompting, discrete information is added to the dataset as described in @xcite . for example, discrete prompt tokens such as "it", "is", "[mask]" can be used to classify a movie review. to illustrate, given the input text @xmath0, the input embedding sequence would be formulated as ## s2t behaviors analysis in a simple implementation of an end-to-end s2t model, only the top-most representation is used for decoding, ignoring the interactions among layers @xcite . to better understand the fusion of speech and text features, we averagely divide the encoder layers into three levels: bottom, middle, and top, and analyze the properties learned in each layer. specifically, we design a fine-grained linguistic probing benchmark, namely speech-senteval, for the s2t model following @xcite and leave the vanilla text probing benchmark for the t2t model to further understand linguistic properties. subsequently, we focus on different views between acoustic (speech view) and linguistic properties (text view) in different layers of the s2t model in order to explore the acoustic and linguistic behaviors, respectively. ## abstract prompt learning for st motivated by the analyses in section 3, augmenting the representation power of higher encoder layers with a plug-and-play strategy is essential to achieve better linguistic knowledge that serves for high-quality speech translation. besides efficient and plug-in properties, the method is required to enhance the higher layers without disturbing the phonetic information learned in the lower layers. accordingly, we precisely meet these requirements by introducing a soft prompt strategy @xcite that has been empirically successful in several natural language understanding tasks but remains under-explored in the field of speech translation. specifically, higher encoder layers in pretrained st models will be equipped with learnable soft-prompt representations to capture additional abstract information, e.g., linguistic knowledge. we call the proposed method abstract prompt. ## conclusion in this paper, we take the first step toward understanding the fusion of speech and text features in the s2t model by probing tasks, taking the text-totext model as a reference. specifically, we design and release a 10gb linguistic probing benchmark, speech-senteval, for the s2t task. we find that the uppermost encoder layers of the s2t model can not learn linguistic knowledge efficiently, which is vital for translation. based on these insights, we propose a straightforward plug-in prompt-learning strategy, coined as abstract prompt, on the highlevel layers to broaden the representation ability of the pre-trained st models. experimental results on four widely-used datasets show that promptst can deliver significant improvements over a strong baseline by capturing richer linguistic knowledge.