Towards a Deep Understanding of Multilingual End-to-End Speech Translation (2023)

Abstract: In this paper, we employ Singular Value Canonical Correlation Analysis (SVCCA) to analyze representations learnt in a multilingual end-to-end speech translation model trained over 22 languages. SVCCA enables us to estimate representational similarity across languages and layers, enhancing our understanding of the functionality of multilingual speech translation and its potential connection to multilingual neural machine translation. The multilingual speech translation model is trained on the CoVoST 2 dataset in all possible directions, and we utilize LASER to extract parallel bitext data for SVCCA analysis. We derive three major findings from our analysis: (I) Linguistic similarity loses its efficacy in multilingual speech translation when the training data for a specific language is limited. (II) Enhanced encoder representations and well-aligned audio-text data significantly improve translation quality, surpassing the bilingual counterparts when the training data is not compromised. (III) The encoder representations of multilingual speech translation demonstrate superior performance in predicting phonetic features in linguistic typology prediction. With these findings, we propose that releasing the constraint of limited data for low-resource languages and subsequently combining them with linguistically related high-resource languages could offer a more effective approach for multilingual end-to-end speech translation.

## introduction recent years have witnessed the rapid development of end-to-end (e2e) speech-to-text translation (st) @xcite , which has demonstrated remarkable performance and outperformed conventional cascaded systems @xcite @xcite . the primary advantage of end-to-end st over cascaded st is that the new architecture avoids * corresponding author. recent years have also witnessed that multilingual neural machine translation (nmt) has attracted growing attention @xcite @xcite . one crucial characteristic of multilingual nmt is its knowledge transfer capability, where knowledge learnt from high-resource languages is leveraged to improve translation quality of low-resource languages. inspired by the success of multilingual nmt, methods used in multilingual nmt have been adapted to multilingual end-to-end speech translation. these include the combination of pre-trained models and fine-tuning @xcite and the incorporation of adapter modules into the encoder/decoder layers @xcite . however, due to the limited availability of multilingual st training data, multilingual e2e st has been relatively understudied compared to bilingual e2e st. this not only makes insights and findings into multilingual e2e st rare, but also leaves many related questions (e.g., in which way methods in multilingual nmt can be successfully adapted to multilingual e2e st) unanswered. in this paper, our key interest is an in-depth analysis into multilingual e2e st. we hope findings from the analysis could shed light on its future development. previous studies on the interpretability of multilingual pre-trained models @xcite or multilingual nmt models @xcite have tried to make the black-box of multilingual models more interpretable by understanding the distribution of language representations learnt by these models. these works have provided valuable insights into multilinguality and have advanced the development of improved multilingual models. singular value canonical correlation analysis (svcca; @xcite is a commonly used approach for investigating the representation similarity. it enables us to compare the representation similarity obtained through the same data points across different models, layers and languages. svcca has been successfully applied to understand the representation of language models and multilingual nmt models. in this paper, we conduct a comprehensive analysis into a multilingual end-to-end speech translation model trained on 22 languages, utilizing svcca as a tool. the research questions we seek to answer are as follows: answering these questions provides insights into multilingual e2e st. our findings are as follows: based on these observations, we conclude that, for low-resource languages, increasing the amount of parallel training data is more crucial than relying solely on the knowledge transfer ability of the multilingual end-to-end speech translation model. additionally, building a high-quality languagespecific sub-space is crucial for low-resource translation quality. ## main results we present the analysis results on the covost 2 dataset in however, on the other hand, all of the highresource languages gain significant improvements in translation quality, even surpassing the performance of the corresponding bilingual st models. this phenomenon is particularly evident in the @xmath1, where all the languages can be considered high-resource. the mul-tilingual system outperforms the bilingual systems by an average bleu of 3.17. similarly, in the @xmath2, the high-resource languages (french, german, catalan, and spanish) also demonstrate improved performance with an average bleu of 23.81, compared to the average bleu of 21.64 achieved by the bilingual systems. from these comparisons, we can draw the following conclusions: (i) improved audio representations enhance the information encoding capability of the encoder component, resulting in better translation quality for high/mid-resource languages. in the @xmath3, the multilingual training audio data enhances the encoder's ability by providing a more suitable encoding space, thereby boosting the performance for high-resource languages. (ii) the introduction of well-aligned audiotext data also benefits speech translation quality. in the @xmath4, although the total amount of audio data remains the same as in the bilingual setting, aligning this data with multilingual text helps the model learn better alignment between audio and text. this phenomenon has been extensively studied in the context of end-to-end bilingual speech translation, where it is referred to as the modality gap @xcite @xcite . (iii) the performance of low-resource languages is still limited by the availability of aligned audio-text data. this scarcity of data hampers the model's ability to capture the nuances and specific characteristics of low-resource languages, leading to lower translation quality compared to high/mid-resource languages. for the @xmath5, it still surpasses bilingual systems for all high-resource languages. however, it falls behind the @xmath6. we conjecture that this performance gap is due to the limited model capacity, where related parameters are affected by interference from mid-and low-resource languages in the @xmath7. interestingly, the performance on the high-resource languages in the @xmath8. this behavior is reminiscent of the behavior observed in multilingual nmt models, where the performance of high-resource languages is normally sacrificed to benefit low-resource languages. in this case, the high-resource languages correspond to the languages in the @xmath9, while the low-resource languages are the languages in the @xmath10, which have sufficient training data. however, the languages in the @xmath11. these results provide support for the observation that the multilingual st model exhibits a different pattern compared to the multilingual nmt model. the multilingual st model leverages better audio representations and alignments between audios and texts to achieve improved translation quality for languages that have sufficient parallel training data. however, the linguistic aspect loses its effectiveness in the multilingual st model, as low-resource languages are constrained by the limited amount of training data and do not benefit from their linguistically related high-resource languages. this finding aligns with what we observe in section 5.1. overall, the amount of parallel training data is more crucial than the linguistic relatedness of languages for the performance of the multilingual st model. ## language similarity to thoroughly examine the impact of language similarity on the multilingual e2e st model trained in the @xmath12, we conducted svcca analysis on our laser-mined evaluation datasets with semantically similar sentences, as mentioned in section 3.1. it's worth noting that the lasermined evaluation datasets are created specifically for the @xmath13, as the covost 2 dataset already provides a multi-way-parallel test set for the @xmath14, where the english audio remains the same across all languages. the svcca scores are computed based on layer-wise hidden states of the encoder in the @xmath15. this analysis allows us to examine the similarity between languages across different layers of the model. in the upcoming sections, our discussion will center around language similarity from multiple perspectives. we consider language family as an important factor for analysis, e.g., indo-european, which encompasses a significant portion of the languages in our dataset. we also examine language branch, like romance and germanic. additionally, the amount of available training data for each language is taken into account during our analysis. finally, we delve into patterns related to the writing systems of languages on the decoder side for translation generation. by exploring these aspects, we aim to gain insights into how linguistic factors, training data availability, and written script characteristics influence the performance of multilingual e2e st. ## linguistic typology we conduct experiments on the linguistic typology prediction for our multilingual end-to-end speech translation model trained on the @xmath16. dataset we employ typological features from uriel typological database foot_4 @xcite for experiments. uriel is a typological compendium which accommodates diverse linguistic resources from several typological databases such as wals @xcite , phoible @xcite , ethnology @xcite and glottolog @xcite . we used lang2vec library to query uriel database which provides uniform interface to access various linguistic features. we mainly use syntax, phonology and phonetic inventory typological features in our work. prediction methods we adopt two methods commonly used in previous studies: k-nearest neighbors approach (k-nn) and logistic regression @xcite . we utilize the averaged sentence representations ## discussion our study reveals several key observations that might be of interest to researchers and practitioners working in multilingual e2e st. our analysis identifies two factors that affect multilingual speech translation quality: linguistic similarity and the amount of training data. the availability of an adequate amount of training data is crucial for the model to learn language-specific sub-spaces, particularly for low-resource languages. linguistic similarity can facilitate knowledge transfer across languages, especially when low-resource languages are not constrained by limited data. thus, addressing data scarcity in the multilingual setting and improving the representation space for individual languages should be explored to enhance multilingual translation quality. in another perspective, learning a high-quality representation space for a low-resource language with limited data is also a path to achieve linguistic fairness of multilingual e2e st. it is widely recognized that enhancing audio representations can improve the performance of the acoustic model, and this principle applies to multilingual/bilingual e2e st models as well. ## conclusion in this study, we have analyzed language representational similarity learnt by a multilingual endto-end speech translation model trained on 22 languages via svcca. through our analysis, we have findings that shed light on the performance of such models. we observe that the amount of available data plays a significant role in limiting the effectiveness of knowledge transfer across languages in multilingual speech translation. using svcca to evaluate the similarity across languages, we observe a clustering effect in terms of language branch, indicating aggregation of linguistic features within language families. we also use learnt language representations to probe linguistic typology and find that the multilingual st model performs better on phonetic-related features compared to syntax features.