Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens (2023)

Abstract: We argue that translation quality alone is not a sufficient metric for measuring knowledge transfer in multilingual neural machine translation. To support this claim, we introduce Representational Transfer Potential (RTP), which measures representational similarities between languages. We show that RTP can measure both positive and negative transfer (interference), and find that RTP is strongly correlated with changes in translation quality, indicating that transfer does occur. Furthermore, we investigate data and language characteristics that are relevant for transfer, and find that multi-parallel overlap is an important yet under-explored feature. Based on this, we develop a novel training scheme, which uses an auxiliary similarity loss that encourages representations to be more invariant across languages by taking advantage of multi-parallel data. We show that our method yields increased translation quality for low- and mid-resource languages across multiple data and model setups.

## introduction multilingual neural machine translation (mnmt) @xcite can support multiple translation directions in a single model, with low-resource languages benefiting most and high-resource languages degrading in quality @xcite . however, there is a large discrepancy among low-resource languages, with some languages benefiting a lot, while others see relatively little improvement. conflicting findings have emerged in cross-lingual knowledge transfer research, leaving the underlying causes for this discrepancy unclear. for example, some studies have found that token overlap can be leveraged to increase translation performance @xcite , while others have found that token overlap is unimportant for cross-lingual transfer @xcite . in the context of transferring knowledge from a parent translation model to a child model, some research has shown that quality improvements are larger when using a closely related parent @xcite , while others found that unrelated language pairs can work even better @xcite . another finding is that an english-centric model benefits most from positive transfer for directions into english, while improvement in the other directions is modest @xcite . one of the most striking observations in the literature is that the improvements of many-to-one mnmt can be explained to a large extent by the increased amount of target data @xcite , rather than by cross-lingual knowledge transfer. understanding cross-lingual knowledge transfer in the context of mnmt is an under-explored research direction @xcite . despite some existing studies that have examined mnmt representations, none have yet connected these representations to knowledge transfer. for instance, when translating "voiture" in french and "auto" in german to "car" in english, one would expect that the cross-attention context vectors for french-english and german-english would be similar. however, @xcite show that clustering occurs on the sentence level rather than the word level. even identical sentences in various languages do not occupy the same position in the representation space @xcite , and encoder representations are dependent on the target language @xcite instead of source meaning. in this paper, we investigate the relationship between cross-lingual transfer and cross-attention similarities between languages, which we formalise as representational transfer potential (rtp). this allows us to reason about knowledge transfer in a way translation quality (bleu) is unable to capture. we investigate cross-attention because it acts as bottleneck between the encoder (mostly respon-sible for representing the source sentence) and the decoder. we find that rtp can be used to quantify positive, as well as negative transfer (also known as interference). furthermore, we show that these similarities correlate with improvements in translation quality, indicating that there is knowledge transfer, and the improved translation quality is not only due to the increased data on the target side. our approach allows us to identify the dataset and language characteristics that are relevant for transfer, such as multi-parallel overlap between languages. based on our findings, we propose a method for training a multilingual translation model using an auxiliary similarity loss that exploits multi-parallel data, thereby increasing the degree of language invariance across source representations. contrary to common perception, a significant amount of multi-parallel data exists within parallel datasets such as wmt, making it more abundant than commonly assumed @xcite . our method works by alternately feeding parallel and multi-parallel batches to a model. for multi-parallel batches, we minimize an auxiliary similarity loss that encourages context vectors, resulting from cross-attention, to be similar. our results show that this approach leads to increased performance for low-resource languages across multiple data and model setups. ## analyzing transfer in many-to-many models in this section, we aim to delve deeper into the understanding of knowledge transfer across languages in mnmt models, moving beyond the commonly used metric of translation quality as a proxy for transfer. by exploring the relationship between transfer and hidden representations in a multilingual model, we aim to gain insight into why certain languages benefit more from multilingual training (as discussed in section 3). furthermore, we aim to develop training strategies that can increase representational similarity and thus enhance knowledge transfer (as outlined in section 4). ## analyzing causes for transfer next, we investigate characteristics that are relevant for transfer. our objective is to use dataset and linguistic features to predict the representational similarities xsim (ℓ,ℓ ,y) , as defined in equation 1. ## optimising for representational invariance some features that we have shown to be predictive of transfer have been used in previous work. higher vocab overlap leads to more positive transfer @xcite @xcite . temperature sampling addresses dataset size (imbalance) @xcite . backtranslated data can be used for similar effect @xcite . grouping languages by their linguistic similarity outperforms english-centric models @xcite . in contrast, there are no such methods for multi-parallel data. parallel datasets contain a large number of hidden multi-parallel sentences that remain unused, and resurfacing these improves multilingual translation quality @xcite . however, these approaches only add multi-parallel data, but do not explicitly exploit multi-parallel properties as part of the learning objective. in contrast, we describe a method that explicitly leverages the characteristics of multi-parallel data. we introduce an auxiliary similarity loss that encourages context vectors to be more similar when generating the same target token. when sampling a parallel batch, consisting of a source sentence x and the corresponding target sentence y, we optimize the cross-entropy loss as usual. when sampling a multi-parallel batch, consisting of meaning equivalent triples {x 1 , x 2 , y} (as defined in section 2.3), such that @xmath0, we optimize a similarity loss function: ## related work analyzing mnmt investigating representations using singular value canonical correlation analysis (svcca, @xcite showed that encoder representations cluster on linguistic similarity, and encoder representations are dependent on the target language @xcite . additionally, the set of most important attention heads are similar across language pairs, which enables language clustering @xcite . furthermore, representations of different languages cluster together when they are semantically related @xcite . in particular, visualising cross-attention per decoding time-step shows that meaning equivalent sentences generally cluster together @xcite . however, the extent of these phenomena has not been quantified per language. moreover, these studies have primarily focused on representations in isolation, or its relation with linguistic similarity, with less focus on the role of representations in knowledge transfer. in contrast, we explicitly connect representations to transfer, which allows for a deeper understanding of the impact of transfer on translation quality. ## conclusion previous research has primarily measured knowledge transfer in terms of bleu scores, leaving open the question of whether improvements in translation quality are due to transfer or other factors such as target data distribution. to address this gap, we proposed a new measure of knowledge transfer, representational transfer potential (rtp), which measures the representational similarities between languages. we demonstrated that rtp is capable of measuring both positive and negative transfer (interference). a key finding is that rtp is positively correlated with improved translation quality, indicating that the observed improvements in translation quality are a result of knowledge transfer rather than other factors. additionally, we explored the role of dataset and language characteristics in predicting transfer, and found that multi-parallel overlap is highly predictive for the degree of transfer, yet under-explored in existing literature. we proposed a novel learning objective that explicitly leverages multi-parallel properties, by incorporating an auxiliary similarity loss that encourages representations to be invariant across languages. our results show that a higher degree of invariance yields substantial improvements in translation quality in low-and mid-resource languages.