Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation (2022)

Abstract: The cornerstone of multilingual neural translation is shared representations across languages. Given the theoretically infinite representation power of neural networks, semantically identical sentences are likely represented differently. While representing sentences in the continuous latent space ensures expressiveness, it introduces the risk of capturing of irrelevant features which hinders the learning of a common representation. In this work, we discretize the encoder output latent space of multilingual models by assigning encoder states to entries in a codebook,which in effect represents source sentences in a new artificial language. This discretization process not only offers a new way to interpret the otherwise black-box model representations,but, more importantly, gives potential for increasing robustness in unseen testing conditions. We validate our approach on large-scale experiments with realistic data volumes and domains. When tested in zero-shot conditions, our approach is competitive with two strong alternatives from the literature. We also use the learned artificial language to analyze model behavior, and discover that using a similar bridge language increases knowledge-sharing among the remaining languages.

## introduction a promising potential of multilingual @xcite @xcite neural machine translation (nmt) is knowledge-sharing between languages. to enable knowledge-sharing, a prerequisite is the ability to capture common features of languages, especially between related ones. constructed languages such as interlingua and esperanto are excellent examples of human-designed structures based on the commonalities of a wide range of related languages. for data-driven models, however, it is difficult to leverage such resources due to data scarcity: there is little parallel data to these constructed languages, and creating new translation heavily depends on source sentence learning a new language (english) ↓ ↓ ↓ ↓ discrete codes 3 609 57 1042 source sentence belajar bahasa baru (indonesian) ↓ ↓ ↓ discrete codes 3 57 258 our goal is to 1) improve inference-time robustness, 2) have more interpretable intermediate representations. a potential advantage of representing inputs in discrete tokens is robustness, a property especially relevant when nmt systems must cope with unexpected testing conditions. by discretization, we restrict the continuous latent space to a finite size, providing the possibility for model intermediate representations to fall back to a position seen in training. for instance, in zero-shot translation, where the model translates directions never seen in training, the inference-time behavior is often unstable @xcite @xcite . in practice, pivoting through an intermediate language typically gives a strong performance upper bound difficult to surpass by direct zero-shot translation @xcite @xcite . mapping the source sentences to discrete codes could act as a pseudo-pivoting step, which we hope to make the model more robust under zero-shot conditions. the discrete codes also provide a new way to interpret model representations. while there are a wealth of methods to analyze knowledge-sharing in multilingual nmt @xcite @xcite , they mostly either measure translation performance as a proxy, or involve sophisticated post-processing after model training, e.g. correlation scores between model hidden states @xcite , training classifiers to probe linguistic features @xcite , or pruning model submodules @xcite . in contrast, when the model hidden states are directly associated with discrete tokens, they are directly more interpretable. this characteristic is especially relevant in unseen testing conditions, where it is important to pinpoint the underlying cause of model behavior. despite the advantages, discretizing the latent space of nmt models makes them inherently less expressive than their fully continuous counterparts. maintaining translation performance relative to the continuous models is therefore a challenge. to strike a balance between expressiveness and discretization, we propose a soft discretization approach: in training, we assign each encoder hidden state to an entry in a fixed-size codebook. this step in effect clusters encoder hidden states to one of the many cluster centers in the latent space. the codebook where the cluster centers come from is then trained along with the translation model. to ensure that the decoder receives sufficient context information, we make it access both the discretized or continuous context, as illustrated in figure 1 . in our experiments on data from the large-scale multilingual translation shared task @xcite from wmt21 @xcite , our approach is able to learn meaningful discrete codes and achieve translation performance competitive with models with continuous latent spaces. our main contributions are: robustness in zero-shot conditions zero-shot generalization is a widely-discussed direction in machine learning research @xcite @xcite . in the context of nmt, early multilingual models already possess some capability of zero-shot translation of directions unseen in training @xcite . however, zero-shot performance has been shown highly sensitive to, among other factors, training data diversity @xcite , language token strategies @xcite , and dropout configurations @xcite . a main cause of the degraded quality is that the zero-shot inference generates off-target translation @xcite into a language other than the desired one. in recent shared tasks @xcite , generating synthetic data by back-translation @xcite to eliminate zero-shot conditions has been a dominant approach for improving upon pure unsupervised settings @xcite @xcite @xcite . a main motivating factor for converting zero-shot conditions to semi-supervised ones is that the latter provides more robust and consistent inference-time behavior. in this light, to fully realize the potential of knowledge-sharing in multilingual nmt, improving zero-shot robustness is an essential task. ## learning discrete codes as motivated in §1, we aim to learn to represent sources sentences with a sequence of discrete codes to learn a meaningful discretization, the learned cluster centers must fulfill some requirements: 1) avoid trivial solutions where all points are assigned to one or a few codebook entries, 2) carry sufficient context information for the decoder for the translation task, despite being less expressive than the encoder output prior to the discretization step. ## experimental setup to experiment on realistic data volumes, we use the parallel data foot_0 from the large-scale multilingual machine translation shared task @xcite from wmt 2021 @xcite . we focus on small-task-2 on southeast asian languages. to study model robustness in zero-shot conditions and the role of language relatedness, we select parallel data between the two high-resource languages: indonesian (id) and english (en) and three other languages in the austronesian family: javanese (jv), malay (ms), and filipino/tagalog (tl). this leads to two data conditions: ## main results we first discuss the translation performance of our multilingual systems ( §5.1), and then use the learned discrete codes to investigate cross-lingual knowledge-sharing of the trained models ( §5.2). ## analyses on learned discrete codes next we further investigate the discrete codes regarding its usefulness for the learned representations ( §6.1) as well as the translation task ( §6.2). ## analyses on zero-shot translation our experiments so far use single-bridge languages and are evaluated in part on zero-shot directions. we now study the impact when either of the two conditions changes: 1) when parallel data is available for previously zero-shot directions; 2) when using multiple bridge languages. zero-shot conditions could be avoided by creating synthetic data from back-translation @xcite or mining additional parallel data @xcite . both approaches introduce additional workflows into the pipeline of building translation systems. we are therefore interested in the following question: how much parallel data do we need to perform better than direct zero-shot translation? the training corpora from the shared task ( §4.1) provides an oracle condition to answer this question. as shown in to our surprise, adding 1% oracle bitext (22k sentence pairs in total) of the previously zero-shot directions already results in comparable performance to the best zero-shot performance (18.4 and 17.3 spbleu for id-bridge and en-bridge respectively). however, this comes with some degradation on supervised directions of 0.4 spbleu for id-bridge and 0.7 for en-bridge. this is likely due to the temperature-based sampling aggressively upsampling the extremely low-resource directions, meanwhile causing the model to deprioritize other higher-resource directions. when increasing oracle bitext to 10% (220k sentence pairs in total), the system outperforms direct zeroshot performance. lastly, the additional gain appear to diminish when going from 10% to all oracle data. for id-bridge, the performance appears saturated at 10%: adding the remaining 90% parallel data does not give additional gain. on the contrary, for en-bridge, the system appears to still improve, especially on y ↔z directions (+0.5 spbleu). the performance on these directions nevertheless still falls behind the id-bridge direction by 0.8 spbleu (18.9 vs 19.7 spbleu). an explanation is that the en-bridge system requires more data to train as a result of the bridge language being very distant to the rest, thereby increasing the difficulty of multitasking over all the translation directions. this echos with the previous finding that using related bridge languages eases the multilingual translation task and increases knowledge-sharing ( §5.1). ## conclusion in this work, we focus on learning to represent source sentences of multilingual nmt models by discrete codes. on multiple large-scale experiments, we show that our approach not only increase the model robustness in zero-shot conditions, but also offers more interpretable intermediate representations. we leverage the latter property to investigate the role of bridge languages, and show that using a more related bridge language leads to increased knowledge-sharing, not only between the bridge language and remaining but also between all other languages involved in training. a limitation is that the discrete codes only give a mechanism to compare hidden representations, but are not directly interpretable by humans. a potential improvement would be to use an existing codebook that corresponds to an actual human language. besides this, as next steps, we plan to improve the generation process of the discrete codes. the first direction is to make the code lookup conditionallydependent along the time dimension and learn to shrink the sequence length of the discrete codes, thereby creating a more compact representation. another direction is to explicitly incentivize more shared codes between different, and especially related, languages during training. this would bring the discrete codes closer to a language-independent representation.