Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation (2023)

Abstract: Using a shared vocabulary is common practice in Multilingual Neural Machine Translation (MNMT). In addition to its simple design, shared tokens play an important role in positive knowledge transfer, which manifests naturally when the shared tokens refer to similar meanings across languages. However, when words overlap is small, e.g., using different writing systems, transfer is inhibited. In this paper, we propose a re-parameterized method for building embeddings to alleviate this problem. More specifically, we define word-level information transfer pathways via word equivalence classes and rely on graph networks to fuse word embeddings across languages. Our experiments demonstrate the advantages of our approach: 1) the semantics of embeddings are better aligned across languages, 2) our method achieves evident BLEU improvements on high- and low-resource MNMT, and 3) only less than 1.0% additional trainable parameters are required with a limited increase in computational costs, while the inference time is identical to baselines.

## introduction multilingual systems @xcite typically use a shared vocabulary to build the word space uniformly. for instance, in mnmt scenarios, this is achieved by combining all source and target training sentences together and training a shared language-agnostic tokenizer, e.g., bpe @xcite , to split the words into tokens. such a design is simple and scales easily. moreover, shared tokens also encourage positive knowledge transfer when they refer to equivalent or similar meanings. research on cross-lingual word embeddings @xcite shows that exploiting a weak supervision signal from identical words remarkably boosts crosslingual word embedding or bilingual lexicon induction, i.e., word translation. this point is also held for higher-level representation learning, e.g., multilingual bert @xcite , where @xcite show that knowledge transfer is more pronounced between languages with higher lexical overlap. for bilingual machine translation, aji et al. (2020) investigate knowledge transfer in a parent-child transfer setting foot_1 and reveal that word embeddings play an important role, particularly if they are correctly aligned. all findings above point to the importance of word-level knowledge transfer for downstream systems, no matter whether this transfer is achieved through sharing, mapping, or alignment. to some extent, knowledge transfer in multilingual translation systems @xcite can be seen as a special case in the aforementioned bilingual translation setting, where model parameters and vocabulary are both shared and trained from scratch, and then knowledge transfer should emerge naturally. however, the shared vocabulary, as one of the core designs, has limitations: 1) when languages use different writing systems, there is little word overlap and knowledge sharing suffers. 2) even if languages use similar writing systems, shared tokens may have completely different meanings in different languages, increasing ambiguity. in this paper, we target the first issue of wordlevel knowledge sharing across languages. as illustrated in figure 1 , due to different surface forms, bicycle in english and fiets in dutch have the same meaning but are placed differently in the embedding space, unlike the shared word station. our goal is to pull together embeddings of meaningequivalent words in different languages, including languages with different writing systems, facilitating knowledge transfer for downstream systems. for simplicity, we choose english as the pivot and encourage words in other languages to move closer to their english counterparts. to this end, we define and mine subword-level knowledge propagation paths and integrate them into a graph which is represented as an adjacency matrix. then, we leverage graph networks @xcite to model information flow. the embeddings of meaning-equivalent words will transfer information along with the paths defined in the graph and finally converge into a re-parameterized embedding table, which is used by the mnmt model as usual. at a higher level, our approach establishes priors of word equivalence and then injects them into the embedding table to explicitly encourage knowledge transfer between words in the vocabulary. we choose multilingual translation as test bed to investigate the impact of word-based knowledge transfer. several experiments show the advantages of our approach: 1) our re-parameterized embedding table exhibits enhanced multilingual capabilities, resulting in consistently improved alignment of word-level semantics across languages, encouraging word-level knowledge transfer beyond identical surface forms. 2) our method consistently outperforms the baseline by a substantial margin (up to 2.3 average bleu points) for high-and lowresource mnmt, which empirically demonstrates the benefits of our re-parameterized embeddings. 3) our method scales: the extra training time and memory costs are small, while the inference time is exactly the same as benchmark systems. moreover, we demonstrate our method adapts to massive language pairs and a large vocabulary. ## related work prior work has demonstrated the importance of word-level knowledge transfer. for instance, @xcite show that by replacing some specific words with their cross-lingual equivalent counterparts, a downstream cross-lingual partof-speech tagging task can benefit from its highresource counterpart. using cross-lingual word embedding @xcite can be seen as an extension of this idea at the representation level. e.g., by fixing and applying pretrained embeddings to cross-lingual tasks. for transfer learning in bilingual nmt, amrhein and sennrich (2020) show that the child and parent models should share a substantial part of the vocabulary. for languages with different scripts, vocabulary overlap is minimal and transfer suffers. to alleviate this issue, they leverage a romanization tool, which maps characters in various scripts to latin characters, often based on pronunciation, to increase overlap. however, romanization has some limitations because 1) romanization operates at the character level, sometimes including pronunciation features, and hence its benefits are mostly limited to named entities, and 2) the process of romanization is not always reversible. @xcite extend surface-level characters normalization and apply it to multilingual translation. besides romanization signals, they further introduce phonetic and transliterated signals to augment training data and unify writing systems. their approach also shows the merits of larger vocabulary overlaps. however, the aforementioned limitations of romanization and transliteration still hold. compared to previous work, our method is more general and practical. we mine equivalent words at the semantic level and inject priors into embeddings to improve knowledge transfer. firstly, it avoids the flaws in romanization-or pronunciationbased approaches. secondly, the graph-based transfer framework naturally adapts to multilingual scenarios, where many-to-many equivalences exist. ## reparameterization framework in this section, we describe 1) how to build an equivalence graph using pair-wise data and 2) how to re-parameterize the embeddings via our graphbased approach. note that in the discussion below we use the term 'word' as referring to an actual word or a subword if a word segmentation approach such as bpe @xcite is used. ## experiments and results in this section, we apply our approach to train multilingual translation models under different configurations. ## conclusions in this paper, we target the shortcomings of solely relying on a shared vocabulary for knowledge transfer. broadly speaking, our approach is to mine word equivalence across languages and inject such priors into the embedding table using a graph network, thereby significantly improving transfer. our experiments show that our approach results in embeddings with a higher degree of multilinguality, leading to consistent improvements in mnmt. e.g., our approach achieves 2.3 average bleu gain on iwslt14, and the improvements still hold even on a much larger and more diverse dataset, namely ec30 (61m bitext, 30 language pairs). also, even for the minimal setting of mmt, i.e., bilingual translation, the performance gains are still held for 30+m dataset. at the same time, our framework remains practical: 1) through the multi-hop mechanism, the pivot language (english) bridges the way of knowledge transfer among non-english language pairs. therefore, even when only english-centric bitext datasets are available, multilingual transfer can be achieved. 2) a negligible number of additional trainable parameters are required with a limited increase in computational costs during training. meanwhile, the inference latency is exactly as same as benchmark systems by storing and deploying the re-parameterized embeddings for online systems.