Subword Mapping and Anchoring across Languages (2021)

Abstract: State-of-the-art multilingual systems rely on shared vocabularies that sufficiently cover all considered languages. To this end, a simple and frequently used approach makes use of subword vocabularies constructed jointly over several languages. We hypothesize that such vocabularies are suboptimal due to false positives (identical subwords with different meanings across languages) and false negatives (different subwords with similar meanings). To address these issues, we propose Subword Mapping and Anchoring across Languages (SMALA), a method to construct bilingual subword vocabularies. SMALA extracts subword alignments using an unsupervised state-of-the-art mapping technique and uses them to create cross-lingual anchors based on subword similarities. We demonstrate the benefits of SMALA for cross-lingual natural language inference (XNLI), where it improves zero-shot transfer to an unseen language without task-specific data, but only by sharing subword embeddings. Moreover, in neural machine translation, we show that joint subword vocabularies obtained with SMALA lead to higher BLEU scores on sentences that contain many false positives and false negatives.

## introduction nlp systems that operate in more than one language have been proven effective in tasks such as cross-lingual natural language understanding and machine translation @xcite @xcite . the performance of such systems is strongly connected to their use of an input space that can sufficiently represent all the considered languages @xcite @xcite . conceptually, an effective cross-lingual input space should exploit latent similarities between languages. state-of-the-art multilingual systems take advantage of cross-lingual similarities in their input spaces through the use of a shared vocabulary of subwords. this vocabulary is learned on the concatenation of multilingual training corpora, using heuristic subword segmentation algorithms @xcite @xcite , which handle the open vocabulary problem by identifying tokens at multiple granularity levels, based on character n-gram frequencies. therefore, the embeddings of subwords that appear in several languages act as anchors between these languages and, thus, provide implicit crosslingual information that leads to improved performance @xcite @xcite . cross-lingual transfer in joint subword models may be limited by false positives, i.e. identical subwords with different meanings in two languages, a phenomenon also known as 'oversharing' @xcite . moreover, they do not benefit from false negatives, i.e. different subwords with identical meanings. examples of false positives are: die, a definite article in german and a verb in english; also, meaning 'so' or 'therefore' in german, not 'as well' as in english; or fast, which in german means 'almost', not 'quick'. examples of false negatives are and and und, very and sehr, people and menschen -all pairs being near synonyms that could benefit from a unique embedding rather than two. a unique embedding would not constrain the models to always represent or translate them in the same way, as representations are highly contextualized. in this paper, we address the problem of false positives and negatives by employing subword similarity to create cross-lingual anchors. specifically, using cross-lingual mapping, we determine subword alignments for a set of subwords, and then share their representations. in this way, we relax the requirements for isomorphism and common scripts between languages on which previous studies rely. we demonstrate that this can improve both cross-lingual transfer of language models and machine translation (mt). our contributions are the following: we release our code online foot_0 . ## smala: subword mapping and anchoring across languages our motivation is to create cross-lingual vocabularies that are parameter-efficient and exploit the similarity of concepts between different languages. we propose a method for subword mapping and anchoring across languages (smala), which combines the powerful initialization of mapping methods with the anchoring properties of joint training, while attempting to alleviate the limitations of both methods. we first learn subwords separately for each language and then train the corresponding em-beddings. we then apply a mapping method to obtain similarity scores between the embeddings, which we use to extract alignments between subwords of the two languages. we finally tie the parameters of the aligned subwords to create anchors during training. we describe hereafter in detail the two main components of our approach. ## language model transfer with smala for the first set of experiments, we attempt to transfer a pretrained language model (lm) from one language (l 1 ) to another language (l 2 ), by leveraging the linguistic knowledge that was implicitly encoded in l 1 's embedding layer. following previous work @xcite , we create an embedding layer for l 2 and initialize it by sharing parameters using smala. in this way, we aim to reduce the computational budget of crosslingual transfer via parameter sharing without sacrificing performance, but removing the need for a common script and the pitfalls of false positives and false negatives. we transfer the model following the same steps as tran (2020). we start from a pretrained lm that we continue training on masked language modeling (mlm) using monolingual data from both the original and the target languages (l 1 and l 2 ). the bilingual model has two separate embedding layers, one for l 1 and one for l 2 , while the rest of the encoder is common to l 1 and l 2 . each languagespecific embedding layer is used both as the first and last layer (tied embeddings). during this training phase, we keep including monolingual data from l 1 to avoid degradation in performance in the original language and maximize cross-lingual transfer @xcite . we update the weights of the whole model during this phase, since updating only the embeddings would not significantly reduce computation time (due to the need to calculate all activations for backpropagation) and has actually a negative impact on performance, as we observed in our initial experiments. at this stage, the transferred model could be used for any cross-lingual natural language understanding task @xcite or for unsupervised machine translation @xcite @xcite . in a second stage, we fine-tune the model for xnli @xcite on labeled data in l 1 (english), using l 1 embeddings and freezing the embedding layer. finally, we zero-shot transfer the model to l 2 data by simply changing the languagespecific embedding layer. ## experiments with machine translation in the second set of experiments, we apply smala to mt by leveraging subword alignments to create shared bilingual vocabularies from scratch, instead of joint subword vocabularies learned on concatenated source and target corpora. ## analysis in this section we attempt to quantify the effect of false positives and false negatives on each of the tasks. ## conclusion in this work we introduced smala, a novel approach to construct shared subword vocabularies that leverages similarity instead of identical subwords to create anchors. we demonstrate that our approach outperforms current methods for joint construction of multilingual subword vocabularies in cases where there is no cross-lingual signal, apart from the anchors. when cross-lingual supervision is available, our approach performs comparably to the baseline, while showing improved performance in cases with numerous false positive and false negatives. in future work, we aim to extend our method to more than two languages. we also intend to explore the effectiveness of smala for closely related languages and compare smala to other approaches, such as those using transliteration. in addition, we aim to apply smala to settings of varying cross-lingual supervision levels, such as unsupervised mt.