Learning Nearest Neighbour Informed Latent Word Embeddings to Improve Zero-Shot Machine Translation (2023)

Abstract: Multilingual neural translation models exploit cross-lingual transfer to perform zero-shot translation between unseen language pairs. Past efforts to improve cross-lingual transfer have focused on aligning contextual sentence-level representations. This paper introduces three novel contributions to allow exploiting nearest neighbours at the token level during training, including: (i) an efficient, gradient-friendly way to share representations between neighboring tokens; (ii) an attentional semantic layer which extracts latent features from shared embeddings; and (iii) an agreement loss to harmonize predictions across different sentence representations. Experiments on two multilingual datasets demonstrate consistent gains in zero shot translation over strong baselines.

## introduction many-to-many multilingual neural translation models @xcite @xcite share a single representation space across multiple language pairs, which enables them to perform zero-shot translations between unseen pairs @xcite @xcite . prior work on zeroshot translation has focused on aligning contextual, sentence-level representations from multiple languages @xcite , to make these more 'universal' or language-agnostic @xcite . non-contextual, token-level representations offer another space in which this kind of alignment could be pursued, but this space has not been thoroughly explored in prior work. even lexicon-based methods @xcite , which exploit tokenlevel anchors from multilingual dictionaries @xcite , still use these to align representations at the sentence level. in this work, we explore a novel technique for sharing information across languages at the token level, which exploits nearest neighbours (nns) to aggregate information from subwords across multiple languages. when analysing embedding spaces, many authors speak in terms of "neighborhoods" or "subspaces" which group together tokens from a particular semantic field or other natural class. these neighborhoods form implicitly as a model learns similarities between embedded words or subwords. we propose to make this neighborhood structure explicit by forcing a model to consider a token's neighbors when learning its embedding. specifically, we dynamically perturb a translation model's token embeddings at training time by averaging them with their nearest neighbors; thus a token like soccer may end up mixed together with related tokens such as football, fußball, or futbol from potentially distinct languages (figure 1 ). this encourages the model to organize its subword embeddings in such a way that nearby tokens convey similar information to one another. we hypothesize that this process will produce a more structured embedding space which will in turn enable more fluent outputs. this process only uses the model's embedding layer, and does not require any offline dictionaries or additional data. our experiments and ablations show that this simple technique significantly increases the effectiveness of translation models on the iwslt17 and ted59 massively multilingual datasets. concretely, our contributions include: (i) an efficient, gradient-friendly, soft representation-mixing technique which exploits token-level neighbors without changing the transformer architecture; (ii) an attentional semantic layer which extracts features from mixed representations to give neighbour-informed latent word embeddings, and which is a drop-in replacement for a conventional embedding layer; and (iii) an agreement loss which harmonizes predictions with and without neighbor-informed embeddings. ## translation with nearest neighbour augmented embeddings we describe our model for nearest-neighbour informed token level embeddings (figure 2 ) of subwords from multiple source languages. nearest neighbor retrieval let l emb be a word embedding layer that performs a lookup embp¨q using weights w emb p r |v|ˆd , where v is a joint subword vocabulary over all languages and d is a fixed embedding dimension. given the embedding q " embpwq p r 1ˆd of a subword w, we wish to find q's nearest neighbour n (or neighbors n 1 , ..., n k ) using maximum inner product search (mips) over the weight matrix w emb : approximate solutions to (1) can be efficiently computed on-the-fly using anisotropic vector quantization @xcite . 1 given the approximate nearest neighbors (anns) n 1 , ..., n k of subword w, we compute a weighted average over these tokens' embeddings 1 exact and approximate solutions yield similar results, but approximation gives significant gains in training speed. ## analysis ablation study. ## conclusion we described a novel approach to harness nearest neighbors at the token level and learn nearestneighbour informed word embeddings for every word in a source language for many-to-many multilingual translation. our experiments show that this simple yet effective approach results in consistently better zero-shot translations across multiple multilingual datasets. additionally, our model produces translations in the right target language with high accuracy. our analysis shows that our model learns to organize subwords into semantically-related neighborhoods, and reduces the separation between monolingual subspaces in the embedding space.