Word embedding is central to neural machine translation ( NMT ) , which has attracted intensive research interest in recent years . In NMT , the source embedding plays the role of the entrance while the target embedding acts as the terminal . These layers occupy most of the model parameters for representation learning . Furthermore , they indirectly interface via a soft-attention mechanism , which makes them comparatively isolated . In this paper , we propose shared-private bilingual word embeddings , which give a closer relationship between the source and target embeddings , and which also reduce the number of model parameters . For similar source and target words , their embeddings tend to share a part of the features and they cooperatively learn these common representation units . Experiments on 5 language pairs belonging to 6 different language families and written in 5 different alphabets demonstrate that the proposed model provides a significant performance boost over the strong baselines with dramatically fewer model parameters . With the introduction of ever more powerful architectures , neural machine translation ( NMT ) has become the most promising machine translation method ( Kalchbrenner and Blunsom , 2013 ; Sutskever et al . , 2014 ; Bahdanau et al . , 2015 ) . For word representation , different architecturesincluding , but not limited to , recurrence-based ( Chen et al . , 2018 ) , convolution-based ( Gehring et al . , 2017 ) and transformation-based ( Vaswani et al . , 2017 ) NMT models-have been taking advantage of the distributed word embeddings to capture the syntactic and semantic properties of words ( Turian et al . , 2010 ) . Figure 1 : Comparison between ( a ) standard word embeddings and ( b ) shared-private word embeddings . In ( a ) , the English word " Long " and the German word " Lange " , which have similar lexical meanings , are represented by two private d-dimension vectors . While in ( b ) , the two word embeddings are made up of two parts , indicating the shared ( lined nodes ) and the private ( unlined nodes ) features . This enables the two words to make use of common representation units , leading to a closer relationship between them . NMT usually utilizes three matrices to represent source embeddings , target input embeddings , and target output embeddings ( also known as pre-softmax weight ) , respectively . These embeddings occupy most of the model parameters , which constrains the improvements of NMT because the recent methods become increasingly memory-hungry ( Vaswani et al . , 2017 ; Chen et al . , 2018 ) . 1 Even though converting words into subword units ( Sennrich et al . , 2016b ) , nearly 55 % of model parameters are used for word representation in the Transformer model ( Vaswani et al . , 2017 ) . To overcome this difficulty , several methods are proposed to reduce the parameters used for word representation of NMT . Press and Wolf ( 2017 ) propose two weight tying ( WT ) methods , called decoder WT and three-way WT , to substantially reduce the parameters of the word embeddings . Decoder WT ties the target input embedding and target output embedding , which has become the new de facto standard of practical NMT ( Sen- Figure 2 : Shared-private bilingual word embeddings perform between the source and target words or sub-words ( a ) with similar lexical meaning , ( b ) with same word form , and ( c ) without any relationship . Different sharing mechanisms are adapted into different relationship categories . This strikes the right balance between capturing monolingual and bilingual characteristics . The closeness of relationship decides the portion of features to be used for sharing . Words with similar lexical meaning tend to share more features , followed by the words with the same word form , and then the unrelated words , as illustrated by the lined nodes . nrich et al . , 2017 ) . Three-way WT uses only one matrix to represent the three word embeddings , where the source and target words that have the same word form tend to share a word vector . This method can also be adapted to sub-word NMT with a shared source-target sub-word vocabulary and it performs well in language pairs with many of the same characters , such as English-German and English-French ( Vaswani et al . , 2017 ) . Unfortunately , this method is not applicable to languages that are written in different alphabets , such as Chinese-English ( Hassan et al . , 2018 ) . Another challenge facing the source and target word embeddings of NMT is the lack of interactions . This degrades the attention performance , leading to some unaligned translations that hurt the translation quality . Hence , Kuang et al . ( 2018 ) propose to bridge the source and target embeddings , which brings better attention to the related source and target words . Their method is applicable to any language pairs , providing a tight interaction between the source and target word pairs . However , their method requires additional components and model parameters . In this work , we aim to enhance the word representations and the interactions between the source and target words , while using even fewer parameters . To this end , we present a languageindependent method , which is called sharedprivate bilingual word embeddings , to share a part of the embeddings of a pair of source and target words that have some common characteristics ( i.e. similar words should have similar vectors ) . Figure 1 illustrates the difference between the standard word embeddings and shared-private word embeddings of NMT . In the proposed method , each source ( or target ) word is represented by a word embedding that consists of the shared features and the private features . The shared features can also be regarded as the prior alignments connecting the source and target words . The private features allow the words to better learn the monolingual characteristics . Meanwhile , the features shared by the source and target embeddings result in a significant reduction of the number of parameters used for word representations . The experimental results on 6 translation datasets of different scales show that our model with fewer parameters yields consistent improvements over the strong Transformer baselines . In this work , we propose a novel sharing technique to improve the learning of word embeddings for NMT . Each word embedding is composed of shared and private features . The shared features act as a prior alignment guidance for the attention model to improve the quality of attention . Meanwhile , the private features enable the words to better capture the monolingual characteristics , result in an improvement of the overall translation quality . According to the degree of relevance between a parallel word pair , the word pairs are categorized into three different groups and the number of shared features is different . Our experimental results show that the proposed method outperforms the strong Transformer baselines while using fewer model parameters .

Challenge: Word embeddings occupy a large amount of memory, and weight tying does not mitigate this issue for distant language pairs on translation tasks.
Approach: They propose a language independet method where a model shares embeddings between source and target only when words have some common characteristics.
Outcome: Experiments on machine translation datasets involving multiple language families and scripts show that the proposed model outperforms baseline models while using fewer parameters.