Original: T OWARDS M ULTI -S ENSE C ROSS -L INGUAL A LIGNMENT OF C ONTEXTUAL E MBEDDINGS
Revised: T OWARDS M ULTI -S ENSE C ROSS -L INGUAL A LIGNMENT OF C ONTEXTUAL E MBEDDINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Cross-lingual word embeddings (CLWE) have been proven useful in many crosslingual tasks.
Revised: Cross-lingual word embeddings (CLWE) have been proven useful in many crosslingual tasks.

Original: However, most existing approaches to learn CLWE including the ones with contextual embeddings are sense agnostic.
Revised: However, most existing approaches to learn CLWE including the ones with contextual embeddings are sense agnostic.

Original: In this work, we propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only.
Revised: In this work, we propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only.

Original: We operationalize our framework by ﬁrst proposing a novel sense-aware cross entropy loss to model word senses explicitly.
Revised: We operationalize our framework by ﬁrst proposing a novel sense-aware cross entropy loss to model word senses explicitly.

Original: The monolingual ELMo and BERT models pretrained with our sense-aware cross entropy loss demonstrate signiﬁcant performance improvement for word sense disambiguation tasks.
Revised: The monolingual ELMo and BERT models pretrained with our sense-aware cross entropy loss demonstrate signiﬁcant performance improvement for word sense disambiguation tasks.

Original: We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs (English to German/Spanish/Japanese/Chinese).
Revised: We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs (English to German/Spanish/Japanese/Chinese).

Original: Compared with the best baseline results, our cross-lingual models achieve 0.52%, 2.09% and 1.29% average performance improvements on zero-shot cross-lingual NER, sentiment classiﬁcation and XNLI tasks, respectively.
Revised: Compared with the best baseline results, our cross-lingual models achieve 0.52%, 2.09% and 1.29% average performance improvements on zero-shot cross-lingual NER, sentiment classiﬁcation and XNLI tasks, respectively.

Original: We will release our code.
Revised: We will release our code.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cross-lingual word embeddings (CLWE) provide a shared representation space for knowledge transfer between languages, yielding state-of-the-art performance in many cross-lingual natural language processing (NLP) tasks.
Revised: Cross-lingual word embeddings (CLWE) provide a shared representation space for knowledge transfer between languages, yielding state-of-the-art performance in many cross-lingual natural language processing (NLP) tasks.

Original: Most of the previous works have focused on aligning static embeddings.
Revised: Most of the previous works have focused on aligning static embeddings.

Original: To utilize the richer information captured by the pre-trained language model, more recent approaches attempt to extend previous methods to align contextual representations.
Revised: To utilize the richer information captured by the pre-trained language model, more recent approaches attempt to extend previous methods to align contextual representations.

Original: Aligning the dynamic and complex contextual spaces poses signiﬁcant challenges, so most of the existing approaches only perform coarse-grained alignment.
Revised: Aligning the dynamic and complex contextual spaces poses signiﬁcant challenges, so most of the existing approaches only perform coarse-grained alignment.

Original: Schuster et al.
Revised: Schuster et al.

Original: (2019) compute the average of contextual embeddings for each word as an anchor, and then learn to align the static anchors using a bilingual dictionary.
Revised: (2019) compute the average of contextual embeddings for each word as an anchor, and then learn to align the static anchors using a bilingual dictionary.

Original: In another work, Aldarmaki & Diab (2019) use parallel sentences in their approach, where they compute sentence representations by taking the average of contextual word embeddings, and then they learn a projection matrix to align sentence representations.
Revised: In another work, Aldarmaki & Diab (2019) use parallel sentences in their approach, where they compute sentence representations by taking the average of contextual word embeddings, and then they learn a projection matrix to align sentence representations.

Original: They ﬁnd that the learned projection matrix also works well for word-level NLP tasks.
Revised: They ﬁnd that the learned projection matrix also works well for word-level NLP tasks.

Original: Besides, unsupervised multilingual language models (Devlin et al., 2018; Artetxe & Schwenk, 2019; Conneau et al., 2019;
Revised: Besides, unsupervised multilingual language models (Devlin et al., 2018; Artetxe & Schwenk, 2019; Conneau et al., 2019;

Original: Liu et al., 2020) pretrained on multilingual corpora have also demonstrated strong cross-lingual transfer performance.
Revised: Liu et al., 2020) pretrained on multilingual corpora have also demonstrated strong cross-lingual transfer performance.

Original: Cao et al.
Revised: Cao et al.

Original: (2020) and Wang et al.
Revised: (2020) and Wang et al.

Original: (2020) show that unsupervised multilingual language model can be further aligned with parallel sentences.
Revised: (2020) show that unsupervised multilingual language model can be further aligned with parallel sentences.

Original: Though contextual word embeddings are intended to provide different representations of the same word in distinct contexts, Schuster et al.
Revised: Though contextual word embeddings are intended to provide different representations of the same word in distinct contexts, Schuster et al.

Original: ﬁnd that the contextual embeddings of different senses of one word are much closer compared with that of different words.
Revised: ﬁnd that the contextual embeddings of different senses of one word are much closer compared with that of different words.

Original: This contributes to the anisomorphic embedding distribution of different languages and causes problems for cross-lingual alignment.
Revised: This contributes to the anisomorphic embedding distribution of different languages and causes problems for cross-lingual alignment.

Original: For example, it will be difﬁcult to align the English word bank and its Japanese translations 銀 行 and 岸 that correspond to its two different senses, since the contextual embeddings of different senses of bank are close to each other while those of 銀 行 and 岸 are far.
Revised: For example, it will be difﬁcult to align the English word bank and its Japanese translations 銀 行 and 岸 that correspond to its two different senses, since the contextual embeddings of different senses of bank are close to each other while those of 銀 行 and 岸 are far.

Original: Recently, Zhang et al.
Revised: Recently, Zhang et al.

Original: (2019) propose two solutions to handle multi-sense words: 1) remove multi-sense words and then align anchors in the same way as Schuster et al.
Revised: (2019) propose two solutions to handle multi-sense words: 1) remove multi-sense words and then align anchors in the same way as Schuster et al.

Original: (2019); 2) generate cluster level average anchor for contextual embeddings of multi-sense words and then learn a projection matrix in an unsupervised way with MUSE (Conneau et al., 2017).
Revised: (2019); 2) generate cluster level average anchor for contextual embeddings of multi-sense words and then learn a projection matrix in an unsupervised way with MUSE (Conneau et al., 2017).

Original: They do not make good use of the bilingual dictionaries, which are usually easy to obtain, even in low-resource scenarios.
Revised: They do not make good use of the bilingual dictionaries, which are usually easy to obtain, even in low-resource scenarios.

Original: Moreover, their projection-based approach still cannot handle the anisomorphic embedding distribution problem.
Revised: Moreover, their projection-based approach still cannot handle the anisomorphic embedding distribution problem.

Original: In this work, we propose a novel sense-aware cross entropy loss to model multiple word senses explicitly, and then leverage a sense level translation task on top of it for cross-lingual model pretraining.
Revised: In this work, we propose a novel sense-aware cross entropy loss to model multiple word senses explicitly, and then leverage a sense level translation task on top of it for cross-lingual model pretraining.

Original: The proposed sense level translation task enables our models to provide more isomorphic and better aligned cross-lingual embeddings.
Revised: The proposed sense level translation task enables our models to provide more isomorphic and better aligned cross-lingual embeddings.

Original: We only use the cross-lingual signal from bilingual dictionaries for supervision.
Revised: We only use the cross-lingual signal from bilingual dictionaries for supervision.

Original: Our pretrained models demonstrate consistent performance improvements on zero-shot cross-lingual NER, sentiment classiﬁcation and XNLI tasks.
Revised: Our pretrained models demonstrate consistent performance improvements on zero-shot cross-lingual NER, sentiment classiﬁcation and XNLI tasks.

Original: Though pretrained on less data, our model achieves the state-of-the-art result on zero-shot cross-lingual German NER task.
Revised: Though pretrained on less data, our model achieves the state-of-the-art result on zero-shot cross-lingual German NER task.

Original: To the best of our knowledge, we are the ﬁrst to perform sense-level contextual embedding alignment with only bilingual dictionaries.
Revised: To the best of our knowledge, we are the ﬁrst to perform sense-level contextual embedding alignment with only bilingual dictionaries.

Original:  B ACKGROUND : PREDICTION TASKS OF LANGUAGE MODELS
Revised:  B ACKGROUND : PREDICTION TASKS OF LANGUAGE MODELS

Original: Next token prediction and masked token prediction are two common tasks in neural language model pretraining.
Revised: Next token prediction and masked token prediction are two common tasks in neural language model pretraining.

Original: We take two well-known language models, ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018), as examples to illustrate these two tasks (architectures are shown in Appendix A).
Revised: We take two well-known language models, ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018), as examples to illustrate these two tasks (architectures are shown in Appendix A).

Original: Next token prediction ELMo uses next token prediction tasks in a bidirectional language model.
Revised: Next token prediction ELMo uses next token prediction tasks in a bidirectional language model.

Original: Given a sequence of N tokens ( t 1 , t 2 , . . .
Revised: Given a sequence of N tokens ( t 1 , t 2 , . . .

Original: , t N ) , it ﬁrst prepares a context independent representation for each token by using a convolutional neural network over the characters or by word embedding lookup ( a.k.a. input embeddings ).
Revised: , t N ) , it ﬁrst prepares a context independent representation for each token by using a convolutional neural network over the characters or by word embedding lookup ( a.k.a. input embeddings ).

Original: These representations are then fed into L layers of LSTMs to generate the contextual representations: h i,j for token t i at layer j .
Revised: These representations are then fed into L layers of LSTMs to generate the contextual representations: h i,j for token t i at layer j .

Original: The model assigns a learnable output embedding w for each token in the vocabulary, which has the same dimension as h i,L .
Revised: The model assigns a learnable output embedding w for each token in the vocabulary, which has the same dimension as h i,L .

Original: Then, the forward language model predicts the token at position k with: p ( t k | t 1 , t 2 , . . .
Revised: Then, the forward language model predicts the token at position k with: p ( t k | t 1 , t 2 , . . .

Original: , t k −
Revised: , t k −

Original: [Equation]
Revised: [Equation]

Original: where k (cid:48) is the index of token t k in the vocabulary, V is the size of the vocabulary, and ( w 1 , . . .
Revised: where k (cid:48) is the index of token t k in the vocabulary, V is the size of the vocabulary, and ( w 1 , . . .

Original: , w V ) are the output embeddings for the tokens in the vocabulary.
Revised: , w V ) are the output embeddings for the tokens in the vocabulary.

Original: The backward language model is similar to the forward one, except that tokens are predicted in the reverse order.
Revised: The backward language model is similar to the forward one, except that tokens are predicted in the reverse order.

Original: Since the forward and backward language models are very similar, we will only describe our proposed approach in the context of the forward language model in the subsequent sections.
Revised: Since the forward and backward language models are very similar, we will only describe our proposed approach in the context of the forward language model in the subsequent sections.

Original: Masked token prediction The Masked Language Model (MLM) in BERT is a typical example of masked token prediction.
Revised: Masked token prediction The Masked Language Model (MLM) in BERT is a typical example of masked token prediction.

Original: Given a sequence ( t 1 , t 2 , . . .
Revised: Given a sequence ( t 1 , t 2 , . . .

Original: , t N ) , this approach randomly masks a certain percentage (15%) of the tokens and generates a masked sequence ( m 1 , m 2 , . . .
Revised: , t N ) , this approach randomly masks a certain percentage (15%) of the tokens and generates a masked sequence ( m 1 , m 2 , . . .

Original: , m N ) , where m k =
Revised: , m N ) , where m k =

Original: [ mask ] if the token at position k is masked, otherwise m k
Revised: [ mask ] if the token at position k is masked, otherwise m k

Original: = t k .
Revised: = t k .

Original: BERT ﬁrst prepares the context independent representations ( x 1 , x 2 , . . .
Revised: BERT ﬁrst prepares the context independent representations ( x 1 , x 2 , . . .

Original: , x N ) of the masked sequence via token embeddings.
Revised: , x N ) of the masked sequence via token embeddings.

Original: It is then fed into L layers of transformer encoder (Vaswani et al., 2017) to generate “bidirectional” contextual token representations.
Revised: It is then fed into L layers of transformer encoder (Vaswani et al., 2017) to generate “bidirectional” contextual token representations.

Original: The ﬁnal layer representations are then used to predict the masked token at position k as follows:
Revised: The ﬁnal layer representations are then used to predict the masked token at position k as follows:

Original: [Equation]
Revised: [Equation]

Original: where k (cid:48) , V , h and w are similarly deﬁned as in Eq.
Revised: where k (cid:48) , V , h and w are similarly deﬁned as in Eq.

Original: Unlike ELMo, BERT ties the input and output embeddings.
Revised: Unlike ELMo, BERT ties the input and output embeddings.

Original:  P ROPOSED FRAMEWORK
Revised:  P ROPOSED FRAMEWORK

Original: We ﬁrst describe our proposed sense-aware cross entropy loss to model multiple word senses explicitly in language model pretraining.
Revised: We ﬁrst describe our proposed sense-aware cross entropy loss to model multiple word senses explicitly in language model pretraining.

Original: Then, we present our joint training approach with sense alignment objective for cross-lingual mapping of contextual word embeddings.
Revised: Then, we present our joint training approach with sense alignment objective for cross-lingual mapping of contextual word embeddings.

Original: The proposed framework can be applied to most of the recent neural language models, such as ELMo, BERT and their variants.
Revised: The proposed framework can be applied to most of the recent neural language models, such as ELMo, BERT and their variants.

Original: See
Revised: See

Original: Table 1 for a summary of the main notations used in this paper.
Revised: Table 1 for a summary of the main notations used in this paper.

Original: 3. S ENSE - AWARE CROSS ENTROPY LOSS
Revised: 3. S ENSE - AWARE CROSS ENTROPY LOSS

Original: [Table]
Revised: [Table]

Original: Limitations of original training objectives
Revised: Limitations of original training objectives

Original: The training tasks with Eq.
Revised: The training tasks with Eq.

Original: 1 and 2 maximize the normalized dot product of contextual representations ( h k − 1 ,L or h k,L ) with a weight vector w k (cid:48) .
Revised: 1 and 2 maximize the normalized dot product of contextual representations ( h k − 1 ,L or h k,L ) with a weight vector w k (cid:48) .

Original: The only difference is that h k − 1 ,L in Eq.
Revised: The only difference is that h k − 1 ,L in Eq.

Original: 1 encodes the information of previous tokens in the sequence, while h k,L in Eq.
Revised: 1 encodes the information of previous tokens in the sequence, while h k,L in Eq.

Original: 2 encodes the information of the masked sequence.
Revised: 2 encodes the information of the masked sequence.

Original: Therefore, without loss of generality, we use h k ∗ ,L to denote the contextual representation for predicting the next or masked token t k .
Revised: Therefore, without loss of generality, we use h k ∗ ,L to denote the contextual representation for predicting the next or masked token t k .

Original: Even though contextual language models like
Revised: Even though contextual language models like

Original: ELMo and BERT provide a different token representation for each distinct context, the learned representations are not guaranteed to be sense separated.
Revised: ELMo and BERT provide a different token representation for each distinct context, the learned representations are not guaranteed to be sense separated.

Original: For example, Schuster et al.
Revised: For example, Schuster et al.

Original: (2019) computed the average of ELMo embeddings for each word as an anchor , and found that the average cosine distance between contextual embeddings of multi-sense words and their corresponding anchors are much smaller than the average distance between anchors, which mean that the embeddings of different senses of one word are relatively near to each other comparing to that of different words.
Revised: (2019) computed the average of ELMo embeddings for each word as an anchor , and found that the average cosine distance between contextual embeddings of multi-sense words and their corresponding anchors are much smaller than the average distance between anchors, which mean that the embeddings of different senses of one word are relatively near to each other comparing to that of different words.

Original: We also observed the same with
Revised: We also observed the same with

Original: BERT embeddings.
Revised: BERT embeddings.

Original: This ﬁnding suggests that sense clusters of a multi-sense word’s appearances are not well separated in the embedding space, and the current contextual language models still have room for improvement by considering ﬁner-grained word sense disambiguation.
Revised: This ﬁnding suggests that sense clusters of a multi-sense word’s appearances are not well separated in the embedding space, and the current contextual language models still have room for improvement by considering ﬁner-grained word sense disambiguation.

Original: Notice that there is only one weight vector w k (cid:48) for predicting the token t k in the original training tasks.
Revised: Notice that there is only one weight vector w k (cid:48) for predicting the token t k in the original training tasks.

Original: Ideally, we should treat the appearances of a multi-sense word in different contexts as different tokens, and train the language models to predict different senses of the word.
Revised: Ideally, we should treat the appearances of a multi-sense word in different contexts as different tokens, and train the language models to predict different senses of the word.

Original: In the following, we propose a novel sense-aware cross entropy loss to explicitly model different senses of a word in different contexts.
Revised: In the following, we propose a novel sense-aware cross entropy loss to explicitly model different senses of a word in different contexts.

Original: Sense-aware cross entropy loss Given a sequence ( t 1 , t 2 , . .
Revised: Sense-aware cross entropy loss Given a sequence ( t 1 , t 2 , . .

Original: . , t N ) , our proposed framework generates contextual representations ( h k,j for token t k in layer j ∈ { 1 , . . .
Revised: . , t N ) , our proposed framework generates contextual representations ( h k,j for token t k in layer j ∈ { 1 , . . .

Original: , L } ) in the same way as the standard LMs.
Revised: , L } ) in the same way as the standard LMs.

Original: Different from existing methods, our approach maintains multiple context-dependent output embeddings (henceforth, sense vectors) for each token.
Revised: Different from existing methods, our approach maintains multiple context-dependent output embeddings (henceforth, sense vectors) for each token.

Original: Speciﬁcally, let S be the maximum number of senses per token.
Revised: Speciﬁcally, let S be the maximum number of senses per token.

Original: Each word v i in the vocabulary contains S separate sense vectors ( w i, 1 , w i, 2 , . . .
Revised: Each word v i in the vocabulary contains S separate sense vectors ( w i, 1 , w i, 2 , . . .

Original: , w i,S ) , where each w i,s corresponds to a different sense (see Appendix for some interesting visualization examples).
Revised: , w i,S ) , where each w i,s corresponds to a different sense (see Appendix for some interesting visualization examples).

Original: Following the notation in Section 2, we use k (cid:48) to denote the index of the output token t k in the vocabulary.
Revised: Following the notation in Section 2, we use k (cid:48) to denote the index of the output token t k in the vocabulary.

Original: Therefore, the sense vectors of t k can be represented by ( w k (cid:48) , 1 , w k (cid:48) , 2 , . . .
Revised: Therefore, the sense vectors of t k can be represented by ( w k (cid:48) , 1 , w k (cid:48) , 2 , . . .

Original: , w k (cid:48) ,S ) , which are randomly initialized and of the same dimension as h k ∗ ,L .
Revised: , w k (cid:48) ,S ) , which are randomly initialized and of the same dimension as h k ∗ ,L .

Original: Note that we untie the input and output embeddings in our framework.
Revised: Note that we untie the input and output embeddings in our framework.

Original: We propose a word sense selection method shown in Algorithm 1 to select the most likely sense vector when training with sense-level cross entropy loss.
Revised: We propose a word sense selection method shown in Algorithm 1 to select the most likely sense vector when training with sense-level cross entropy loss.

Original: Figure 1 shows the architecture of our proposed models.
Revised: Figure 1 shows the architecture of our proposed models.

Original: Assuming sense s (cid:48) is selected for token t k (which means sense vector w k (cid:48) ,s (cid:48) should be used), we have the following new prediction task:
Revised: Assuming sense s (cid:48) is selected for token t k (which means sense vector w k (cid:48) ,s (cid:48) should be used), we have the following new prediction task:

Original: [Equation]
Revised: [Equation]

Original: The sense-aware cross entropy loss for word sense prediction is deﬁned as follows:
Revised: The sense-aware cross entropy loss for word sense prediction is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Word sense selection algorithm
Revised: Word sense selection algorithm

Original: Word sense selection when training the language model can be handled as a non-stationary data stream clustering problem (Aggarwal et al., 2004; Khalilian &
Revised: Word sense selection when training the language model can be handled as a non-stationary data stream clustering problem (Aggarwal et al., 2004; Khalilian &

Original: Mustapha, 2010; Abdullatif et al., 2018).
Revised: Mustapha, 2010; Abdullatif et al., 2018).

Original: The most intuitive way to select the corresponding sense
Revised: The most intuitive way to select the corresponding sense

Original: [Figure]
Revised: [Figure]

Original: Figure (c) shows an example of word sense selection, where the two sense clusters of t k (assume its vocabulary index is k (cid:48) ) are shifting in space.
Revised: Figure (c) shows an example of word sense selection, where the two sense clusters of t k (assume its vocabulary index is k (cid:48) ) are shifting in space.

Original: Center vectors c k (cid:48) , 1 and c k (cid:48) , 2 are used to locate cluster centers.
Revised: Center vectors c k (cid:48) , 1 and c k (cid:48) , 2 are used to locate cluster centers.

Original: Given h k,L , the algorithm performs dimension reduction on both h k,L and center vectors, and then ﬁnds the most close cluster center c k (cid:48) , 2 , so we know the output embedding corresponding to sense 2 ( w k (cid:48) , 2 ) should be used in the loss function.
Revised: Given h k,L , the algorithm performs dimension reduction on both h k,L and center vectors, and then ﬁnds the most close cluster center c k (cid:48) , 2 , so we know the output embedding corresponding to sense 2 ( w k (cid:48) , 2 ) should be used in the loss function.

Original: c k (cid:48) , 2 also makes a small step towards h k,L .
Revised: c k (cid:48) , 2 also makes a small step towards h k,L .

Original: vector for h k ∗ ,L is to select the vector w k (cid:48) ,s with the maximum dot product value h T k ∗ ,L w k (cid:48) ,s , or cosine similarity value cossim ( h k ∗ ,L , w k (cid:48) ,s ) .
Revised: vector for h k ∗ ,L is to select the vector w k (cid:48) ,s with the maximum dot product value h T k ∗ ,L w k (cid:48) ,s , or cosine similarity value cossim ( h k ∗ ,L , w k (cid:48) ,s ) .

Original: However, our experiments show that these methods do not work well due to curse of dimensionality, suboptimal learning rate and noisy h k ∗ ,L .
Revised: However, our experiments show that these methods do not work well due to curse of dimensionality, suboptimal learning rate and noisy h k ∗ ,L .

Original: We apply an online k-means algorithm to cluster different senses of a word in Algorithm 1.
Revised: We apply an online k-means algorithm to cluster different senses of a word in Algorithm 1.

Original: For each sense vector w i,s , we maintain a cluster center c i,s which is of the same dimension as w i,s .
Revised: For each sense vector w i,s , we maintain a cluster center c i,s which is of the same dimension as w i,s .

Original: Therefore, each token v i in the vocabulary has S such cluster center vectors, denoted by C i =
Revised: Therefore, each token v i in the vocabulary has S such cluster center vectors, denoted by C i =

Original: ( c i, 1 , c i, 2 , . . .
Revised: ( c i, 1 , c i, 2 , . . .

Original: , c i,S ) .
Revised: , c i,S ) .

Original: When predicting token t k in a given sequence, we apply Algorithm 1 to select the best sense vector based on h k,L (see Figure 1).
Revised: When predicting token t k in a given sequence, we apply Algorithm 1 to select the best sense vector based on h k,L (see Figure 1).

Original: Notice that h k,L is different from h k ∗ ,L for next token prediction (Figure 1a) for which h k ∗ ,L = h k − 1 ,L .
Revised: Notice that h k,L is different from h k ∗ ,L for next token prediction (Figure 1a) for which h k ∗ ,L = h k − 1 ,L .

Original: The cluster centers C
Revised: The cluster centers C

Original: i are not neural network parameters; instead, they are randomly initialized using a normal distribution N (0 , σ 2 ) and updated through Algorithm 1.
Revised: i are not neural network parameters; instead, they are randomly initialized using a normal distribution N (0 , σ 2 ) and updated through Algorithm 1.

Original: In addition, we also maintain a projection matrix P for dimension reduction to facilitate effective sense clustering.
Revised: In addition, we also maintain a projection matrix P for dimension reduction to facilitate effective sense clustering.

Original: P ∈ R d × d (cid:48) projects h k,L and c i,s from dimension d to d (cid:48) , and is shared by all tokens in vocabulary.
Revised: P ∈ R d × d (cid:48) projects h k,L and c i,s from dimension d to d (cid:48) , and is shared by all tokens in vocabulary.

Original: Similar to C , P is also randomly initialized with normal distribution N (0 , 1) , and then updated through Algorithm 2.
Revised: Similar to C , P is also randomly initialized with normal distribution N (0 , 1) , and then updated through Algorithm 2.

Original: Both Algorithm 1 and 2 run in parallel, and are interrupted when the language model stops training.
Revised: Both Algorithm 1 and 2 run in parallel, and are interrupted when the language model stops training.

Original: Some rationales behind our algorithm design are the following:
Revised: Some rationales behind our algorithm design are the following:

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: > Q then 8:
Revised: > Q then 8:

Original: Pop the oldest element from queue H .
Revised: Pop the oldest element from queue H .

Original: 9: end if 10: if m > =
Revised: 9: end if 10: if m > =

Original: M then 11:
Revised: M then 11:

Original: P = the ﬁrst d (cid:48)
Revised: P = the ﬁrst d (cid:48)

Original: PCA components of H 12: m =13: end if 14: output: P 15: until interrupted
Revised: PCA components of H 12: m =13: end if 14: output: P 15: until interrupted

Original: • Directly computing cosine similarity between c k (cid:48) ,s and h k,L suffers from the curse of dimensionality.
Revised: • Directly computing cosine similarity between c k (cid:48) ,s and h k,L suffers from the curse of dimensionality.

Original: We maintain P for dimension reduction.
Revised: We maintain P for dimension reduction.

Original: Although many algorithms use random projection for dimension reduction, we ﬁnd using PCA components can help improve clustering accuracy.
Revised: Although many algorithms use random projection for dimension reduction, we ﬁnd using PCA components can help improve clustering accuracy.

Original: • Since the neural model parameters keep being updated during training, the sense clusters become non-stationary, i.e., their locations keep changing.
Revised: • Since the neural model parameters keep being updated during training, the sense clusters become non-stationary, i.e., their locations keep changing.

Original: Experiments shows that when using P for dimension reduction, a slightly larger projection dimension d (cid:48) will make the clustering algorithm less sensitive to cluster location change.
Revised: Experiments shows that when using P for dimension reduction, a slightly larger projection dimension d (cid:48) will make the clustering algorithm less sensitive to cluster location change.

Original: We use d (cid:48)
Revised: We use d (cid:48)

Original: = 16 for ELMo, and d (cid:48)
Revised: = 16 for ELMo, and d (cid:48)

Original: = 14 for BERT.
Revised: = 14 for BERT.

Original: We also notice that the sense clustering works well even if P is updated sporadically.
Revised: We also notice that the sense clustering works well even if P is updated sporadically.

Original: We can set a relatively large update interval in Algorithm 2 to reduce computation cost.
Revised: We can set a relatively large update interval in Algorithm 2 to reduce computation cost.

Original: • A separate sense learning rate α should be set for the clustering algorithm.
Revised: • A separate sense learning rate α should be set for the clustering algorithm.

Original: A large α makes the algorithm less robust to noise, while a small α leads to slow convergence.
Revised: A large α makes the algorithm less robust to noise, while a small α leads to slow convergence.

Original: • It is essential to use the current token’s contextual representation h k,L for sense selection even though we use h k ∗ ,L = h k − 1 ,L in the next token prediction task.
Revised: • It is essential to use the current token’s contextual representation h k,L for sense selection even though we use h k ∗ ,L = h k − 1 ,L in the next token prediction task.

Original: If we use h k − 1 ,L for sense selection, experiments show that most of the variance comes from input embedding x k − 1 .
Revised: If we use h k − 1 ,L for sense selection, experiments show that most of the variance comes from input embedding x k − 1 .

Original: This introduces too much noise for word sense clustering.
Revised: This introduces too much noise for word sense clustering.

Original: Dynamic pruning of redundant word senses To make the training more efﬁcient, we keep track of relative sense selection frequency for each token in the vocabulary.
Revised: Dynamic pruning of redundant word senses To make the training more efﬁcient, we keep track of relative sense selection frequency for each token in the vocabulary.

Original: Assume token v i has initial senses ( v i, 1 , v i, 2 , . .
Revised: Assume token v i has initial senses ( v i, 1 , v i, 2 , . .

Original: , v i,S ) , for which we compute the relative frequency ρ ( v i,s ) such that 0 ≤ ρ ( v i,s ) ≤and (cid:80) s ρ ( v i,s )
Revised: , v i,S ) , for which we compute the relative frequency ρ ( v i,s ) such that 0 ≤ ρ ( v i,s ) ≤and (cid:80) s ρ ( v i,s )

Original: = 1 .
Revised: = 1 .

Original: A lower ρ ( v i,s ) means the sense is less frequently selected compared with others.
Revised: A lower ρ ( v i,s ) means the sense is less frequently selected compared with others.

Original: We check the relative frequencies after every E training steps, and if ρ ( v i,s ) <
Revised: We check the relative frequencies after every E training steps, and if ρ ( v i,s ) <

Original: β (a threshold hyper-parameter), v i,s is removed from the list of senses of v i .
Revised: β (a threshold hyper-parameter), v i,s is removed from the list of senses of v i .

Original: Remark on model size and parameters The sense cluster centers C and the projection matrix
Revised: Remark on model size and parameters The sense cluster centers C and the projection matrix

Original: P are only used to facilitate sense selection during model pretraining, which are not neural model parameters.
Revised: P are only used to facilitate sense selection during model pretraining, which are not neural model parameters.

Original: The sense vectors w i,s will no longer be used after pretraining, which can also be discarded.
Revised: The sense vectors w i,s will no longer be used after pretraining, which can also be discarded.

Original: Therefore, our models and the original models have exactly the same number of parameters when transferred to downstream tasks.
Revised: Therefore, our models and the original models have exactly the same number of parameters when transferred to downstream tasks.

Revised: Remark on model complexity The computational complexity of our algorithm is linear with respect to the size of data, so our method is scalable to train on very large datasets.

Original: 3. J OINT TRAINING WITH SENSE LEVEL TRANSLATION
Revised: 3. J OINT TRAINING WITH SENSE LEVEL TRANSLATION

Original: Training language model with sense-aware cross entropy loss helps to learn contextual token representations that are sufﬁciently distinct for different senses ( § 4.1).
Revised: Training language model with sense-aware cross entropy loss helps to learn contextual token representations that are sufﬁciently distinct for different senses ( § 4.1).

Original: In this subsection, we extend it to cross-lingual settings and present a novel approach to learn cross-lingual contextual word embeddings at the sense level.
Revised: In this subsection, we extend it to cross-lingual settings and present a novel approach to learn cross-lingual contextual word embeddings at the sense level.

Original: Our approach uses a bilingual seed dictionary, 2 and can be applied to both next and masked token prediction tasks.
Revised: Our approach uses a bilingual seed dictionary, 2 and can be applied to both next and masked token prediction tasks.

Original: For training the cross-lingual LM, we concatenate the (non-parallel) corpora of two languages, Land L 2 , and construct a joint vocabulary O =
Revised: For training the cross-lingual LM, we concatenate the (non-parallel) corpora of two languages, Land L 2 , and construct a joint vocabulary O =

Original: O L 1 ∪ O L 2 , where O L 1 and O L 2 are the vocabularies of L 1 and L 2 , respectively.
Revised: O L 1 ∪ O L 2 , where O L 1 and O L 2 are the vocabularies of L 1 and L 2 , respectively.

Original: Algorithm 1 is used to model the senses of tokens in the joint vocabulary.
Revised: Algorithm 1 is used to model the senses of tokens in the joint vocabulary.

Original: In addition to predicting the correct monolingual sense p ( t k,s (cid:48) | context ) in Eq.
Revised: In addition to predicting the correct monolingual sense p ( t k,s (cid:48) | context ) in Eq.

Original: 3, we also train the model to predict its sense level translation.
Revised: 3, we also train the model to predict its sense level translation.

Original: Let v j be the translation of t k and sense v j,s ∗ of v j be the best sense level translation under the given context, we add the following sense-level translation prediction task to maximize probability of v j,s ∗ .
Revised: Let v j be the translation of t k and sense v j,s ∗ of v j be the best sense level translation under the given context, we add the following sense-level translation prediction task to maximize probability of v j,s ∗ .

Original: [Equation]
Revised: [Equation]

Original: where w j,s ∗ is the corresponding sense vector of v j,s ∗ .
Revised: where w j,s ∗ is the corresponding sense vector of v j,s ∗ .

Original: Similar to the previous subsection, we maintain sense cluster centers
Revised: Similar to the previous subsection, we maintain sense cluster centers

Original: C i for each token v
Revised: C i for each token v

Original: i ∈
Revised: i ∈

Original: O and the shared projection matrix P to select the best translation sense.
Revised: O and the shared projection matrix P to select the best translation sense.

Original: Assume t k has T translations in dictionary, and each translation has S senses, then there are T × S possible sense level translations for t k in the given context.
Revised: Assume t k has T translations in dictionary, and each translation has S senses, then there are T × S possible sense level translations for t k in the given context.

Original: If the cossim ( h k,L P , c j,s ∗ P ) value is the largest among the T × S sense cluster centers, then we select v j,s ∗ as the closest translation.
Revised: If the cossim ( h k,L P , c j,s ∗ P ) value is the largest among the T × S sense cluster centers, then we select v j,s ∗ as the closest translation.

Original: An example is shown in Figure 2.
Revised: An example is shown in Figure 2.

Original: If token t k has at least one translation in the dictionary, the translation cross entropy loss can be computed as:
Revised: If token t k has at least one translation in the dictionary, the translation cross entropy loss can be computed as:

Original: [Equation]
Revised: [Equation]

Original: If token t k has no translation in the seed dictionary, we use Eq.
Revised: If token t k has no translation in the seed dictionary, we use Eq.

Original: 4 as the only loss.
Revised: 4 as the only loss.

Original: The joint training loss is deﬁned as follows:
Revised: The joint training loss is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Further alignment (optional)
Revised: Further alignment (optional)

Original: Our sense-aware pretraining tries to move similar senses of two different languages close to each other as illustrated in Figure 2.
Revised: Our sense-aware pretraining tries to move similar senses of two different languages close to each other as illustrated in Figure 2.

Original: This process makes the sense distributions of the two languages more isomorphic (some sense vector visualization examples are shown in Appendix C).
Revised: This process makes the sense distributions of the two languages more isomorphic (some sense vector visualization examples are shown in Appendix C).

Original: Applying the linear projection approach proposed by Schuster et al.
Revised: Applying the linear projection approach proposed by Schuster et al.

Original: (2019) on top of the language model pretrained with our framework can further improve cross-lingual transfer on some tasks.
Revised: (2019) on top of the language model pretrained with our framework can further improve cross-lingual transfer on some tasks.

Original: See Appendix B for more details of our implementation.
Revised: See Appendix B for more details of our implementation.

Original:  E XPERIMENTS 4. E XPERIMENTS USING MONOLINGUAL MODELS
Revised:  E XPERIMENTS 4. E XPERIMENTS USING MONOLINGUAL MODELS

Original: To verify the effectiveness of our proposed sense-aware cross entropy loss, we implement the monolingual models on top of ELMo and BERT with the changes described in § 3.1, which are named
Revised: To verify the effectiveness of our proposed sense-aware cross entropy loss, we implement the monolingual models on top of ELMo and BERT with the changes described in § 3.1, which are named

Original: SaELMo (Sense-aware ELMo) and SaBERT (Sense-aware BERT) respectively.
Revised: SaELMo (Sense-aware ELMo) and SaBERT (Sense-aware BERT) respectively.

Original: The algorithm for dynamic pruning of redundant word senses is optional, which is implemented on SaELMo only.
Revised: The algorithm for dynamic pruning of redundant word senses is optional, which is implemented on SaELMo only.

Original: Pretraining settings We use the one billion word language modeling benchmark data (Chelba et al., 2013) to pretrain all the monolingual models.
Revised: Pretraining settings We use the one billion word language modeling benchmark data (Chelba et al., 2013) to pretrain all the monolingual models.

Original: The corpus is preprocessed with the provided scripts, and then converted to lowercase.
Revised: The corpus is preprocessed with the provided scripts, and then converted to lowercase.

Original: We do not apply any subword tokenization.
Revised: We do not apply any subword tokenization.

Original: We use similar hyper-parameters as Peters et al.
Revised: We use similar hyper-parameters as Peters et al.

Original: (2018) to train the ELMo and SaELMo models, and similar hyperparameters as Devlin et al.
Revised: (2018) to train the ELMo and SaELMo models, and similar hyperparameters as Devlin et al.

Original: (2018) to train 4-layer BERT-Tiny and SaBERT-Tiny.
Revised: (2018) to train 4-layer BERT-Tiny and SaBERT-Tiny.

Original: Next sentence prediction task is disabled in BERT-Tiny and SaBERT-Tiny, since this task is irrelevant to our proposed changes.
Revised: Next sentence prediction task is disabled in BERT-Tiny and SaBERT-Tiny, since this task is irrelevant to our proposed changes.

Original: See Appendix D.1 for a complete list of hyper-parameters.
Revised: See Appendix D.1 for a complete list of hyper-parameters.

Original: Word sense disambiguation (WSD)
Revised: Word sense disambiguation (WSD)

Original: Since our context-aware cross entropy loss is designed to learn word senses better in the context, we ﬁrst conduct experiments to compare our monolingual model with the original models on the WSD task (Raganato et al., 2017), which is a task to associate words in context with the most suitable entry in a pre-deﬁned sense inventory.
Revised: Since our context-aware cross entropy loss is designed to learn word senses better in the context, we ﬁrst conduct experiments to compare our monolingual model with the original models on the WSD task (Raganato et al., 2017), which is a task to associate words in context with the most suitable entry in a pre-deﬁned sense inventory.

Original: We use a similar framework as Peters et al.
Revised: We use a similar framework as Peters et al.

Original: (2018) to evaluate the monolingual models.
Revised: (2018) to evaluate the monolingual models.

Original: 3 We use SemCor 3.0 (Miller et al., 1993) as training data, and Senseval/SemEval series (Edmonds & Cotton, 2001; Moro &
Revised: 3 We use SemCor 3.0 (Miller et al., 1993) as training data, and Senseval/SemEval series (Edmonds & Cotton, 2001; Moro &

Original: Navigli, 2015; Navigli et al., 2013; Pradhan et al., 2007; Snyder & Palmer, 2004) as test data.
Revised: Navigli, 2015; Navigli et al., 2013; Pradhan et al., 2007; Snyder & Palmer, 2004) as test data.

Original: We use the pretrained models to compute the average of contextual representations for each sense in training data, and then classify the senses of the target words in test sentences by ﬁnding the nearest neighbour.
Revised: We use the pretrained models to compute the average of contextual representations for each sense in training data, and then classify the senses of the target words in test sentences by ﬁnding the nearest neighbour.

Original: WSD results are presented in Table 2.
Revised: WSD results are presented in Table 2.

Original: SaELMo shows signiﬁcant performance improvements over the baseline ELMo model in all of the ﬁve test sets.
Revised: SaELMo shows signiﬁcant performance improvements over the baseline ELMo model in all of the ﬁve test sets.

Original: SaBERT-Tiny also outperforms BERT-Tiny except on SE07, which is the smallest among the ﬁve test sets.
Revised: SaBERT-Tiny also outperforms BERT-Tiny except on SE07, which is the smallest among the ﬁve test sets.

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTS USING BILINGUAL MODELS
Revised: 4. E XPERIMENTS USING BILINGUAL MODELS

Original: To verify the effectiveness of our cross-lingual framework, we implement the bilingual models on top of ELMo, named Bi-SaELMo that does not use linear projection for further alignment and
Revised: To verify the effectiveness of our cross-lingual framework, we implement the bilingual models on top of ELMo, named Bi-SaELMo that does not use linear projection for further alignment and

Original: Bi-SaELMo+Proj that uses the linear projection.
Revised: Bi-SaELMo+Proj that uses the linear projection.

Original: Sense vectors and cluster center vectors are not shared between the forward and backward language models.
Revised: Sense vectors and cluster center vectors are not shared between the forward and backward language models.

Original: We use ELMo+Proj and JointELMo+Proj as our baseline models, where ELMo+Proj is proposed by Schuster et al.
Revised: We use ELMo+Proj and JointELMo+Proj as our baseline models, where ELMo+Proj is proposed by Schuster et al.

Original: and
Revised: and

Original: Joint-ELMo+Proj is implemented following the framework recently proposed by Wang et al.
Revised: Joint-ELMo+Proj is implemented following the framework recently proposed by Wang et al.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) combine joint training and projection, and claim their framework is applicable to any projection method, so we implement the same projection method as Schuster et al.
Revised: (2020) combine joint training and projection, and claim their framework is applicable to any projection method, so we implement the same projection method as Schuster et al.

Original: (2019) did for
Revised: (2019) did for

Original: Joint-ELMo+Proj.
Revised: Joint-ELMo+Proj.

Original: We also report results of ELMo and Joint-ELMo , which are the counterparts of
Revised: We also report results of ELMo and Joint-ELMo , which are the counterparts of

Original: ELMo+Proj and Joint-ELMo+Proj without using linear projection.
Revised: ELMo+Proj and Joint-ELMo+Proj without using linear projection.

Original: Pretraining settings To pretrain language models, we sample a 500-million-token corpus for each language from the English, German, Spanish, Japanese and Chinese Wikipedia dump.
Revised: Pretraining settings To pretrain language models, we sample a 500-million-token corpus for each language from the English, German, Spanish, Japanese and Chinese Wikipedia dump.

Original: The dictionaries used for pretraining models and learning the projection matrix were downloaded from the MUSE (Conneau et al., 2017)
Revised: The dictionaries used for pretraining models and learning the projection matrix were downloaded from the MUSE (Conneau et al., 2017)

Original: GitHub page 4 .
Revised: GitHub page 4 .

Original: We also add JMDict (Breen, 2004) to the en-jp
Revised: We also add JMDict (Breen, 2004) to the en-jp

Original: MUSE dictionary.
Revised: MUSE dictionary.

Original: Bilingual models were pretrained on en-de , en-es , en-jp and en-zh concatenated data with similar parameters as the monolingual models.
Revised: Bilingual models were pretrained on en-de , en-es , en-jp and en-zh concatenated data with similar parameters as the monolingual models.

Original: ELMo and ELMo+Proj were pretrained on monolingual data, while the projection matrix of ELMo+Proj was learned using bilingual data.
Revised: ELMo and ELMo+Proj were pretrained on monolingual data, while the projection matrix of ELMo+Proj was learned using bilingual data.

Original: See
Revised: See

Original: Appendix D.2 for a complete list of hyper-parameters.
Revised: Appendix D.2 for a complete list of hyper-parameters

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Zero-shot cross-lingual NER
Revised: Zero-shot cross-lingual NER

Original: A BiLSTM-CRF model implemented with the
Revised: A BiLSTM-CRF model implemented with the

Original: Flair framework (Akbik et al., 2018) is used for this task.
Revised: Flair framework (Akbik et al., 2018) is used for this task.

Original: For the CoNLL-2002 (Tjong
Revised: For the CoNLL-2002 (Tjong

Original: Kim Sang, 2002) and CoNLL-2003 (Sang & De Meulder, 2003) datasets, the NER model was trained on English data, and evaluated on Spanish and German test data.
Revised: Kim Sang, 2002) and CoNLL-2003 (Sang & De Meulder, 2003) datasets, the NER model was trained on English data, and evaluated on Spanish and German test data.

Original: For the OntoNotes 5.0 (Weischedel et al., 2013) dataset, the NER model was trained on all English data and evaluated on all Chinese data.
Revised: For the OntoNotes 5.0 (Weischedel et al., 2013) dataset, the NER model was trained on all English data and evaluated on all Chinese data.

Original: We report the average F1 ofruns in Table 3.
Revised: We report the average F1 ofruns in Table 3.

Original: The results show that all of the models using linear projection outperform their counterparts (not using linear projection), since minimizing token level distance is more important for cross-lingual
Revised: The results show that all of the models using linear projection outperform their counterparts (not using linear projection), since minimizing token level distance is more important for cross-lingual

Original: NER tasks.
Revised: NER tasks.

Original: Our sense-aware pretraining makes sense distributions of two languages more isomorphic, which further improves linear projection performance.
Revised: Our sense-aware pretraining makes sense distributions of two languages more isomorphic, which further improves linear projection performance.

Original: Our model Bi-SaELMo+Proj demonstrates consistent performance improvement in all the three languages.
Revised: Our model Bi-SaELMo+Proj demonstrates consistent performance improvement in all the three languages.

Original: Moreover, our model outperforms ﬁnetuned XLM/XLM-R and Multilingual BERT on German data, and achieves state of the art even though it is pretrained on less data.
Revised: Moreover, our model outperforms ﬁnetuned XLM/XLM-R and Multilingual BERT on German data, and achieves state of the art even though it is pretrained on less data.

Original: Zero-shot cross-lingual sentiment classiﬁcation We use the multi-lingual multi-domain Amazon review data (Prettenhofer & Stein, 2010) for evaluation on cross-lingual sentiment classiﬁcation.
Revised: Zero-shot cross-lingual sentiment classiﬁcation We use the multi-lingual multi-domain Amazon review data (Prettenhofer & Stein, 2010) for evaluation on cross-lingual sentiment classiﬁcation.

Original: The ratings in review data are converted into binary labels.
Revised: The ratings in review data are converted into binary labels.

Original: The average of contextual word representations is used as the document/sentence representation for each review text/summary, which is then fed into a two-dense-layer model for sentiment classiﬁcation.
Revised: The average of contextual word representations is used as the document/sentence representation for each review text/summary, which is then fed into a two-dense-layer model for sentiment classiﬁcation.

Original: All the models are trained on English, and evaluated on German and Japanese test data in the same domain.
Revised: All the models are trained on English, and evaluated on German and Japanese test data in the same domain.

Original: We report the average accuracy of 5 runs in Table 4.
Revised: We report the average accuracy of 5 runs in Table 4.

Original: Different from the NER task, the linear projection approach for cross-lingual alignment does not work for this task, since it may add noise to embedding features.
Revised: Different from the NER task, the linear projection approach for cross-lingual alignment does not work for this task, since it may add noise to embedding features.

Original: Our model
Revised: Our model

Original: Bi-SaELMo demonstrates consistent improvements in all of the 6 evaluation tasks.
Revised: Bi-SaELMo demonstrates consistent improvements in all of the 6 evaluation tasks.

Original: The performance of Bi-SaELMo is signiﬁcantly better than Joint-ELMo, which shows that our sense-level translation pretraining objective improves cross-lingual embedding alignment
Revised: The performance of Bi-SaELMo is signiﬁcantly better than Joint-ELMo, which shows that our sense-level translation pretraining objective improves cross-lingual embedding alignment.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Zero-shot cross-lingual natural language inference (XNLI)
Revised: Zero-shot cross-lingual natural language inference (XNLI)

Original: We use XNLI (Conneau et al., 2018) and MultiNLI (Williams et al., 2018) data for evaluation on this task.
Revised: We use XNLI (Conneau et al., 2018) and MultiNLI (Williams et al., 2018) data for evaluation on this task.

Original: The Bi-LSTM baseline modelwas trained on MultiNLI English training data, and then evaluated on XNLI German, Spanish,
Revised: The Bi-LSTM baseline modelwas trained on MultiNLI English training data, and then evaluated on XNLI German, Spanish,

Original: Chinese test data.
Revised: Chinese test data.

Original: We report the average zero-shot XNLI accuracy of 2 runs in Table 5.
Revised: We report the average zero-shot XNLI accuracy of 2 runs in Table 5.

Original: Our models show consistent improvements over the baselines on all of the three data sets.
Revised: Our models show consistent improvements over the baselines on all of the three data sets.

Original: For zero-shot transfer to Chinese, both of our models outperform the best baseline by more than 2 points, which again demonstrates the effectiveness of our framework on distant language pairs.
Revised: For zero-shot transfer to Chinese, both of our models outperform the best baseline by more than 2 points, which again demonstrates the effectiveness of our framework on distant language pairs.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Cross-lingual word embedding demonstrates strong performance in many cross-lingual transfer tasks.
Revised: Cross-lingual word embedding demonstrates strong performance in many cross-lingual transfer tasks.

Original: The projection-based approach has a long line of research on aligning static embeddings (Mikolov et al., 2013;
Revised: The projection-based approach has a long line of research on aligning static embeddings (Mikolov et al., 2013;

Original: Xing et al., 2015; Smith et al., 2017; Joulin et al., 2018).
Revised: Xing et al., 2015; Smith et al., 2017; Joulin et al., 2018).

Original: It assumes that the embedding spaces of different languages have an isomorphic structure, and ﬁt an orthogonal matrix to project multiple monolingual embedding spaces to a shared space.
Revised: It assumes that the embedding spaces of different languages have an isomorphic structure, and ﬁt an orthogonal matrix to project multiple monolingual embedding spaces to a shared space.

Original: Recent studies (Schuster et al., 2019; Aldarmaki & Diab, 2019) have extended this approach to contextual representation alignment.
Revised: Recent studies (Schuster et al., 2019; Aldarmaki & Diab, 2019) have extended this approach to contextual representation alignment.

Original: Besides, there are also many discussions on the limitations of the projection-based approach, arguing that the isomorphic assumption is not true in general (Nakashole & Flauger, 2018; Patra et al., 2018;
Revised: Besides, there are also many discussions on the limitations of the projection-based approach, arguing that the isomorphic assumption is not true in general (Nakashole & Flauger, 2018; Patra et al., 2018;

Original: Søgaard et al., 2018; Ormazabal et al., 2019).
Revised: Søgaard et al., 2018; Ormazabal et al., 2019).

Original: Joint training is another line of research and early methods (Gouws et al., 2015; Luong et al., 2015; Ammar et al., 2016) learn static word embeddings of multiple languages simultaneously.
Revised: Joint training is another line of research and early methods (Gouws et al., 2015; Luong et al., 2015; Ammar et al., 2016) learn static word embeddings of multiple languages simultaneously.

Original: Extending joint training to cross- or multi-lingual language model pretraining has gained more attention recently.
Revised: Extending joint training to cross- or multi-lingual language model pretraining has gained more attention recently.

Original: As discussed above, unsupervised multilingual language models (Devlin et al., 2018; Artetxe & Schwenk, 2019; Conneau & Lample, 2019; Conneau et al., 2019;
Revised: As discussed above, unsupervised multilingual language models (Devlin et al., 2018; Artetxe & Schwenk, 2019; Conneau & Lample, 2019; Conneau et al., 2019;

Original: Liu et al., 2020) also demonstrate strong cross-lingual transfer performance.
Revised: Liu et al., 2020) also demonstrate strong cross-lingual transfer performance.

Original: There has been some work on sense-aware language models/embeddings (Rothe & Sch¨utze, 2015;
Revised: There has been some work on sense-aware language models/embeddings (Rothe & Sch¨utze, 2015;

Original: Pilehvar & Collier, 2016; Hedderich et al., 2019), but most of them require WordNet (Miller, 1998) or other additional resource for supervision. 
Revised: Pilehvar & Collier, 2016; Hedderich et al., 2019), and most of them require WordNet (Miller, 1998) or other additional resource for supervision. ˇSuster et al.

Revised: (2016) utilize both monolingual and bilingual information from parallel corpora to learn multi-sense word embeddings.

Original: In recent studies, Peters et al.
Revised: Peters et al.

Original: (2019) embed WordNet knowledge into BERT with attention mechanism, Levine et al.
Revised: (2019) embed WordNet knowledge into BERT with attention mechanism. Levine et al.

Original: (2019) pretrain SenseBERT to predict both the masked words and their WordNet supersenses.
Revised: (2019) pretrain SenseBERT to predict both the masked words and their WordNet supersenses.

Original: Unlike these methods, our language models learn word senses in a fully self-supervised way.
Revised: Similar to our framework, there are also some unsupervised approaches, but most of them are used to learn static embeddings. Huang et al. (2012) learn word representations with both local and global context, and then apply a clustering algorithm to learn multi-prototype vectors. Neelakantan et al.

Revised: (2014) propose an extension to the

Revised: Skip-gram model that leverage k-means clustering algorithm learns multiple embeddings per word type.

Original:   C ONCLUSIONS
Revised: Lee & Chen (2017) leverage reinforcement learning for modularized unsupervised sense level embedding learning. Boyd-Graber et al. (2020) use Gumbel softmax for sense disambiguation when learning sense embeddings.  C ONCLUSIONS

Original: In this paper, we have introduced a novel sense-aware cross entropy loss to model word senses explicitly, then we have further proposed a sense-level alignment objective for cross-lingual model pretraining using only bilingual dictionaries.
Revised: In this paper we have introduced a novel sense-aware cross entropy loss to model word senses explicitly, then we have further proposed a sense-level alignment objective for cross-lingual model pretraining using only bilingual dictionaries.

Original: The results of the experiments show the effectiveness of our monolingual and bilingual models on WSD, zero-shot cross-lingual NER, sentiment classiﬁcation and XNLI tasks.
Revised: The results of the experiments show the effectiveness of our monolingual and bilingual models on WSD, zero-shot cross-lingual NER, sentiment classiﬁcation and XNLI tasks.

Original: In future work, we will study how to effectively extend our method to multilingual models.
Revised: In future work, we will study how to effectively extend our method to multilingual models.

Original: In addition, using the sense cluster centers to learn the linear projection matrix would be another promising direction to further improve cross-lingual alignment.
Revised: In addition, using the sense cluster centers to learn the linear projection matrix would be another promising direction to further improve cross-lingual alignment

Original: A  P REDICTION TASKS OF LANGUAGE MODELS
Revised: A PPENDIX A P REDICTION TASKS OF LANGUAGE MODELS

Original: Next token prediction and masked token prediction are two common tasks in neural language model (LM) pretraining.
Revised: Next token prediction and masked token prediction are two common tasks in neural language model (LM) pretraining.

Original: We take two well-known language models, ELMo and BERT, as examples to illustrate these two tasks, which are shown in Figure 
Revised: We take two well-known language models, ELMo and BERT, as examples to illustrate these two tasks, which are shown in Figure 3.

Original: [Figure]
Revised: [Figure]

Original: B F URTHER ALIGNMENT ( OPTIONAL )
Revised: B F URTHER ALIGNMENT ( OPTIONAL )

Original: Applying the linear projection approach proposed by Schuster et al.
Revised: Applying the linear projection approach proposed by Schuster et al.

Original: (2019) on top of our framework can further improve cross-lingual transfer on some tasks.
Revised: (2019) on top of our framework can further improve cross-lingual transfer on some tasks.

Original: After our cross-lingual model is ﬁnished training on the concatenated corpora of two languages, L 1 and L 2 , it is used to generate contextual token embeddings for the word pairs in the seed dictionary D
Revised: After our cross-lingual model is ﬁnished training on the concatenated corpora of two languages, L 1 and L 2 , it is used to generate contextual token embeddings for the word pairs in the seed dictionary D

Original: =
Revised: =

Original: { ( t L 1 i , t L 2 i ) } |D| i =1 6 .
Revised: { ( t L 1 i , t L 2 i ) } |D| i =1 6 .

Original: Then, we compute the average of all contextual embeddings for each token t L j i , denoted by a L j i .
Revised: Then, we compute the average of all contextual embeddings for each token t L j i , denoted by a L j i .

Original: Finally, a linear projection matrix W ∈ R d × d is learned to minimize cross-lingual embedding distance:[Equation]
Revised: Finally, a linear projection matrix W ∈ R d × d is learned to minimize cross-lingual embedding distance:

Original: [Equation]
Revised: [Equation]

Original: C V ISUALIZATION OF SENSE VECTORS
Revised: C V ISUALIZATION OF SENSE VECTORS

Original: We visualize 7 the sense vectors of each model in a two dimensional PCA, and show some examples in Figures 4 to 7.
Revised: We visualize 7 the sense vectors of each model in a two dimensional PCA, and show some examples in Figures 4 to 7.

Original: For our English monolingual model (SaELMo), the vectors close to two different sense vectors of the word may are shown in (a) and (b) of Figure 4, respectively.
Revised: For our English monolingual model (SaELMo), the vectors close to two different sense vectors of the word may are shown in (a) and (b) of Figure 4, respectively.

Original: We observe that senses are well clustered in these two subﬁgures, where cluster (a) corresponds to “month”, and cluster (b) corresponds to “auxiliary verb”.
Revised: We observe that senses are well clustered in these two subﬁgures, where cluster (a) corresponds to “month”, and cluster (b) corresponds to “auxiliary verb”.

Original: We do the same for the English-Japanese bilingual model (Bi-SaELMo, without projection), and show the vectors close to two different sense vectors of the English word bank in (c) and (d) of Figure 5.
Revised: We do the same for the English-Japanese bilingual model (Bi-SaELMo, without projection), and show the vectors close to two different sense vectors of the English word bank in (c) and (d) of Figure 5.

Original: We can see both English and Japanese sense vectors ( trade , 銀 行 , 証 券 , etc.)
Revised: We can see both English and Japanese sense vectors ( trade , 銀 行 , 証 券 , etc.)

Original: in (c), most of which correspond to the sense “organization”, though there are some noises.
Revised: in (c), most of which correspond to the sense “organization”, though there are some noises.

Original: Similarly, most of the sense vectors in (d) correspond to sense “river bank”.
Revised: Similarly, most of the sense vectors in (d) correspond to sense “river bank”.

Original: Another two examples are shown in Figures 6 and 7.
Revised: Another two examples are shown in Figures 6 and 7.

Original: Our framework exhibits good sense clustering and sense level cross-lingual alignment behaviour in these examples.
Revised: Our framework exhibits good sense clustering and sense level cross-lingual alignment behaviour in these examples.

Original: All sense vectors are dumped at training step 200,000, which is before pretraining complete.
Revised: All sense vectors are dumped at training step 200,000, which is before pretraining complete.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: D P RETRAINING DETAILS D. M ONOLINGUAL MODEL
Revised: D P RETRAINING DETAILS D. M ONOLINGUAL MODEL

Original: All the monolingual models were trained for one million steps.
Revised: All the monolingual models were trained for one million steps.

Original: For better sense clustering performance, the maximum number of senses ( S in word sense selection algorithm) was set to 1 for the ﬁrst 20,000 steps to quickly get a reasonable initial model, and then increased to 5 afterwards when pretraining SaELMo and SaBERT-Tiny, which is controlled by hyperparameter n context in our implementation.
Revised: For better sense clustering performance, the maximum number of senses ( S in word sense selection algorithm) was set to 1 for the ﬁrst 20,000 steps to quickly get a reasonable initial model, and then increased to 5 afterwards when pretraining SaELMo and SaBERT-Tiny, which is controlled by hyperparameter n context in our implementation.

Original: For SaELMo, we set n context to 6, so that the model initialize 6 senses for each token
Revised: For SaELMo, we set n context to 6, so that the model initialize 6 senses for each token

Original: [Figure]
Revised: [Figure]

Original: but only use the ﬁrst sense in the 20,000 steps, and then use the other 5 senses (the ﬁrst sense will be disabled) afterwards.
Revised: but only use the ﬁrst sense in the 20,000 steps, and then use the other 5 senses (the ﬁrst sense will be disabled) afterwards.

Original: We implement this for SaBERT-Tiny in a slightly different way, where n context can be set to 5 directly to achieve the same effect.
Revised: We implement this for SaBERT-Tiny in a slightly different way, where n context can be set to 5 directly to achieve the same effect.

Original: We use two NVIDIA V100 GPUs to pretrain SaELMo, which takes about 15 days to complete training.
Revised: We use two NVIDIA V100 GPUs to pretrain SaELMo, which takes about 15 days to complete training.

Original: We use one NVIDIA V100 GPU to pretrain SaBERT-Tiny, which takes about 5 days.
Revised: We use one NVIDIA V100 GPU to pretrain SaBERT-Tiny, which takes about 5 days.

Original: See Tables 6 and 7 for the hyperparameters used to pretrain SaELMo and SaBERT-Tiny respectively
Revised: See Tables 6 and 7 for the hyperparameters used to pretrain SaELMo and SaBERT-Tiny respectively

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D. B ILINGUAL MODEL
Revised: D. B ILINGUAL MODEL

Original: As metioned in the paper
Revised: As metioned in the paper

Original: [Figure]
Revised: [Figure]

Original: we use Wikipedia dump to pretrain the bilingual models.
Revised: we use Wikipedia dump to pretrain the bilingual models.

Original: The Stanford
Revised: The Stanford

Original: CoreNLP tokenizer (Manning et al., 2014) is used to tokenize English, German, Spanish and Chinese data.
Revised: CoreNLP tokenizer (Manning et al., 2014) is used to tokenize English, German, Spanish and Chinese data.

Original: And the spaCy tokenizer is used to tokenize Japanese data.
Revised: And the spaCy tokenizer is used to tokenize Japanese data.

Original: All data are converted to lowercase.
Revised: All data are converted to lowercase.

Original: We convert Chinese data to simpliﬁed font to make it consistent with evaluation task datasets.
Revised: We convert Chinese data to simpliﬁed font to make it consistent with evaluation task datasets.

Original: All the language models used in cross-lingual experiments were pretrained for 600,000 steps from scratch.
Revised: All the language models used in cross-lingual experiments were pretrained for 600,000 steps from scratch.

Original: Similar to our monolingual models, maximum number of senses ( S in word sense selection algorithm) was set to 1 for the ﬁrst 20,000 steps, and the increased to 3 afterwards when pretraining
Revised: Similar to our monolingual models, maximum number of senses ( S in word sense selection algorithm) was set to 1 for the ﬁrst 20,000 steps, and the increased to 3 afterwards when pretraining

Original: Bi-SaELMo and Bi-SaELMo+Proj.
Revised: Bi-SaELMo and Bi-SaELMo+Proj.

Original: 8 We use two NVIDIA V100 GPUs to pretrain each Bi-SaELMo model, which takes about 10 days to complete the training.
Revised: 8 We use two NVIDIA V100 GPUs to pretrain each Bi-SaELMo model, which takes about 10 days to complete the training.

Original: See Table 8 for the hyperparameters used to pretrain Bi-SaELMo/Bi-SaELMo+Proj
Revised: See Table 8 for the hyperparameters used to pretrain Bi-SaELMo/Bi-SaELMo+Proj

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 8 Theoretically, in a reasonable range, it is expected that a larger S would be more helpful to capture the ﬁne-grained senses, however, due to limited computation power,
Revised: 8 Theoretically, in a reasonable range, it is expected that a larger S would be more helpful to capture the ﬁne-grained senses. However, due to limited computation power,