Original: L EARNING C ROSS -L INGUAL S ENTENCE R EPRESENTATIONS VIA A M ULTI - TASK D UAL -E NCODER M ODEL
Revised: L EARNING C ROSS -L INGUAL S ENTENCE R EPRESENTATIONS VIA A M ULTI - TASK D UAL -E NCODER M ODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural language models have been shown to achieve an impressive level of performance on a number of language processing tasks. The majority of these models, however, are limited to producing predictions for only English texts due to limited amounts of labeled data available in other languages.
Revised: A signiﬁcant roadblock in multilingual neural language modeling is the lack of labeled non-English data.

Original: One potential method for overcoming this issue is learning cross-lingual text representations that can be used to transfer the performance from training on English tasks to non-English tasks, despite little to no task-speciﬁc non-English data.
Revised: One potential method for overcoming this issue is learning cross-lingual text representations that can be used to transfer the performance from training on English tasks to non-English tasks, despite little to no task-speciﬁc non-English data.

Original: In this paper, we explore a natural setup for learning cross-lingual sentence representations: the dual-encoder.
Revised: In this paper, we explore a natural setup for learning crosslingual sentence representations: the dual-encoder.

Original: We provide a comprehensive evaluation of our cross-lingual representations on a number of monolingual, cross-lingual, and zero-shot/few-shot learning tasks, and also give an analysis of different learned cross-lingual embedding spaces.
Revised: We provide a comprehensive evaluation of our cross-lingual representations on a number of monolingual, crosslingual, and zero-shot/few-shot learning tasks and also give an analysis of different learned cross-lingual embedding spaces

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There has been a signiﬁcant amount of recent work on developing models that can produce sentence representations that are useful for a number of language processing tasks (Kiros et al., 2015; Conneau et al., 2017; Subramanian et al., 2018; Logeswaran & Lee, 2018; Cer et al., 2018).
Revised: There has been a signiﬁcant amount of recent work on developing models that can produce sentence representations that are useful for a number of language processing tasks (Kiros et al., 2015; Conneau et al., 2017; Subramanian et al., 2018; Logeswaran & Lee, 2018; Cer et al., 2018).

Original: However, these models are trained on largely monolingual data, and can thus only be used for tasks in a single language.
Revised: However, these models are trained on largely monolingual data, and can thus only be used for tasks in a single language.

Original: A promising direction for extending the previous models to multiple languages is learning cross-lingual embedding spaces (Schwenk et al., 2017; Eriguchi et al., 2018; Singla et al., 2018), which could be used to transfer performance in one language to others.
Revised: A promising direction for extending the previous models to multiple languages is learning cross-lingual embedding spaces (Schwenk et al., 2017; Eriguchi et al., 2018; Singla et al., 2018), which could be used to transfer performance in one language to others.

Original: We develop a novel approach for cross-lingual representation learning by combining the dualencoder architectures used for learning sentence representations (Logeswaran & Lee, 2018; Cer et al., 2018) and for bi-text retrieval (Guo et al., 2018).
Revised: We develop a novel approach for cross-lingual representation learning by combining the dualencoder architectures used for learning sentence representations (Logeswaran & Lee, 2018; Cer et al., 2018) and for bi-text retrieval (Guo et al., 2018).

Original: By doing so, we learn representations that maintain state-of-the-art performance in tasks for a source language while simultaneously obtaining state-of-the-art performance in zero-shot learning tasks for a target language.
Revised: By doing so, we learn representations that maintain state-of-the-art performance in tasks for a source language while simultaneously obtaining state-of-the-art performance in zero-shot learning tasks for a target language.

Original: For a given sourcetarget language pair, we construct a multi-task training scheme using native source language tasks, native target language tasks, and a bridging source-target translation  task to learn sentence representations that are aligned between the source and target languages.
Revised: For a given sourcetarget language pair, we construct a multi-task training scheme using native source language tasks, native target language tasks, and a bridging source-target translation retrieval task to learn sentence representations that are aligned between the source and target languages.

Original: We then evaluate the learned representations on several monolingual and cross-lingual tasks, and also provide a graph-based analysis of the learned representations.
Revised: We then evaluate the learned representations on several monolingual and cross-lingual tasks, and also provide a graphbased analysis of the learned representations.

Original: We ﬁnd that multi-task training using additional monolingual tasks improves performance over models that only make use of parallel data on both cross-lingual semantic textual similarity (STS) (Cer et al., 2017) and Søgaard et al.
Revised: We ﬁnd that multi-task training using additional monolingual tasks improves performance over models that only make use of parallel data on both cross-lingual semantic textual similarity (STS) (Cer et al., 2017) and Søgaard et al.

Original: (2018)’s cross-lingual eigen-similarity metric.
Revised: (2018)’s cross-lingual eigen-similarity metric.

Original: The results show that the addition of monolingual dataactually improves the embedding alignment of sentences and their translations.
Revised: The results show that the addition of monolingual data improves the embedding alignment of sentences and their translations.

Original: Furthermore, we ﬁnd that cross-lingual training with additional monolingual data leads to far better transfer learning performance, and we show that our cross-lingual representations outperform state-of-the-art zero-shot learning systems in sentiment classiﬁcation and natural language inference.
Revised: Furthermore, we ﬁnd that cross-lingual training with additional monolingual data leads to far better transfer learning performance, and we show that our cross-lingual representations outperform state-of-the-art zero-shot learning models in sentiment classiﬁcation and natural language inference.

Original:  M ULTI -T ASK DUAL - ENCODER M ODEL
Revised:  M ULTI -T ASK DUAL - ENCODER M ODEL

Original: The core of our approach is the idea of modeling various tasks as ranking input-response pairs by encoding them via two encoders, with the crucial task for learning cross-lingual representations 
Revised: The core of our approach is the idea of modeling various tasks as ranking input-response pairs by encoding them via two encoders, with the crucial task for learning cross-lingual representations being translation ranking .

Revised: For translation ranking, as well as for our other tasks, we take an input sentence s Ii and an associated response sentence s Ri , and we seek to rank s Ri over all other possible response sentences s R j ∈ S R .

Revised: To do so, we model the conditional probability P ( s R i | s I i ) as:

Original: Encoder s Ii s Ri g(s Ii ) g(s Ri ) g R
Revised: Encoder s Ii s Ri g(s Ii ) g(s Ri ) g R

Original: (s Ri ) (g(s Ii ))
Revised: (s Ri ) (g(s Ii ))

Original: T g R (s Ri )
Revised: T g R (s Ri )

Original: Conversational Response
Revised: Conversational Response

Original: Encoder
Revised: Encoder

Original: Fully-connected layers
Revised: Fully-connected layers

Original: Encoder s Pi s Ii g P (s Pi )
Revised: Encoder s Pi s Ii g P (s Pi )

Original: g(s Ii )
Revised: g(s Ii )

Original: g
Revised: g

Original: I (s Ii ) (g P (s Ii ))
Revised: I (s Ii ) (g P (s Ii ))

Original: T g I
Revised: T g I

Original: (s Pi )
Revised: (s Pi )

Original: Quick-Thought
Revised: Quick-Thought

Original: Encoder
Revised: Encoder

Original: Fully-connected layers g(s Pi )
Revised: Fully-connected layers g(s Pi )

Original: Fully-connected layers s Ri g(s Ri )
Revised: Fully-connected layers s Ri g(s Ri )

Original: g R
Revised: g R

Original: (s Ri )
Revised: (s Ri )

Original: Encoder
Revised: Encoder

Original: Fully-connected layers (g I (s Ii ))
Revised: Fully-connected layers (g I (s Ii ))

Original: T g R (s Ri )
Revised: T g R (s Ri )

Original: Encoder s i t i g(s i ) g(t i ) (g(s i ))
Revised: Encoder s i t i g(s i ) g(t i ) (g(s i ))

Original: T g(t i )
Revised: T g(t i )

Original: Translation
Revised: Translation

Original: Encoder Encoder s Ii s iR g(s Ii ) =
Revised: Encoder Encoder s Ii s iR g(s Ii ) =

Original: u(u 1 , u 2 , |u 1 -u 2 |, u 1 *u 2 )
Revised: u(u 1 , u 2 , |u 1 -u 2 |, u 1 *u 2 )

Original: NLI
Revised: NLI

Original: Encoder
Revised: Encoder

Original: Fully-connected layers
Revised: Fully-connected layers

Original: Softmax g(s Ri )
Revised: Softmax g(s Ri )

Original: =
Revised: =

Original: u
Revised: u

Original: Native Tasks
Revised: Native Tasks

Original: Bridging Taskbeing translation ranking . For translation ranking, as well as for our other tasks, we take an input sentence s Ii and an associated response sentence s Ri , and we seek to rank s Ri over all other possible response sentences s Rj ∈ S R . To do so, we model the conditional probability P ( s Ri | s Ii ) as:
Revised: Bridging Task

Original: [Equation]
Revised: [Equation]

Original: Where g I and g R are the input and response sentence encoding functions that compose the dualencoder.
Revised: Where g I and g R are the input and response sentence encoding functions that compose the dualencoder.

Original: Since the normalization term in equation 1 is computationally intractable, we follow the approaches of Henderson et al.
Revised: Since the normalization term in equation 1 is computationally intractable, we follow the approaches of Henderson et al.

Original: and instead choose to model an approximate conditional probability (cid:101) P ( s Ri | s Ii ) :
Revised: and instead choose to model an approximate conditional probability (cid:101) P ( s Ri | s Ii ) :

Original: [Equation]
Revised: [Equation]

Original: Where K denotes the size of a single batch of training examples, and the s Rj correspond to the response sentences associated with the other input sentences in the same batch as s Ii .
Revised: Where K denotes the size of a single batch of training examples, and the s Rj correspond to the response sentences associated with the other input sentences in the same batch as s Ii .

Original: We parametrize g I and g R as deep neural networks that are trained to minimize the negative log-likelihood of (cid:101) P ( s Ri | s Ii ) for each task.
Revised: We parametrize g I and g R as deep neural networks that are trained to minimize the negative log-likelihood of (cid:101) P ( s Ri | s Ii ) for each task.

Original: In order to produce a single sentence encoding function g that can be evaluated on downstream tasks, we share several layers between the input and response encoders and treat the ﬁnal output of these shared layers as g .
Revised: In order to produce a single sentence encoding function g that can be evaluated on downstream tasks, we share several layers between the input and response encoders and treat the ﬁnal output of these shared layers as g .

Original: Additionally, these layers are modeled after the Universal Sentence Encoder (USE) model of Cer et al.
Revised: Additionally, these layers are modeled after the Universal Sentence Encoder (USE) model of Cer et al.

Original: (2018), since it is the state-of-the-art model that is most amenable to our setup.
Revised: (2018), since it is the state-of-the-art model that is most amenable to our setup.

Original: To learn cross-lingual representations, we train g on several tasks from mirrored corpora across languages 1 for the source-target language pairs English-French
Revised: To learn cross-lingual representations, we train g on several tasks from mirrored corpora across languages 1 for the source-target language pairs English-French

Original: (en-fr), English-Spanish (enes), and English-German (en-de).
Revised: (en-fr), English-Spanish (enes), and English-German (en-de).

Original: The resulting model structure is illustrated in Figure 1.
Revised: The resulting model structure is illustrated in Figure 1.

Original: 2. E NCODER A RCHITECTURE
Revised: 2. E NCODER A RCHITECTURE

Original: Word and Character Embeddings.
Revised: Word and Character Embeddings.

Original: As part of the training process for learning the cross-lingual sentence encoding function g , we learn embeddings for the words and characters present in the training data for a given source-target language pair.
Revised: As part of the training process for learning the cross-lingual sentence encoding function g , we learn embeddings for the words and characters present in the training data for a given source-target language pair.

Original: Word embeddings are learned end-to-end, as we noticed that pre-trained embeddings did not make a difference for ﬁnal performance.
Revised: Word embeddings are learned end-to-end, as we noticed that pre-trained embeddings did not make a difference for ﬁnal performance.

Original: Character embeddings are learned in a similar manner, but with the added stipulation that we consider character n-gram embeddings instead of single character embeddings by using a single feedforward layer with tanh activation on top of character n-grams.
Revised: Character embeddings are learned in a similar manner, but with the added stipulation that we consider character n-gram embeddings instead of single character embeddings by using a single feedforward layer with tanh activation on top of character n-grams.

Original: Each word in an input sentence then obtains a characterembedding representation by having its character n-gram representations summed together.
Revised: Each word in an input sentence then obtains a character representation by having its character n-gram representations summed together.

Original: To have the sentence encoder g leverage the word and character embeddings together without drastically increasing its number of parameters, we sum the word and character embeddings before using them as input to g .
Revised: To have the sentence encoder g leverage the word and character representations together without drastically increasing its number of parameters, we sum the word and character embeddings before using them as input to g .

Original: Transformer Encoder.
Revised: Transformer Encoder.

Original: The actual architecture of the shared encoder g consists of three 2 layers of transformer stacks, which contain the feed-forward and multi-head attention sub-layers described in
Revised: The actual architecture of the shared encoder g consists of three 2 layers of transformer stacks, which contain the feed-forward and multi-head attention sub-layers described in

Original: Vaswani et al.
Revised: Vaswani et al.

Original: The transformer encoder output is a variable-length sequence at each stack.
Revised: The transformer encoder output is a variable-length sequence at each stack.

Original: We average encodings of all sequence positions in the ﬁnal layer as the ﬁnal sentence encoding.
Revised: We average encodings of all sequence positions in the ﬁnal layer as the ﬁnal sentence encoding.

Original: This embedding is then fed into different sets of feedforward layers that are used for each task.
Revised: This embedding is then fed into different sets of feedforward layers that are used for each task.

Original: For our transformer layers, we use 8 attentions heads, a hidden size of 512, and a ﬁlter size of 2048 in all experiments.
Revised: For our transformer layers, we use 8 attentions heads, a hidden size of 512, and a ﬁlter size of 2048.

Original: 2. M ULTI - TASK T RAINING S ETUP
Revised: 2. M ULTI - TASK T RAINING S ETUP

Original: To learn a function g that is capable of strong cross-lingual matching and transfer learning performance for a source-target language pair while also maintaining monolingual downstream task performance, we employ four unique task types for each language pair.
Revised: To learn a function g that is capable of strong cross-lingual matching and transfer learning performance for a source-target language pair while also maintaining monolingual downstream task performance, we employ four unique task types for each language pair.

Original: Speciﬁcally, we employ a conversation response prediction task, a quick thought task, a natural language inference task , and a bridging task – translation ranking .
Revised: Speciﬁcally, we employ a conversation response prediction task, a quick thought task, a natural language inference task , and a bridging task – translation ranking .

Original: Six total tasks are used in training, as the ﬁrst two tasks are mirrored across languages.
Revised: Six total tasks are used in training, as the ﬁrst two tasks are mirrored across languages.

Original: Conversation Response Prediction.
Revised: Conversation Response Prediction.

Original: We model the conversation response prediction task in the same manner as Yang et al.
Revised: We model the conversation response prediction task in the same manner as Yang et al.

Original: We minimize the negative log-likelihood of (cid:101) P ( s Ri | s Ii ) , where s Ii is a single comment and s Ri is its associated response comment.
Revised: We minimize the negative log-likelihood of (cid:101) P ( s R i | s I i ) , where s Ii is a single comment and s Ri is its associated response comment.

Original: For the response side, we model g R ( s Ri ) as two fully-connected feedforward layers of size 320 and 512 with tanh activation on top of g ( s Ri ) .
Revised: For the response side, we model g R ( s Ri ) as two fully-connected feedforward layers of size 320 and 512 with tanh activation on top of g ( s Ri ) .

Original: For the input side, however, we simply let g I ( s Ii ) =
Revised: For the input side, however, we simply let g I ( s Ii ) =

Original: g ( s Ii ) , as we noticed in early experiments that letting the optimization of the conversational response task more directly inﬂuence the parameters of the underlying sentence encoder g led to better downstream task performance.
Revised: g ( s Ii ) , as we noticed in early experiments that letting the optimization of the conversational response task more directly inﬂuence the parameters of the underlying sentence encoder g led to better downstream task performance.

Original: Quick Thought.
Revised: Quick Thought.

Original: We use a modiﬁed version of the Quick Thought task detailed by Logeswaran &
Revised: We use a modiﬁed version of the Quick Thought task detailed by Logeswaran &

Original: Lee (2018).
Revised: Lee (2018).

Original: We minimize the sum of the negative log-likelihoods of (cid:101) P ( s Ri | s Ii ) and (cid:101) P ( s Pi | s Ii ) , where s Ii is a sentence taken from an article and s Pi and s Ri are its predecessor and successor sentences respectively.
Revised: We minimize the sum of the negative log-likelihoods of (cid:101) P ( s Ri | s Ii ) and (cid:101) P ( s Pi | s Ii ) , where s Ii is a sentence taken from an article and s Pi and s Ri are its predecessor and successor sentences respectively.

Original: For this task, we model all three of g P ( s Pi ) , g I ( s Ii ) , and g R ( s Ri ) using separate, fully-connected feedforward layers of size 320 and 512 with tanh activation on top of g , as we did for g R ( s Ri ) in our conversational modeling task.
Revised: For this task, we model all three of g P ( s Pi ) , g I ( s Ii ) , and g R ( s Ri ) using separate, fully-connected feedforward layers of size 320 and 512 with tanh activation on top of g , as we did for g R ( s Ri ) in our conversational modeling task.

Original: Natural Language Inference (NLI).
Revised: Natural Language Inference (NLI).

Original: We also include an English-only natural language inference task based on Bowman et al.
Revised: We also include an English-only natural language inference task based on Bowman et al.

Original: For this task, we ﬁrst encode an input sentence s I i and its corresponding response hypothesis s Ri into vectors u 1 and u 2 using g .
Revised: For this task, we ﬁrst encode an input sentence s Ii and its corresponding response hypothesis s Ri into vectors u 1 and u 2 using g .

Original: The vectors u 1 , u 2 are then used to construct a feature vector ( u 1 , u 2 , | u 1 − u 2 | , u 1 ∗ u 2 ) , where ( · ) represents concatenation and ∗ represents element-wise multiplication.
Revised: The vectors u 1 , u 2 are then used to construct a feature vector ( u 1 , u 2 , | u 1 − u 2 | , u 1 ∗ u 2 ) , where ( · ) represents concatenation and ∗ represents element-wise multiplication.

Original: The form of this feature vector is derived from the original experiments of Bowman et al.
Revised: The form of this feature vector is derived from the original experiments of Bowman et al.

Original: This feature vector is then fed into a single feedforward layer of size 512 that is used to perform the 3-way NLI classiﬁcation.
Revised: This feature vector is then fed into a single feedforward layer of size 512 that is used to perform the 3-way NLI classiﬁcation.

Original: Translation Ranking.
Revised: Translation Ranking.

Original: Our translation task setup is identical to the one used by Guo et al.
Revised: Our translation task setup is identical to the one used by Guo et al.

Original: (2018) for bi-text retrieval.
Revised: (2018) for bi-text retrieval.

Original: We minimize the negative log-likelihood of (cid:101) P ( s i | t i ) , where ( s i , t i ) is a sourcetarget translation pair.
Revised: We minimize the negative log-likelihood of (cid:101) P ( s i | t i ) , where ( s i , t i ) is a sourcetarget translation pair.

Original: Since the translation task is intended to align the sentence representations for the source and target languages, we do not use any kind of task-speciﬁc feedforward layers and instead use g as both g I and g R .
Revised: Since the translation task is intended to align the sentence representations for the source and target languages, we do not use any kind of task-speciﬁc feedforward layers and instead use g as both g I and g R .

Original: Following Guo et al.
Revised: Following Guo et al.

Original: , we append 5 translations that are similar to the correct translation to each training example as “hard-negatives”.
Revised: , we append 5 translations that are similar to the correct translation to each training example as “hard-negatives”.

Original: Similarity is determined via a version of our model trained only on the translation ranking task.
Revised: Similarity is determined via a version of our model trained only on the translation ranking task.

Original: We did not see additional gains from using more than 5 hard-negatives.
Revised: We did not see additional gains from using more than 5 hard-negatives.

Original:  E XPERIMENTS 3. C ORPORA
Revised:  E XPERIMENTS 3. C ORPORA

Original: We draw upon multiple, openly available data sources and training corpora for the training of the tasks mentioned above.
Revised: We draw upon multiple, openly available data sources and training corpora for the training of the tasks mentioned above.

Original: For each of our datasets, we use 90% of the data for training, and the remaining 10% for development/validation.
Revised: For each of our datasets, we use 90% of the data for training, and the remaining 10% for development/validation.

Original: Our data preprocessing procedures are described in detail in the supplementary material.
Revised: Our data preprocessing procedures are described in detail in the supplementary material.

Original: Reddit.
Revised: Reddit.

Original: We preprocess the Reddit data extracted by Al-Rfou et al.
Revised: We preprocess the Reddit data extracted by Al-Rfou et al.

Original: into 600 million inputresponse comment pairs for training our conversation response prediction task.
Revised: into 600 million inputresponse comment pairs for training our conversation response prediction task.

Original: We also translate this data using the Google neural machine translation (NMT) system of Wu et al.
Revised: We also translate this data using the Google neural machine translation (NMT) system of Wu et al.

Original: Wikipedia.
Revised: Wikipedia.

Original: To get native, non-English data, we extract triplets of contiguous sentences from English, French, Spanish, and German articles take from Wikipedia.
Revised: To get native, non-English data, we extract triplets of contiguous sentences from English, French, Spanish, and German articles take from Wikipedia.

Original: Our ﬁnal extracted corpus of
Revised: Our ﬁnal extracted corpus of

Original: Wikipedia sentence triplets consists of 127.9, 49.5, 29.8, and 49.3 million triplets for English,
Revised: Wikipedia sentence triplets consists of 127.9, 49.5, 29.8, and 49.3 million triplets for English,

Original: French, Spanish, and German respectively, which we use to train our Quick Thought task.
Revised: French, Spanish, and German respectively, which we use to train our Quick Thought task.

Original: Stanford Natural Language Inference (SNLI).
Revised: Stanford Natural Language Inference (SNLI).

Original: The NLI data we use is taken from the Stanford
Revised: The NLI data we use is taken from the Stanford

Original: Natural Language Inference (SNLI) dataset of Bowman et al.
Revised: Natural Language Inference (SNLI) dataset of Bowman et al.

Original: (2015), which consists of 570K sentence pairs associated with one of three labels: entailment , contradiction , or neutral .
Revised: (2015), which consists of 570K sentence pairs associated with one of three labels: entailment , contradiction , or neutral .

Original: The corpus is split into training (550K), validation (10K), and testing sets (10K).
Revised: The corpus is split into training (550K), validation (10K), and testing sets (10K).

Original: Translation.
Revised: Translation.

Original: The data for training the translation task is constructed using a system similar to the approach described by Guo et al.
Revised: The data for training the translation task is constructed using a system similar to the approach described by Guo et al.

Original: The ﬁnal constructed corpus contains around 600M en-fr pairs, 470M en-es pairs and 500M en-de pairs.
Revised: The ﬁnal constructed corpus contains around 600M en-fr pairs, 470M en-es pairs and 500M en-de pairs.

Original: 3. M ODEL C ONFIGURATION
Revised: 3. M ODEL C ONFIGURATION

Original: In all of our experiments, multi-task training is done by cycling through the different tasks (translation pairs, Reddit, Wikipedia, NLI) and performing an optimization step for a single task at a time.
Revised: In all of our experiments, multi-task training is done by cycling through the different tasks (translation pairs, Reddit, Wikipedia, NLI) and performing an optimization step for a single task at a time.

Original: We train all of our models with a batch size of 100 using stochastic gradient descent with a learning rate of 0.008.
Revised: We train all of our models with a batch size of 100 using stochastic gradient descent with a learning rate of 0.008.

Original: All of our models are trained for 30 million steps.
Revised: All of our models are trained for 30 million steps.

Original: All input text is tokenized prior to being used for training.
Revised: All input text is tokenized prior to being used for training.

Original: We build a vocab containing 200 thousand unigram tokens with 10 thousand hash buckets for out-of-vocabulary tokens.
Revised: We build a vocab containing 200 thousand unigram tokens with 10 thousand hash buckets for out-of-vocabulary tokens.

Original: The character n-gram vocab contains 200 thousand hash buckets used for 3 and 4 grams.
Revised: The character n-gram vocab contains 200 thousand hash buckets used for 3 and 4 grams.

Original: Both the word and character n-gram embedding sizes are 320.
Revised: Both the word and character n-gram embedding sizes are 320.

Original: All hyperparameters are tuned based on performance on the development portion (random 10% slice) of our datasets.
Revised: All hyperparameters are tuned based on performance on the development portion (random 10% slice) of our datasets.

Original: Finally, as an additional training heuristic, we multiply the gradients to the word and character embeddings by a factor of 100 3 .
Revised: Finally, as an additional training heuristic, we multiply the gradients to the word and character embeddings by a factor of 100 3 .

Original: We found that using this embedding gradient multiplier alleviated vanishing gradient issues and greatly improved training.
Revised: We found that using this embedding gradient multiplier alleviated vanishing gradient issues and greatly improved training.

Original: We compare the proposed cross-lingual multi-task (referred to simply as multi-task) models with baseline models that are trained using only the translation ranking task, which we dub as the “translation-ranking” models.
Revised: We compare the proposed cross-lingual multi-task (referred to simply as multi-task) models with baseline models that are trained using only the translation ranking task, which we dub as the “translation-ranking” models.

Original: 3. M ODEL P ERFORMANCE ON E NGLISH D OWNSTREAM T ASKS
Revised: 3. M ODEL P ERFORMANCE ON E NGLISH D OWNSTREAM T ASKS

Original: We ﬁrst evaluated all of our cross-lingual models on several downstream English tasks taken from
Revised: We ﬁrst evaluated all of our cross-lingual models on several downstream English tasks taken from

Original: SentEval (Conneau & Kiela, 2018) to verify the impact of cross-lingual training.
Revised: SentEval (Conneau & Kiela, 2018) to verify the impact of cross-lingual training.

Original: Each task is described in the supplementary material.
Revised: Each task is described in the supplementary material.

Original: Results on the tasks are summarized in Table 1.
Revised: Results on the tasks are summarized in Table 1.

Original: We note that cross-lingual training does not hinder the effectiveness of our encoder on English tasks, as the multi-task models are close to state-of-the-art in each of the downstream tasks.
Revised: We note that cross-lingual training does not hinder the effectiveness of our encoder on English tasks, as the multi-task models are close to state-of-the-art in each of the downstream tasks.

Original: For the Text REtrieval Conference (TREC) eval, we actually ﬁnd that our multi-task models outperform the previous state-of-the-art models by a sizable amount.
Revised: For the Text REtrieval Conference (TREC) eval, we actually ﬁnd that our multi-task models outperform the previous state-of-the-art models by a sizable amount.

Original: Table 1 also includes the results for our translation-ranking models on the same downstream English tasks.

Original: We include these results mainly to gauge the level of semantic information that can be learned from using only translation pair data.

Original: As expected, the performance of the translation-ranking models is signiﬁcantly worse than that of the multi-task models.

Original: [Table]

Original: 3. C ROSS - LINGUAL R ETRIEVAL
Revised: 3. C ROSS - LINGUAL R ETRIEVAL

Original: We also evaluate both the multi-task and translation-ranking models’ efﬁcacy in performing crosslingual retrieval by using held-out translation pair data.
Revised: We also evaluate both the multi-task and translation-ranking models’ efﬁcacy in performing crosslingual retrieval by using held-out translation pair data.

Original: Following Henderson et al.
Revised: Following Henderson et al.

Original: (2017), we use precision at N (P@N) as the evaluation metric by checking if a source sentence’s target translation ranks (where ranking is done using dot product) in the top N scored candidates when considering K other randomly selected target sentences.
Revised: (2017), we use precision at N (P@N) as the evaluation metric by checking if a source sentence’s target translation ranks (where ranking is done using dot product) in the top N scored candidates when considering K other randomly selected target sentences.

Original: Unlike Henderson et al.
Revised: Unlike Henderson et al.

Original: (2017), we set K to be 999 instead of 99. This is because using K = 99 results in all metrics quickly shooting up to 99%, which leads us to believe that choosing the correct translation out of only 100 samples is too easy.
Revised: (2017), we set K to beinstead of 99 because using K = 99 results in all metrics quickly shooting up to 99%.

Original: Table 2 summarizes the precision at 1 (P@1) metric of the multi-task models and translation-ranking models. The translation-ranking model remains as a strong baseline for ﬁnding the true translation, with up to 97.8% for the en-de retrieval task.
Revised: The translation-ranking model remains as a strong baseline for ﬁnding the true translation, with 95.4%, 87.5%, 97.5% P@1 for en-fr, en-es and en-de retrieval tasks respectively. The multi-task model performs almost identical with 95.1%, 88.8% and 97.8%, which provides empirical justiﬁca- [Table]

Original: The multi-task model performs almost identical to the translation-ranking model in all metrics, which provides some empirical justiﬁcation that it is possible to maintain embedding space alignment despite optimizing for native tasks in each individual language.
Revised: tion that it is possible to maintain embedding space alignment despite optimizing for native tasks in each individual language.

Original: We also experimented with P@3 and P@10, the results are identical.
Revised: We also experimented with P@3 and P@10, the results are identical

Original: [Table]
Revised: [Table]

Original: 3. M ULTILINGUAL STS
Revised: 3. M ULTILINGUAL STS

Original: We further test whether our learned cross-lingual representations can also perform well in their associated non-English language tasks by evaluating semantic textual similarity (STS) performance on French, Spanish, and German.
Revised: We further test whether our learned cross-lingual representations can also perform well in their associated non-English language tasks by evaluating semantic textual similarity (STS) performance on French, Spanish, and German.

Original: To evaluate Spanish-Spanish (es-es) STS, we use SemEval-2017 task 1 (STS17) track 3 of Cer et al.
Revised: To evaluate Spanish-Spanish (es-es) STS, we use SemEval-2017 task 1 (STS17) track 3 of Cer et al.

Original: (2017), which contains 250 Spanish sentence pairs with human labeled similarity scores.
Revised: (2017), which contains 250 Spanish sentence pairs with human labeled similarity scores.

Original: We also evaluate Spanish-English (es-en) STS by using the track 4(a) task 4 , which contains 250 en-es sentence pairs.
Revised: We also evaluate es-en STS by using the track 4(a) task 4 , which contains 250 en-es sentence pairs.

Original: Beyond English and Spanish, however, there are no standard STS datasets available for other languages.
Revised: Beyond English and Spanish, however, there are no standard STS datasets available for other languages.

Original: As such, we evaluate on a translated version of the STS Benchmark dataset from Cer et al.
Revised: As such, we evaluate on a translated version of the STS Benchmark dataset from Cer et al.

Original: (2017) for French, Spanish, and German.
Revised: (2017) for French, Spanish, and German.

Original: We use Google’s translation system to translate the STS
Revised: We use Google’s translation system to translate the STS

Original: Benchmark sentences to French, Spanish and German.
Revised: Benchmark sentences to French, Spanish and German.

Original: We believe that the results on our pseudomultilingual STS Benchmark dataset are expected to still be a reasonable indicator of multilingual semantic similarly performance, since the NMT encoder-decoder architecture for translation differs signiﬁcantly from our dual-encoder approach.
Revised: We believe that the results on our pseudomultilingual STS Benchmark dataset are expected to still be a reasonable indicator of multilingual semantic similarly performance, since the NMT encoder-decoder architecture for translation differs signiﬁcantly from our dual-encoder approach.

Original: Following Cer et al.
Revised: Following Cer et al.

Original: (2018), we ﬁrst compute the sentence encodings u, v of an STS sentence pair, and then score the sentence pair similarity based on the angular distance between the two vectors, − arccos (cid:16) 
Revised: (2018), we ﬁrst compute the sentence encodings u, v of an STS sentence pair, and then score the sentence pair similarity based on the angular distance between the two vectors, − arccos (cid:16) uv || u

Original: [Equation]
Revised: ||

Original: ||  (cid:17) .
Revised: || v || (cid:17) .

Original: Table 3 shows the Pearson’s correlation coefﬁcient of the STS tasks for all models.
Revised: Table 2 shows the Pearson’s correlation coefﬁcient of the STS tasks for all models.

Original: The ﬁrst column shows the trained model performance on original English STS Benchmark data.
Revised: The ﬁrst column shows the trained model performance on original English STS Benchmark data.

Original: Columns 2 to 4 shows the the performance on the other languages.
Revised: Columns 2 to 4 shows the the performance on the other languages.

Original: All multi-task models remain strong on the translated STS tasks, with around 0.77 for dev and 0.74 for test in all languages.
Revised: All multi-task models remain strong on the translated STS tasks, with around 0.77 for dev and 0.74 for test in all languages.

Original: Lastly, columns 5 and 6 shows the results of en-es models on STS17 tasks.
Revised: Lastly, columns 5 and 6 shows the results of en-es models on STS17 tasks.

Original: The un-tuned multi-task models achieve 0.827 for the es-es task and 0.769 for the es-en task.
Revised: The un-tuned multi-task models achieve 0.827 for the es-es task and 0.769 for the es-en task.

Original: As a point of reference, we also list the two best performing STS systems, Tian et al.
Revised: As a point of reference, we also list the two best performing STS systems, Tian et al.

Original: (2017) (ECNU) and Wu et al.
Revised: (2017) (ECNU) and Wu et al.

Original: (2017) (BIT), reported from Cer et al.
Revised: (2017) (BIT), reported from Cer et al.

Original: Our results are very close to these state-of-the-art feature engineered and mixed systems, which we describe in greater detail in the supplementary material.[Table]
Revised: Our results are very close to these state-of-the-art feature engineered and mixed systems, which we describe in greater detail in the supplementary material.

Original:  Z ERO - SHOT C LASSIFICATION
Revised:  Z ERO - SHOT C LASSIFICATION

Original: To evaluate the transfer learning capabilities of our models
Revised: To evaluate the transfer learning capabilities of our models, we examine how well the multi-task and translation-ranking encoders perform on zero-shot and few-shot classiﬁcation tasks.

Original: [Table]
Revised: [Table]

Original: we examine how well the multi-task and translation-ranking encoders perform on zero-shot and few-shot classiﬁcation tasks.

Original: 4. M ULTILINGUAL NLI
Revised: M ULTILINGUAL NLI

Original: Weﬁrst evaluate the zero-shot classiﬁcation performance of our multi-task models on  multilingual natural language inference (NLI) tasks.
Revised: We evaluate the zero-shot classiﬁcation performance of our multi-task models on two multilingual natural language inference (NLI) tasks.

Original:  We  make use of the professionally translated French and 
Revised: However, prior to doing so, we ﬁrst train a modiﬁed versionof our multi-task models that also includes training on the English Multi-genre NLI (MultiNLI) dataset of Williams et al. in addition to SNLI. We train with MultiNLI to be consistent with the baselines we compare to, which also use MultiNLI. First, we make use of the professionally translated French and Spanish subsets of SNLI created by

Original: Spanish subsets of SNLI created by Agi ´ c & Schluter (2017) for the cross-lingual zero-shot evaluation.
Revised: Agi ´ c & Schluter (2017) for an initial cross-lingual zero-shot evaluation of French and Spanish.

Original:  There are 1000 examples in the translated subsets for each language.
Revised: We refer to these translated subsets as SNLI-X. There are 1000 examples in the translated subsets for each language.

Original: To evaluate, we simply feed the French or Spanish examples into the pre-trained English NLI sub-network of our crosslingual models.
Revised: To evaluate, we simply feed the French and Spanish examples into the pre-trained English NLI sub-network of our multi-task models.

Revised: We also use the more recent dataset (XNLI) of Conneau et al.

Original:  Table 4 lists the accuracy on the English SNLI test set (10k sentence pairs) for all models and the accuracy on the French and Spanish translated subsets (1k sentence pairs) for  our en-fr and enes models.
Revised: (2018), which provides a means for multilingual NLI evaluation in Spanish, French, German, Chinese and more. Since XNLI provides non-European-language evaluations, we also train an English-Chinese (en-zh) version of our multitask model. There are 5000 examples in each XNLI test set, and zero-shot evaluation is once again done by feeding non-English examples into the pre-trained English NLI sub-network. Table 3 lists the accuracy on the English SNLI test set as well as on SNLI-X and XNLI for all of our multi-task models.

Original: The original English SNLI accuracies are around 84% for all of our multi-task models, indicating that English SNLI performance remains stable in the multi-task training setting.
Revised: The original English SNLI accuracies are around 84% for all of our multitask models, indicating that English SNLI performance remains stable in the multi-task training setting.

Original: The zeroshot accuracy on the translated subsets of SNLI are around 74% for both of French and Spanish.
Revised: The zero-shot accuracy on SNLI-X is around 74% for both the en-fr and en-es models. The zero-shot accuracy on XNLI is around 65% for en-es, en-fr, and en-de, and around 63% for en-zh, thereby signiﬁcantly outperforming the pretrained sentence encoding baselines (X-CBOW) described in Conneau et al. The X-CBOW baselines use ﬁxed sentence encoders that are the result of averaging tuned multilingual word embeddings.

Original: Row 4  shows the zero-shot French NLI performance of Eriguchi et al.
Revised: Row 4 of Table 3 shows the zero-shot French NLI performance of Eriguchi et al.

Original: (2018), which is a stateof-the-art zero-shot NLI classiﬁers based on multilingual NMT embeddings.
Revised: (2018), which is a state-of-the-art zero-shot NLI classiﬁers based on multilingual NMT embeddings.

Original: Our multi-task model shows comparable performance to the NMT-based model in both English and French.
Revised: Our multi-task model shows comparable performance to the NMT-based model in both English and French.

Original: 4. A MAZON R EVIEW
Revised: 4. A MAZON R EVIEW

Original: Zero-shot Learning.
Revised: Zero-shot Learning.

Original: We also conduct a zero-shot evaluation based on the Amazon review data extracted by Prettenhofer & Stein (2010).
Revised: We also conduct a zero-shot evaluation based on the Amazon review data extracted by Prettenhofer & Stein (2010).

Original: We preprocess the Amazon reviews and convert the data[Table] into a sentiment classiﬁcation task by considering reviews with strictly more than three stars as positive and strictly less than three stars as negative, in the same manner as Prettenhofer & Stein (2010).
Revised: We preprocess the Amazon reviews and convert the data into a sentiment classiﬁcation task by considering reviews with strictly more than three stars as positive and strictly less than three stars as negative, in the same manner as Prettenhofer & Stein (2010).

Original: Each review contains a summary ﬁeld and a text ﬁeld, which we concatenate to produce a single input.
Revised: Each review contains a summary ﬁeld and a text ﬁeld, which we concatenate to produce a single input.

Original: As the multi-task models are trained with sentence lengths clipped to 64, we only take the ﬁrst 64 tokens from the the concatenated text as the input.
Revised: As the multi-task models are trained with sentence lengths clipped to 64, we only take the ﬁrst 64 tokens from the the concatenated text as the input.

Original: There are 6000 training reviews in
Revised: There are 6000 training reviews in

Original: English, which we split into 90% for training and 10% for development.
Revised: English, which we split into 90% for training and 10% for development.

Revised: [Table]

Original: We ﬁrst encode inputs using the pre-trained multi-task and translation-ranking encoders and feed the encoded vectors into a 2-layer feed-forward network culminating in a softmax layer.
Revised: We ﬁrst encode inputs using the pre-trained multi-task and translation-ranking encoders and feed the encoded vectors into a 2-layer feed-forward network culminating in a softmax layer.

Original: We use layers of size 512 and tanh activation functions in each layer.
Revised: We use layers of size 512 and tanh activation functions in each layer.

Original: We use Adam for optimization with an initial learning rate of 0.0005 and a learning rate decay of 0.9 at every epoch during training.
Revised: We use Adam for optimization with an initial learning rate of 0.0005 and a learning rate decay of 0.9 at every epoch during training.

Original: We use a batch size of 16 and train for 20 total epochs in all experiments.
Revised: We use a batch size of 16 and train for 20 total epochs in all experiments.

Original: We freeze the cross-lingual encoder during training.
Revised: We freeze the cross-lingual encoder during training.

Original: The model architecture and parameters are tuned on the development set.
Revised: The model architecture and parameters are tuned on the development set.

Original: We ﬁrst train the classiﬁer on English data, and then evaluate it on the 6000 French and German
Revised: We ﬁrst train the classiﬁer on English data, and then evaluate it on the 6000 French and German

Original: Amazon review test examples.
Revised: Amazon review test examples.

Original: The results are summarized in Table 5.
Revised: The results are summarized in Table 4.

Original: The accuracy on the English test set is 87.4% for the en-fr model and 87.1% for the en-de model, with the zero-shot accuracy being above 80% for both models.
Revised: The accuracy on the English test set is 87.4% for the en-fr model and 87.1% for the en-de model, with the zero-shot accuracy being above 80% for both models.

Original: The translation-ranking models again perform worse on all metrics.
Revised: The translation-ranking models again perform worse on all metrics.

Original: Once again we compare the proposed model with Eriguchi et al.
Revised: Once again we compare the proposed model with Eriguchi et al.

Original: (2018), and ﬁnd that our zero-shot performance has a reasonable gain on the fr test set 5 .
Revised: (2018), and ﬁnd that our zero-shot performance has a reasonable gain on the fr test set 6 .

Original: [Table]
Revised: [Table]

Original: Few-shot Learning.
Revised: Few-shot Learning.

Original: We further evaluate the proposed multi-task models via few-shot learning, by training on English reviews and only a portion of French and German reviews.
Revised: We further evaluate the proposed multi-task models via few-shot learning, by training on English reviews and only a portion of French and German reviews.

Original: Our few-shot models are compared with baselines of training on French and German reviews only.
Revised: Our few-shot models are compared with baselines of training on French and German reviews only.

Original: Table 6 shows the classiﬁcation accuracy of the few-shot models, where the second row shows the percent of French and German data that is used when training each model.
Revised: Table 5 shows the classiﬁcation accuracy of the few-shot models, where the second row shows the percent of French and German data that is used when training each model.

Original: With as little as 20% of the French or
Revised: With as little as 20% of the French or

Original: German training data, the few-shot models perform nearly as good as the baseline models trained on 100% of the French and German data.
Revised: German training data, the few-shot models perform nearly as good as the baseline models trained on 100% of the French and German data.

Original: Adding more French and German training data leads to further improvements in few-shot model performance, with the few-shot models reaching 85.8% accuracy in French and 84.5% accuracy in German when using all of the French and German data.
Revised: Adding more French and German training data leads to further improvements in few-shot model performance, with the few-shot models reaching 85.8% accuracy in French and 84.5% accuracy in German when using all of the French and German data.

Original:  A NALYSIS OF C ROSS - LINGUAL E MBEDDING S PACES
Revised:  A NALYSIS OF C ROSS - LINGUAL E MBEDDING S PACES

Original: Motivated by the recent work of Søgaard et al.
Revised: Motivated by the recent work of Søgaard et al.

Original: (2018) studying the graph structure of multilingual word representations, we perform a similar analysis for our learned cross-lingual sentence representations.
Revised: (2018) studying the graph structure of multilingual word representations, we perform a similar analysis for our learned cross-lingual sentence representations.

Original: To do so, we take N samples of size K from en-fr, en-es, and en-de translation data and then encode these samples using the corresponding multi-task and translation-ranking models.
Revised: To do so, we take N samples of size K from en-fr, en-es, and en-de translation data and then encode these samples using the corresponding multi-task and translation-ranking models.

Original: We then
Revised: We then

Original: [Table]
Revised: [Table]

Original: compute pairwise distance matrices within each sampled set of encodings, and use these distance matrices to construct graph Laplacians 6 .
Revised: compute pairwise distance matrices within each sampled set of encodings, and use these distance matrices to construct graph Laplacians 7 .

Original: Finally, we obtain the similarity Ψ( S, T ) between each model’s source and target language embedding subsets by comparing the eigenvalues of the source language graph Laplacians to the eigenvalues of the target language graph Laplacians as follows:[Equation] [Table] [Equation]
Revised: Finally, we obtain the similarity Ψ( S, T ) between each model’s source and target language embedding subsets by comparing the eigenvalues of the source language graph Laplacians to the eigenvalues of the target language graph Laplacians as follows:

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Where L ( s ) i and L ( t ) i refer to the graph Laplacians of the source language and target language sentences obtained from the i th sample of source-target translation pairs.
Revised: Where L ( s ) i and L ( t ) i refer to the graph Laplacians of the source language and target language sentences obtained from the i th sample of source-target translation pairs.

Original: A smaller value of Ψ( S, T ) indicates higher eigen-similarity of the source language and target language embedding subsets.
Revised: A smaller value of Ψ( S, T ) indicates higher eigen-similarity of the source language and target language embedding subsets.

Original: Following Søgaard et al.
Revised: Following Søgaard et al.

Original: we use a sample size of K = 10 translation pairs, but we choose to use N = 1000 samples instead of N = 10 (as was done in their work) since we found Ψ( S, T ) to have very high variance at N = 10 .
Revised: we use a sample size of K = 10 translation pairs, but we choose to use N = 1000 samples instead of N = 10 (as was done in their work) since we found Ψ( S, T ) to have very high variance at N = 10 .

Original: The computed values of Ψ( S, T ) for our multi-task and translation-ranking models are summarized in Table 7.
Revised: The computed values of Ψ( S, T ) for our multi-task and translation-ranking models are summarized in Table 6.

Original: [Table]
Revised: [Table]

Original: We ﬁnd that the source and target embedding subsets constructed from the multi-task models exhibit greater average eigen-similarity than those resulting from the translation-ranking models for all source-target language pairs.
Revised: We ﬁnd that the source and target embedding subsets constructed from the multi-task models exhibit greater average eigen-similarity than those resulting from the translation-ranking models for all source-target language pairs.

Original: This result is not necessarily intuitive, since one might expect the translation-ranking model to optimize more for alignment.
Revised: This result is not necessarily intuitive, since one might expect the translation-ranking model to optimize more for alignment.

Original: Given that eigen-similarity correlates with the better performance of the multi-task models in almost all tasks, a potential direction for future work could be to introduce regularization penalties based on graph similarity in multitask training.
Revised: Given that eigen-similarity correlates with the better performance of the multi-task models in almost all tasks, a potential direction for future work could be to introduce regularization penalties based on graph similarity in multitask training.

Original: Interestingly, we also observe that the eigen-similarity gaps between the multi-task and translation-ranking models are not uniform across language pairs (although it may be that translation-ranking requires even more training).
Revised: Interestingly, we also observe that the eigen-similarity gaps between the multi-task and translation-ranking models are not uniform across language pairs (although it may be that translation-ranking requires even more training).

Original: Thus, another direction could be to further study differences in the difﬁculty of aligning different source-target language embeddings.
Revised: Thus, another direction could be to further study differences in the difﬁculty of aligning different source-target language embeddings.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we explored a straightforward framework for training cross-lingual, multi-task dualencoder models.
Revised: In this work, we explored a straightforward framework for training cross-lingual, multi-task dualencoder models.

Original: We showed that by training English-French, English-Spanish, and English-German multi-task models using our setup, we can achieve near-state-of-the-art or state-of-the-art performance in a variety of English tasks while also being able to produce similar caliber results in zeroshot transfer learning tasks for other languages.
Revised: We showed that by training English-French, English-Spanish, and English-German multi-task models using our setup, we can achieve near-state-of-the-art or state-of-the-art performance in a variety of English tasks while also being able to produce similar caliber results in zeroshot transfer learning tasks for other languages.

Original: Finally, we note that the fact that multi-task training can actually improve performance on some downstream English tasks (TREC) is particularly interesting, and believe that there are many possibilities for future explorations of cross-lingual model training.
Revised: Finally, we note that the fact that multi-task training can actually improve performance on some downstream English tasks (TREC) is particularly interesting, and believe that there are many possibilities for future explorations of cross-lingual model training.

Original: A S UPPLEMENTARY M ATERIAL A. D ATA P REPROCESSING
Revised: A S UPPLEMENTARY M ATERIAL A. D ATA P REPROCESSING

Original: In order to effectively use Reddit, Wikipedia, and translation data, we do a signiﬁcant amount of preprocessing on the raw data.
Revised: In order to effectively use Reddit, Wikipedia, and translation data, we do a signiﬁcant amount of preprocessing on the raw data.

Original: We describe our preprocessing procedures for each dataset in the following paragraphs.
Revised: We describe our preprocessing procedures for each dataset in the following paragraphs.

Original: Reddit.
Revised: Reddit.

Original: The raw Reddit corpus extracted by Al-Rfou et al.
Revised: The raw Reddit corpus extracted by Al-Rfou et al.

Original: (2016) consists of 2.4 billion posts and comments from Reddit between 2007 and 2016, making it our largest data source by an order of magnitude.
Revised: (2016) consists of 2.4 billion posts and comments from Reddit between 2007 and 2016, making it our largest data source by an order of magnitude.

Original: To preprocess this data for training our conversation response prediction task, we follow the same procedure as Yang et al.
Revised: To preprocess this data for training our conversation response prediction task, we follow the same procedure as Yang et al.

Original: Essentially, we consider Reddit comments and their children (response comments) as input-response pairs.
Revised: Essentially, we consider Reddit comments and their children (response comments) as input-response pairs.

Original: We ﬁlter out comments that have ≥characters, due to the limitations on the number of input tokens our models can accept.
Revised: We ﬁlter out comments that have ≥characters, due to the limitations on the number of input tokens our models can accept.

Original: We also remove comments that start with “https”, “@”, or “/r/”, and also remove comments whose authors have “bot” in their usernames.
Revised: We also remove comments that start with “https”, “@”, or “/r/”, and also remove comments whose authors have “bot” in their usernames.

Original: Lastly, we ﬁnd that Reddit comments also contain a small mix of nonEnglish text, and we ﬁlter out comments where the percentage of alphabetic characters is ≤ 70% .
Revised: Lastly, we ﬁnd that Reddit comments also contain a small mix of nonEnglish text, and we ﬁlter out comments where the percentage of alphabetic characters is ≤ 70% .

Original: As mentioned in the paper, the ﬁnal, processed Reddit data consists of 600 million input-response pairs.
Revised: As mentioned in the paper, the ﬁnal, processed Reddit data consists of 600 million input-response pairs.

Original: Since Reddit comments are predominantly English, we found the original data to be unsuitable for training encoders for non-English languages.
Revised: Since Reddit comments are predominantly English, we found the original data to be unsuitable for training encoders for non-English languages.

Original: To create conversational corpora for other languages, we translate the entire Reddit dataset using Google’s neural machine translation (NMT) system of
Revised: To create conversational corpora for other languages, we translate the entire Reddit dataset using Google’s neural machine translation (NMT) system of

Original: Wu et al.
Revised: Wu et al.

Original: Wikipedia.
Revised: Wikipedia.

Original: One concern with using the translated Reddit corpus as a monolingual task for nonEnglish languages is that the translated data will propagate any errors made by the NMT system used for translation.
Revised: One concern with using the translated Reddit corpus as a monolingual task for nonEnglish languages is that the translated data will propagate any errors made by the NMT system used for translation.

Original: To ameliorate this problem, we crawl Wikipedia and extract triplets of contiguous sentences from Wikipedia articles.
Revised: To ameliorate this problem, we crawl Wikipedia and extract triplets of contiguous sentences from Wikipedia articles.

Original: We use Wikipedia due to it having well-formed articles with a broad coverage of several languages.
Revised: We use Wikipedia due to it having well-formed articles with a broad coverage of several languages.

Original: We extract all Wikipedia articles for English, French, Spanish and German from the Wikipedia dump of May 5, 2018, and use the sentence splitter model of Gillick (2009) to split the articles into sentences.
Revised: We extract all Wikipedia articles for English, French, Spanish and German from the Wikipedia dump of May 5, 2018, and use the sentence splitter model of Gillick (2009) to split the articles into sentences.

Original: All article titles and article section names are treated as sentences inline.
Revised: All article titles and article section names are treated as sentences inline.

Original: As mentioned in the paper, our ﬁnal extracted Wikipedia corpus consists of 127. million English sentence triplets, 49.5 million French triplets, 29.8 million Spanish triplets, and 49.
Revised: As mentioned in the paper, our ﬁnal extracted Wikipedia corpus consists of 127. million English sentence triplets, 49.5 million French triplets, 29.8 million Spanish triplets, and 49.

Original: million German triplets for French, Spanish and German respectively.
Revised: million German triplets for French, Spanish and German respectively.

Original: Stanford Natural Language Inference (SNLI).
Revised: Stanford Natural Language Inference (SNLI).

Original: We use the English Stanford Natural Language
Revised: We use the English Stanford Natural Language

Original: Inference (SNLI) dataset consisting of 570K sentence pairs of Bowman et al.
Revised: Inference (SNLI) dataset consisting of 570K sentence pairs of Bowman et al.

Original: (2015) as is, without any further preprocessing.
Revised: (2015) as is, without any further preprocessing.

Original: We also use the provided splits for training (550K), validation (10K), and testing (10K).
Revised: We also use the provided splits for training (550K), validation (10K), and testing (10K).

Original: Translation.
Revised: Translation.

Original: The data for training the translation task is crawled from the public web using a system similar to the approach described by Uszkoreit et al.
Revised: The data for training the translation task is crawled from the public web using a system similar to the approach described by Uszkoreit et al.

Original: The extracted data is further cleaned by a pre-trained translation pair scoring system.
Revised: The extracted data is further cleaned by a pre-trained translation pair scoring system.

Original: We then generate “hard-negatives” following Guo et al.
Revised: We then generate “hard-negatives” following Guo et al.

Original: by using a pre-trained coarse translation-ranking model to determine translations that are close to a correct translation.
Revised: by using a pre-trained coarse translation-ranking model to determine translations that are close to a correct translation.

Original: As mentioned in the paper, the ﬁnal constructed corpus contains around 600M en-fr pairs, 470M en-es pairs and 500M en-de pairs.
Revised: As mentioned in the paper, the ﬁnal constructed corpus contains around 600M en-fr pairs, 470M en-es pairs and 500M en-de pairs.

Original: A. D OWNSTREAM T ASK D ESCRIPTIONS
Revised: A. D OWNSTREAM T ASK D ESCRIPTIONS

Original: Each of the English downstream tasks shown in Table 1 in the main body of the paper are described brieﬂy below:
Revised: Each of the English downstream tasks shown in Table 1 in the main body of the paper are described brieﬂy below:

Original: • Movie Reviews (MR): Multi-class prediction (scale of one to ﬁve stars) on movie review snippet data from Pang & Lee (2005).
Revised: • Movie Reviews (MR): Multi-class prediction (scale of one to ﬁve stars) on movie review snippet data from Pang & Lee (2005).

Original: • Customer Reviews (CR): Binary sentiment classiﬁcation of sentences from customer reviews mined by Hu & Liu (2004).
Revised: • Customer Reviews (CR): Binary sentiment classiﬁcation of sentences from customer reviews mined by Hu & Liu (2004).

Original: • Subjectivity (SUBJ): Binary classiﬁcation of sentences from movie reviews (Pang & Lee, 2004) as either objective or subjective.
Revised: • Subjectivity (SUBJ): Binary classiﬁcation of sentences from movie reviews (Pang & Lee, 2004) as either objective or subjective.

Original: • Multi-Perspective Question Answering (MPQA): Multi-class opinion polarity prediction on news data from Wiebe et al.
Revised: • Multi-Perspective Question Answering (MPQA): Multi-class opinion polarity prediction on news data from Wiebe et al.

Original: • Text REtrieval Conference (TREC): Multi-class classiﬁcation of questions obtained from TREC by Li & Roth (2002).
Revised: • Text REtrieval Conference (TREC): Multi-class classiﬁcation of questions obtained from TREC by Li & Roth (2002).

Original: • Stanford Sentiment Treebank (SST): Phrase-level binary sentiment classiﬁcation of text from Socher et al.
Revised: • Stanford Sentiment Treebank (SST): Phrase-level binary sentiment classiﬁcation of text from Socher et al.

Original: • Semantic Text Similarity
Revised: • Semantic Text Similarity

Original: (STS): Semantic textual similarity of sentence pairs in the form of Pearson correlation with human judgments (Cer et al., 2017).
Revised: (STS): Semantic textual similarity of sentence pairs in the form of Pearson correlation with human judgments (Cer et al., 2017).

Original: A. S TATE -O F -T HE -A RT STS S YSTEM D ESCRIPTIONS
Revised: A. S TATE -O F -T HE -A RT STS S YSTEM D ESCRIPTIONS

Original: For the STS evaluations provided in the body of our paper, we compared to the state-of-the-art systems of Tian et al.
Revised: For the STS evaluations provided in the body of our paper, we compared to the state-of-the-art systems of Tian et al.

Original: (2017) and Wu et al.
Revised: (2017) and Wu et al.

Original: We refer the reader to Cer et al.
Revised: We refer the reader to Cer et al.

Original: (2017) for a detailed summary of the characteristics of these systems, but we provide a brief characterization below for convenience:
Revised: (2017) for a detailed summary of the characteristics of these systems, but we provide a brief characterization below for convenience:

Original: • ECNU: The ECNU model is an ensemble of random forest, gradient boosted decision trees, and deep learning models that leverages several manually constructed features such as ngram overlap, edit distance, longest common preﬁx, common substrings, word alignments, and more.
Revised: • ECNU: The ECNU model is an ensemble of random forest, gradient boosted decision trees, and deep learning models that leverages several manually constructed features such as ngram overlap, edit distance, longest common preﬁx, common substrings, word alignments, and more.

Original: • BIT: The BIT model relies mostly on a feature called information content (IC), which is based on the likelihood of occurrence of a concept (in this case, a word).
Revised: • BIT: The BIT model relies mostly on a feature called information content (IC), which is based on the likelihood of occurrence of a concept (in this case, a word).

Original: Information content is then computed and combined for sentences through a hierarchical approach, which Wu et al.
Revised: Information content is then computed and combined for sentences through a hierarchical approach, which Wu et al.

Original: (2017) also augment with word embedding alignment to produce a ﬁnal ensemble model.
Revised: (2017) also augment with word embedding alignment to produce a ﬁnal ensemble model.

Original: A.  F URTHER E XPERIMENTS
Revised: A. A BLATION T ESTS AND F URTHER E XPERIMENTS

Original:  Given that the multi-task models detailed in the body of the paper are trained with an English NLI task but no non-English NLI tasks, we evaluate how much training without this English NLI task affects model performance on downstream English tasks.
Revised: No NLI. Given that the multi-task models detailed in the body of the paper are trained with an English NLI task but no non-English NLI tasks, we evaluate how much training without this English NLI task affects model performance on downstream English tasks.

Original: The performance of these no-NLI models are summarized under the “Cross-lingual Multitask Transformer No SNLI” section of Table 8.
Revised: The performance of these no-NLI models is summarized under the “Cross-lingual Multitask Transformer No SNLI” section of Table 7.

Original: We ﬁnd that training without SNLI leads to comparable or better performance on all English downstream tasks except for STS, where we ﬁnd training with SNLI provides a signiﬁcant bump in performance. 
Revised: We ﬁnd that training without SNLI leads to comparable or better performance on all English downstream tasks except for STS, where we ﬁnd training with SNLI provides a signiﬁcant bump in performance. No Wikipedia.

Revised: We similarly investigate whether there is a signiﬁcant advantage gained from using the only native target language data source, Wikipedia.

Revised: To do so, we train en-fr, en-es, and en-de multi-task models without the Wikipedia-based Quick Thought task.

Revised: The performance of these nowiki models is shown under the “Cross-lingual Multitask Transformer No Wiki” section of Table 7.

Revised: We ﬁnd that training without the Wikipedia Quick Thought task does not have a strong impact on model performance, with some tasks being marginally better and others being marginally worse.

Original:  Additionally, the cross-lingual, multi-task models discussed in the body of our paper use a largely parallel task setup, where for each source-target language pair the monolingual target tasks are mirrored from the monolingual source tasks (the only exception being NLI, which is English-only).
Revised: No target language native tasks. Additionally, the cross-lingual, multi-task models discussed in the body of our paper use a largely parallel task setup, where for each source-target language pair the monolingual target tasks are mirrored from the monolingual source tasks (the only exception being NLI, which is English-only).

Original: To test how useful this mirroring of monolingual tasks is, we train several models without target language monolingual tasks.
Revised: To test how useful this mirroring of monolingual tasks is, we train several models without target language monolingual tasks.

Original: We label these models as the “no non-English native task” models, and also summarize their downstream English task performance in Table 8.
Revised: We label these models as the “no non-English native task” models, and also summarize their downstream English task performance in Table 7.

Original: We note that removing the non-English native tasks actually leads to signiﬁcant decreases in performance on TREC, SST, and STS.
Revised: We note that removing the non-English native tasks actually leads to signiﬁcant decreases in performance on TREC, SST, and STS.

Original: We ﬁnd this quite interesting, as it provides some more empirical justiﬁcation for the notion that cross-lingual training for a source-target language pair can actually improve monolingual source task performance.
Revised: We ﬁnd this quite interesting, as it provides some more empirical justiﬁcation for the notion that cross-lingual training for a source-target language pair can actually improve monolingual source task performance.

Original: Finally, we also evaluate the zero-shot performance of the no non-English native task models on
Revised: Finally, we also evaluate the zero-shot performance of the no non-English native task models on

Original: Amazon review sentiment classiﬁcation.
Revised: Amazon review sentiment classiﬁcation.

Original: As can be seen in Table 9, training without the non-English native tasks leads to lower performance on English and roughly the same zero-shot classiﬁcation performance as training with only translation pair data.
Revised: As can be seen in Table 8, training without the non-English native tasks leads to lower performance on English and roughly the same zero-shot classiﬁcation performance as training with only translation pair data.

Original: [Table] [Table]
Revised: [Table] [Table]