Original: Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Task Analysis Anonymous EMNLP submission
Revised: L ANGUAGE M ODELING T EACHES Y OU M ORE S YNTAX THAN T RANSLATION D OES : L ESSONS L EARNED T HROUGH A UXILIARY T ASK A NALYSIS

Original: Abstract
Revised: A BSTRACT

Original: There is mounting evidence that pretraining can be valuable for neuralnetwork language understanding models, but we do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn.
Revised: Recent work using auxiliary prediction task classiﬁers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al., 2017), are so beneﬁcial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn.

Original: With this in mind, we compare four objectives— language modeling, translation, skip-thought, and autoencoding—on their ability to induce syntactic and part-of-speech information, holding constant thegenre and quantity  of  training data.
Revised: With this in mind, we compare four objectives—language modeling, translation, skip-thought, and autoencoding—on their ability to induce syntactic and part-of-speech information. We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture.

Original: We ﬁnd that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data, which suggests that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information.
Revised: We ﬁnd that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data. These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information.

Original: We also ﬁnd that a randomly-initialized, frozen model can perform strikingly well on our  auxiliary tasks, butthat this effect disappears when the amount of training data for the auxiliary tasks is reduced.
Revised: We also ﬁnd that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.

Original:  Introduction
Revised:  I NTRODUCTION

Original: Representation learning with deep recurrent neural networks has revolutionized natural language processing and replaced many of the expert-designed, linguistic features previously used.
Revised: Representation learning with deep recurrent neural networks has revolutionized natural language processing and replaced many of the expert-designed, linguistic features used previously.

Original: Recently, researchers have begun to investigate the properties ofrepresentations learned by networks by training auxiliary classiﬁers that use the hidden states of frozen pretrained models to perform other tasks.
Revised: Recently, researchers have begun to investigate the properties of learned representations by training auxiliary classiﬁers that use the hidden states of frozen, pretrained models to perform other tasks.

Original: These investigations have shown that when deep LSTM RNNs (Hochreiter and Schmidhuber, 1997) are trained on tasks likemachine translation, they latently identify substantial syntactic and semantic information about their input sentences, including part-of-speech (Shi et al., 2016; Belinkov et al., 2017a,b; Blevins et al., 2018).
Revised: These investigations have shown that when deep LSTMs (Hochreiter & Schmidhuber, 1997) are trained on tasks like translation, they learn substantial syntactic and semantic information about their input sentences, including part-of-speech (Shi et al., 2016; Belinkov et al., 2017a;b; Blevins et al., 2018).

Original: These intriguing ﬁndings lead us to ask the following questions:
Revised: These intriguing ﬁndings lead us to ask the following questions:

Original: How does the training task affect how well modelslatently learn syntactic properties?
Revised: How does the training task affect how well models learn syntactic properties?

Original: Which tasks are better at inducing these properties?
Revised: Which tasks are better at inducing these properties?

Original: How does the amount of data the model is trained on affect these results?
Revised: How does the amount of data the model is trained on affect these results?

Original: When does training on more data help?
Revised: When does training on more data help?

Original: We investigate these questions by holding the data source and model architecture constant, while varying both the training task and the amount of training data.
Revised: We investigate these questions by holding the data source and model architecture constant, while varying both the training task and the amount of training data.

Original: Speciﬁcally, we examine models trained on English-German (En-De) translation, language modeling, skip-thought (Kiros et al., 2015), and autoencoding, in addition to an untrained baseline model.
Revised: Speciﬁcally, we examine models trained on English-German (En-De) translation 1 , language modeling, skip-thought (Kiros et al., Although Belinkov et al. (2017a)

Original: We control for the data domain by exclusively training on datasets from the 2016 Conference on Machine Translation (WMT;
Revised: ﬁnd that translating into morphologically poorer languages leads to a slight improvement in encoder representations, we expect that our study of English-German translation will provide a reasonable overall picture of the representations that can be learned in data-rich translation. a r X i v : . 10040v [ c s .

Revised: C

Revised: L ]

Revised: J a n

Revised: 2015), and autoencoding, and also compare to an untrained LSTM model as a baseline.

Revised: We control for the data domain by exclusively training on datasets from the 2016 Conference on Machine

Original:  Bojar et al., 2016).
Revised: Translation (WMT; Bojar et al., 2016).

Original: We train models on all tasks using the parallel En-Decorpus and a small subset of that corpus, which allows us to makea fair comparison across all ﬁve models.
Revised: We train models on all tasks using the parallel En-De corpus, which allows us to make fair comparisons across tasks.

Revised: We also train models on a subset of the this corpus to examine the effect of training data volume on learned representations.

Original: Additionally, we augment the parallel dataset with a large monolingual corpus from WMT to examine how the performance of the unsupervised tasks (all but translation) scale with more data.
Revised: Additionally, we augment the parallel dataset with a large monolingual corpus from WMT to examine how the performance of the unsupervised tasks (all but translation) scale with more data.

Original: Throughout our work, we focus on the syntactic evaluation tasks of part-of-speech (POS) tagging and Combinatorial Categorical Grammar (CCG) supertagging.
Revised: Throughout our work, we focus on the syntactic evaluation tasks of part-of-speech (POS) tagging and Combinatorial Categorical Grammar (CCG) supertagging.

Original: Supertagging is  a building block for parsing as these tags constrain the ways in which words can compose, largely determining the parse of the sentence.
Revised: Supertagging is considered a building block for parsing as these tags constrain the ways in which words can compose, largely determining the parse of the sentence.

Original: CCG supertagging thus allows us to measure the degree to which models learn syntactic structure above the word.
Revised: CCG supertagging thus allows us to measure the degree to which models learn syntactic structure above the word.

Original: We focus our analysis on representations learned by language models and by the encoders of sequence-to-sequence models, as translation encoders have been found to learn richer representations of POS and morphological information than translation decoders (Belinkov et al., 2017a).
Revised: We focus our analysis on representations learned by language models and by the encoders of sequence-to-sequence models, as translation encoders have been found to learn richer representations of POS and morphological information than translation decoders (Belinkov et al., 2017a).

Original: We ﬁnd that for POS and CCG tagging, bidirectional language models (BiLMs)—created by separately training forward and backward language models, and concatenating their hidden states— outperform models trained on all other tasks.
Revised: We ﬁnd that for POS and CCG tagging, bidirectional language models (BiLMs)—created by separately training forward and backward language models, and concatenating their hidden states— outperform models trained on all other tasks.

Original: Even BiLMs trained on relatively small amounts of data (1 million sentences) outperform translation and skip-thought models trained on larger datasets ( million and 63 million sentences respectively).
Revised: Even BiLMs trained on relatively small amounts of data (1 million sentences) outperform translation and skip-thought models trained on larger datasets (5 million and 63 million sentences respectively).

Original: Our inclusion of an untrained LSTM baseline allows us to study the effect of training on  state representations.
Revised: Our inclusion of an untrained LSTM baseline allows us to study the effect of training on hidden state representations of LSTMs.

Original: We ﬁnd, surprisingly, thatrandomly initialized LSTMs underperform our best trained models by only a few percentage points when we use all of the available labeled  data to trainclassiﬁers for our auxiliary tasks.
Revised: We ﬁnd, surprisingly, that when we use all of the available labeled tag data to train our auxiliary task classiﬁers, our best trained models (BiLMs) only outperform the randomly initialized, untrained LSTMs by a few percentage points.

Original: When we reduce the amount of classiﬁer training data though, the performance of the randomly initialized LSTM model drops far below those of trained models.
Revised: When we reduce the amount of classiﬁer training data though, the performance of the randomly initialized LSTM model drops far below those of trained models.

Original: We hypothesize that this occurs because training the classiﬁers on large amounts of auxiliary task data allows them to memorize conﬁgurations of words seen in the training set and their associated tags.
Revised: We hypothesize that this occurs because training the classiﬁers on large amounts of auxiliary task data allows them to memorize conﬁgurations of words seen in the training set and their associated tags.

Original: We test this hypothesis by training classiﬁers to predict the identity of neighboring words from a given hidden state, and ﬁnd that randomly initialized models outperform all trained models on this task.
Revised: We test this hypothesis by training classiﬁers to predict the identity of neighboring words from a given hidden state, and ﬁnd that randomly initialized models outperform all trained models on this task.

Original: Our ﬁndings demonstrate that our best trained models do well on the tagging tasks because they are truly learning representations that conform to our notions of POS and CCG tagging, and not  because the classiﬁers we train are able to recover neighboring word identity information well.
Revised: Our ﬁndings demonstrate that our best trained models do well on the tagging tasks because they are truly learning representations that conform to our notions of POS and CCG tagging, and not simply because the classiﬁers we train are able to recover neighboring word identity information.  R ELATED W ORK

Original:  Adi et al.
Revised: Evaluating Learned Representations Adi et al.

Original: (2016) introduce the idea of examining sentence vector representations by training auxiliary classiﬁers to take sentence encodings and predict attributes like word order.
Revised: (2016) introduce the idea of examining sentence vector representations by training auxiliary classiﬁers to take sentence encodings and predict attributes like word order.

Original: Belinkov et al.
Revised: Belinkov et al.

Original: (2017a) build on this work by using classiﬁers to examine the hidden states of machine translation models in terms of what they learn about POS and morphology. They ﬁnd that translating into morphologically poorer languages leads to a slight improvement in encoder representations.
Revised: (2017a) build on this work by examining the hidden states of LSTMs trained on translation and ﬁnd that they learn substantial POS and morphological information without direct supervision for these linguistic properties.

Original: However, this effect is small, and we expect that our study of

Original: English-German translation will nonetheless provide a reasonable overall picture of the representations that can be learned in data-rich translation.

Original: Beyond translation, Blevins et al.
Revised: Beyond translation, Blevins et al.

Original: ﬁnd that deep LSTMslatently learn hierarchical syntax when trained on a variety of tasks—including semantic role labeling, language modeling, and dependency parsing.
Revised: ﬁnd that deep LSTMs learn hierarchical syntax when trained on a variety of tasks—including semantic role labeling, language modeling, and dependency parsing.

Revised: However, the models examined by Blevins et al.

Revised: (2018) were also trained on different datasets, so it’s unclear if the differences in syntactic task performance are due to the training objectives or simply differences in the training data.

Original: We build on this work by controlling for model size, and the quantity and genre of the training data, which allows us to make direct comparisons betweendifferent tasks on their ability to induce syntactic information.
Revised: By controlling for model size and the quantity and genre of the training data, we we are able to make direct comparisons between tasks on their ability to induce syntactic information.

Original:  of the work on sentence-level pretraining has focused on sentence-to-vector models and evaluating learned representations on how well they can be used to perform sentence-level classiﬁcation tasks.
Revised: Transfer Learning of Representations Much of the work on sentence-level pretraining has focused on sentence-to-vector models and evaluating learned representations on how well they can be used to perform sentence-level classiﬁcation tasks.

Original: Skip-thought (Kiros et al., 2015)—the technique of training a sequence-to-sequence model to predict the sentence preceding and following each sentence in a running text—represents a prominent early success in this area with unlabeled data, and InferSent (Conneau et al., 2017)—the technique of pretraining encoders on natural language inference data—yields strikingly better performance when such labeled data is available. 
Revised: A prominent early success in this area with unlabeled data is skip-thought (Kiros et al., 2015), the technique of training a sequence-to-sequence model to predict the sentence preceding and following each sentence in a running text. InferSent (Conneau et al., 2017)—the technique of pretraining encoders on natural language inference data— yields strikingly better performance when such labeled data is available. [Table]

Original: Work in this area has recently moved beyond strict sentence-to-vector mapping.
Revised: Work in transfer learning of representations has recently moved beyond strict sentence-to-vector mappings.

Revised: Dai & Le (2015) was one of the earliest works to use unsupervised pretraining of LSTMs for transfer learning on text classiﬁcation tasks.

Original: Newer models that incorporate LSTMs pretrained on datarich tasks, like translation and language modeling, have achieved state-of-the-art results on many tasks—including semantic role labeling and coreference resolution (Peters et al., 2018; McCann et al., 2017; Howard and Ruder, 2018).
Revised: Newer models that incorporate LSTMs pretrained on data-rich tasks, like translation and language modeling, have achieved state-of-the-art results on many tasks—including semantic role labeling and coreference resolution (Peters et al., 2018; McCann et al., 2017; Howard & Ruder, 2018).

Original: Although comparisons have previously been made between translation and language modeling as pretraining tasks (Peters et al., 2018), we investigate this issue more thoroughly by controlling for the quantity and content of the training data.
Revised: Although comparisons have previously been made between translation and language modeling as pretraining tasks (Peters et al., 2018), we investigate this issue more thoroughly by controlling for the quantity and content of the training data.

Original: Training Dataset Size
Revised: Training Dataset Size

Original: The performance of neural models depends immensely on the amount of training data used.
Revised: The performance of neural models depends immensely on the amount of training data used.

Original: Koehn and Knowles (2017) ﬁnd that when training machine translation models on corpora with fewer than 15 million words (English side), statistical machine translation approaches outperform neural ones.
Revised: Koehn & Knowles (2017) ﬁnd that when training machine translation models on corpora with fewer than 15 million words (English side), statistical machine translation approaches outperform neural ones.

Original: Similarly, Hestness et al.
Revised: Similarly, Hestness et al.

Original: (2017) study  data volumedependence on  several tasks—including translation and image classiﬁcation—and ﬁnd that for small amounts of data, neural models perform about as well as [Table] Layer [Table]
Revised: (2017) study the affect of training data volume on performance for several tasks—including translation and image classiﬁcation.

Original: [Table] [Table] [Table] [Table]
Revised: They ﬁnd that for small amounts of data, neural models perform about as well as chance.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] chance. After a certain threshold, model performance improves logarithmically with the amount of training data, but this eventually plateaus.
Revised: After a certain threshold, model performance improves logarithmically with the amount of training data, but this eventually plateaus.

Original: With this in mind, we also vary the amount of training data to investigate the relationship between performance and data volume for each task.
Revised: With this in mind, we also vary the amount of training data to investigate the relationship between performance and data volume for each task.

Original: Randomly Initialized Models Conneau et al.
Revised: Randomly Initialized Models Conneau et al.

Original: (2018) use randomly initialized LSTMs as a baseline when studying sentence-to-vector embedding models.
Revised: (2018) use randomly initialized LSTMs as a baseline when studying sentence-to-vector embedding models.

Original: They ﬁnd that untrained models outperform many trained models on several auxiliary tasks, including predicting word content.
Revised: They ﬁnd that untrained models outperform many trained models on several auxiliary tasks, including predicting word content.

Original: Similarly in vision, untrained convolutional networks have been shown to capture many low-level image statistics and can be used for image denoising (Ulyanov et al., 2017).
Revised: Similarly in vision, untrained convolutional networks have been shown to capture many low-level image statistics and can be used for image denoising (Ulyanov et al., 2017).

Original: Our method of training auxiliary classiﬁers on randomly initialized RNNs builds on the tradition of reservoir computing, in which randomly initialized networks or “reservoirs” are ﬁxed and only “read-out” classiﬁer networks are trained (Lukoˇseviˇcius and Jaeger, 2009).
Revised: Our method of training auxiliary classiﬁers on randomly initialized RNNs builds on the tradition of reservoir computing, in which randomly initialized networks or “reservoirs” are ﬁxed and only “read-out” classiﬁer networks are trained (Lukoˇseviˇcius & Jaeger, 2009).

Original: Echo state networks—reservoir computing with recurrent models—have been used for tasks like speech recognition, language modeling, and time series prediction (Verstraeten et al., 2006; Tong et al., 2007; Sun et al., 2017).
Revised: Echo state networks—reservoir computing with recurrent models—have been used for tasks like speech recognition, language modeling, and time series prediction (Verstraeten et al., 2006; Tong et al., 2007; Sun et al., 2017).

Original:  Methods 3. Main Training Data
Revised:  M ETHODS 3. M AIN T RAINING D ATA

Original: We use the parallel English-German (En-De) dataset from the 2016 ACL Conference on Machine Translation (WMT) shared task on news translation (Bojar et al., 2016).
Revised: We use the parallel English-German (En-De) dataset from the 2016 ACL Conference on Machine Translation (WMT) shared task on news translation (Bojar et al., 2016).

Original: This dataset contains 5 million ordered sentence translation pairs.
Revised: This dataset containsmillion ordered sentence translation pairs.

Original: We also use the 2015 English monolingual news  dataset from the same WMT shared task, which contains approximately 58 million ordered sentences.
Revised: We also use the 2015 English monolingual news discussion dataset from the same WMT shared task, which contains approximately 58 million ordered sentences.

Original: To examine how the volume of training data affects learned representations, we use four corpus sizes: 1, 5, 15, and 63 million sentences (translation is only trained on the smaller two sizes).
Revised: To examine how the volume of training data affects learned representations, we use four corpus sizes: 1, 5, 15, and 63 million sentences (translation is only trained on the smaller two sizes).

Original: We create the 1 million sentence corpora from the 5 million sentence dataset by sampling (i) sentence pairs for translation, (ii) English sentences for autoencoders, and (iii) ordered English sentence pairs for skip-thought and language models.
Revised: We create the 1 million sentence corpora from the 5 million sentence dataset by sampling (i) sentence pairs for translation, (ii) English sentences for autoencoders, and (iii) ordered English sentence [Figure]

Original: Note, we initialize language model
Revised: SoonS [Figure]

Original: LSTM states with the ﬁnal state after reading the previous sentence.
Revised: sheNP was [Figure] pairs for skip-thought and language models 2 .

Original: Similarly, we create  15 million sentence corpora for the unsupervised tasks by sampling sentences from the entire corpus of 63 million sentences.
Revised: Similarly, we create the 15 million sentence corpora for the unsupervised tasks by sampling sentences from the entire corpus of 63 million sentences.

Original: We use word-level representations throughout and use the Moses package (Koehn et al., 2007) to tokenize and truecase our data.
Revised: We use word-level representations throughout and use the Moses package (Koehn et al., 2007) to tokenize and truecase our data.

Original: Finally, we limit both the English and German vocabularies to the 50k most frequent tokens in the training set.
Revised: Finally, we limit both the English and German vocabularies to the 50k most frequent tokens in the training set.

Original: 3. Model Architecture and Training
Revised: 3. M ODEL A RCHITECTURE AND T RAINING

Original: We train all our models using OpenNMT-py (Klein et al., 2017) and use the default options for model sizes, hyperparameters, and training procedure—exceptthat we increase the size of the LSTMs, make the encoders bidirectional, and use validation-based learning rate decay instead of a ﬁxed schedule.
Revised: We train all our models using OpenNMT-py (Klein et al., 2017) and use the default options for model sizes, hyperparameters, and training procedure—except we increase the size of the LSTMs, make the encoders bidirectional, and use validation-based learning rate decay instead of a ﬁxed schedule.

Original: Speciﬁcally, all our models (except language models) are 1000D, twolayer encoder-decoder LSTMs with bidirectional encoders (500D LSTMs in each direction) and 500D embeddings; we train models both with and without attention (Bahdanau et al., 2015).
Revised: Speciﬁcally, all our models (except language models) are 1000D, two-layer encoder-decoder LSTMs with bidirectional encoders (500D per direction) and 500D embeddings. We train models both with and without attention (Bahdanau et al., 2015).

Original: For language models, we train asingle 1000D forward language model and a bidirectional language model—two 500D language models (one forward, one backward) trained separately, whose hidden states are concatenated.
Revised: For language models, we train a 1000D forward language model and a bidirectional language model—two 500D language models (forward and backward) trained separately, whose hidden states are concatenated.

Original: All models arerandomly initialized with a uniform distribution between − 0 . 1 and 0 . 1 , the default in OpenNMT-py.
Revised: All models, including our untrained baseline, are initialized from a uniform distribution ( − 0 . 1 , 0 . 1 ), the default in OpenNMT.

Original: We use the same training procedure for all our models.
Revised: We use the same training procedure for all our models.

Original: We evaluate on the validation set every epoch when training on the 1 and 5 million sentence datasets, and evaluate approximately every 5 million sentences when training on the larger datasets.
Revised: We evaluate on the validation set every epoch when training on the 1 and 5 million sentence datasets, and evaluate approximately everymillion sentences when training on the larger datasets.

Original: We use SGD with an initial learning rate of 1.
Revised: We use SGD with an initial learning rate of 1.

Original: Whenever a model’s validation loss increases relative to the previous evaluation, we halve the learning rate and stop training when the learning rate reaches 0 .
Revised: Whenever a model’s validation loss increases relative to the previous evaluation, we halve the learning rate and stop training when the learning rate reaches 0 .

Original: For each training task and dataset size, we select the model with the lowest validation perplexity to perform auxiliary task evaluations on.
Revised: For each training task and dataset size, we select the model with the lowest validation perplexity to perform auxiliary task evaluations on.

Original: We report model performance in terms of perplexity and BLEU (Papineni et al., 2002) in Table 1.
Revised: We report model performance in terms of perplexity and BLEU (Papineni et al., 2002) in Table 1.

Original: For translation we use beam search ( B = 5 ) when decoding.
Revised: For translation we use beam search ( B = 5 ) when decoding.

Original: 3. Classiﬁer Data and Architecture POS and CCG
Revised: 3. C LASSIFIER D ATA AND A RCHITECTURE

Original:  For Part-of-Speech (POS) tagging evaluation, we use the Wall Street Journal (WSJ) portion of the Penn Treebank (PTB; Marcus et al., 1993)
Revised: POS and CCG For Part-of-Speech (POS) tagging evaluation, we use the Wall Street Journal (WSJ) portion of the Penn Treebank (PTB; Marcus et al., 1993)

Original: We follow the standard WSJ split (train 2-21; dev 22; test 23).
Revised: We follow the standard WSJ split (train 2-21; dev 22; test 23).

Original: The dataset contains approximately 50k sentences and 45 tag types.
Revised: The dataset contains approximately 50k sentences and 45 tag types.

Original: For CCG supertagging, we use CCG Bank (Hockenmaier and Steedman, 2007), which is based on PTB WSJ.
Revised: For CCG supertagging, we use CCG Bank (Hockenmaier & Steedman, 2007), which is based on PTB WSJ.

Original: CCG supertagging provides ﬁne-grained information about the role of each word in its larger syntactic context and is considered almost parsing, since sequences of tags map sentences to small subsets of possible parses.
Revised: CCG supertagging provides ﬁne-grained information about the role of each word in its larger syntactic context and is considered almost parsing, since sequences of tags map sentences to small subsets of possible parses.

Original: The entire dataset contains approximately 50k sentences and 1327 tag types.
Revised: The entire dataset contains approximately 50k sentences andtag types.

Original: We display POS and CCG tags for an example sentence in Figure 
Revised: We display POS and CCG tags for an example sentence in Figure 1.

Original: To study the impact of auxiliary task  data volume, for both datasets we create smaller classiﬁer training sets by sampling 10% and 1% of the sentences.
Revised: To study the impact of auxiliary task training data volume, for both datasets we create smaller classiﬁer training sets by sampling 10% and 1% of the sentences.

Original: We truecase both datasets using the same truecase model trained on WMT and restrict the vocabularies to the 50k tokens used in  our LSTM models.
Revised: We truecase both datasets using the same truecase model trained on WMT and restrict the vocabularies to the 50k tokens used in pretraining our LSTM models.

Original: We use the word-conditional most frequent class as a baseline, which is the most frequently assigned tag class for each  word in the training set; for this baseline we restrict the vocabulary to that of our encoder models (we map all out-of-vocabulary words to a single UNK token).
Revised: In addition to the untrained LSTM baseline, we also compare to the word-conditional most frequent class (WC-MFC)—the most frequently assigned tag class for each distinct word in the training set. For this baseline we restrict the vocabulary to that of our LSTM models and map all out-of-vocabulary words to a single UNK token.

Original: Note that while PTB and WMT are both drawn from news text, there is slight genre mismatch.Soon RB she
Revised: Note that while PTB and WMT are both drawn from news text, there is slight genre mismatch.

Original: P RP was V BD running V BG the DT ofﬁce
Revised: Word Identity For this task, the classiﬁer takes a single LSTM hidden state as input and predicts the identity of the word at a different time step.

Original: NN

Original: [Figure]

Original: Soon

Original: S/S

Original: she

Original: NP was ( S \ NP ) /NP

Original: running

Original: NP > (

Original: [Equation]

Original: ) /NP

Original: the

Original: NP/N ofﬁce

Original: N

Original: [Equation]

Original: [Figure] [Equation] Word Identity For this task, the classiﬁer takes a single LSTM hidden state as input and predicts the identity of the word at a different time step, for example, three steps previous (shift of -3).
Revised: For example, for the sentence I love NLP and a time step shift of -2, we would train the classiﬁer to take the hidden state for NLP and predict the word I.

Original: We use the WSJ data for this task.
Revised: We use the WSJ dataset for this task.

Original: Following Conneau et al.
Revised: Following Conneau et al.

Original: (2018), we take all words that occur between 100 and 1000 times (about 1000 words total) as the possible targets for neighboring word prediction.
Revised: (2018), we take all words that occur between 100 and 1000 times (about 1000 words total) as the possible targets for neighboring word prediction.

Original:  We train multilayer perceptron (MLP) classiﬁers that take an
Revised: Classiﬁer Training Procedure We train multi-layer perceptron (MLP) classiﬁers that take an

Original: LSTM hidden state (from one time step and one layer) and output a distribution over the possible labels (tags or word identities).
Revised: LSTM hidden state (from one time step and one layer) and output a distribution over the possible labels (tags or word identities).

Original: The MLPs we train have a single 1000D hidden layer with a ReLU activation.
Revised: The MLPs we train have a single 1000D hidden layer with a ReLU activation.

Original: For classiﬁer training, we use the same training and learning rate decay procedure used for pretraining the  encoders. 
Revised: For classiﬁer training, we use the same training and learning rate decay procedure used for pretraining the LSTM encoders. [Figure]

Original:  Comparing Pretraining Tasks
Revised:  C OMPARING P RETRAINING T ASKS

Original: In this section, we discuss the main POS and CCG tagging results displayed in Figure 2.
Revised: In this section we discuss the main POS and CCG tagging results displayed in Figure 2.

Original: Overall,
Revised: Overall,

Original: POS and CCG tagging accuracies tend to increase with the amount of data the LSTM encoders are trained on. However, the amount of this improvement is generally small, especially when encoders are already trained on large amounts of data.
Revised: POS and CCG tagging accuracies tend to increase with the amount of data the LSTM encoders are trained on, but the marginal improvement decreases as the amount of training data increases.

Original:  For all pretraining dataset sizes and tasks, bidirectional language model (BiLM) and translation encoder representations perform best in terms of both POS and CCG tagging.
Revised: Language Modeling and Translation For all pretraining dataset sizes, bidirectional language model (BiLM) and translation encoder representations perform best on both POS and CCG tagging.

Original: Translation encoders, however, slightly underperform BiLMs, even when both models are trained on the same amount of data.
Revised: Translation encoders, however, slightly underperform BiLMs, even when both models are trained on the same amount of data.

Original: Interestingly, even BiLMs trained on the smallest amount of data (1 million sentences) outperform models trained on all other tasks and dataset sizes (up to 5 million sentences for translation, and 63 million sentences for skip-thought and autoencoding).
Revised: In fact, even BiLMs trained on the smallest amount of data (1 million sentences) outperform models trained on all other tasks and dataset sizes (up to 5 million sentences for translation, and 63 million sentences for skip-thought and autoencoding).

Original: The consistent superior performance of 
Revised: Especially since BiLMs do not require aligned data to train, the superior performance of BiLM representations on these tasks suggests that BiLMs (like ELMo; Peters et al., 2018) are better than translation encoders (like CoVe; McCann et al., 2017) for transfer learning of syntactic information.

Original: BiLMs—along with the fact that language models do not require aligned data—suggests that for transfer learning of syntactic information, BiLMs are superior to translation encoders.
Revised: One reason BiLMs perform relatively well on these syntactic tasks could be that in contrast to the encoders for the other tasks, LM encoders have a per-token loss.

Revised: Note also that since our evaluation tasks also predict a single label for each token, this could be one reason that BiLMs perform so well on these tasks in particular.

Original: For all amounts of training data, the BiLMs signiﬁcantly outperform the 1000D forward-only language models.
Revised: For all amounts of training data, the BiLMs signiﬁcantly outperform the 1000D forward-only language models.

Original: The gap in performance between bidirectional and forward language models is greater for CCG supertagging than for POS tagging.
Revised: The gap in performance between bidirectional and forward language models is greater for CCG supertagging than for POS tagging.

Original: When using all available auxiliary training data, there is a 2 and 8 percentage point performance gap on POS and CCG tagging respectively.
Revised: When using all available auxiliary training data, there is a 2 and 8 percentage point performance gap in POS and CCG tagging respectively.

Original: This difference in relative performance suggests that bidirectional context information is more important when identifying syntactic structure than when identifying part of speech.
Revised: This difference in relative performance suggests that bidirectional context information is more important for identifying syntactic structure than for identifying part of speech.

Original: Figure 2also illustrates how the best performing BiLMs and translation models tend to be more robust to decreases in classiﬁer data than models trained on other tasks.
Revised: Figure 2 illustrates how the best performing BiLMs and translation models tend to be more robust to decreases in classiﬁer data than models trained on other tasks.

Original: When training on less auxiliary task data, POS tagging performance tends to drop less than CCG supertagging performance.[Figure]
Revised: Also, when training on less auxiliary task data, POS tagging performance tends to drop less than CCG supertagging performance.

Original: percentage For the best model (BiLM trained on 63 million sentences), when using 1% rather than all of the auxiliary task training data, CCG accuracy dropspercentage points, while POS accuracy only drops 2 points.
Revised: For the best model (BiLM trained on 63 million sentences), when using 1% rather than all of the auxiliary task training data, CCG accuracy drops 9 percentage points, while POS accuracy only drops 2 points.

Original: Further examinations of the effect of classiﬁer data volume are displayed in Figure 3.
Revised: Further examinations of the effect of classiﬁer data volume are displayed in Figure 3.

Original:  Although skip-thought encoders consistently underperform both BiLMs and translation encoders in all data regimes we examine, skip-thought models improve the most when increasing the amount of pretraining data, and are the only models whose performance does not seem to have plateaued by 63 million training sentences.
Revised: Skip-Thought Although skip-thought encoders consistently underperform both BiLMs and translation encoders in all data regimes we examine, skip-thought models improve the most when increasing the amount of pretraining data, and are the only models whose performance does not seem to have plateaued by 63 million training sentences.

Original: Skip-thought models without attention are very similar to language models—the main difference is that while skip-thought models have separate encoder and decoder weights (and a bidirectional encoder), language models share weights between the encoder and decoder.

Original: Thus language models can be interpreted as  regularized versions  of skip-thought.
Revised: Since we train our language models on ordered sentences, as we do for skip-thought, our language models can be interpreted as a regularized versions [Figure] of skip-thought,

Revised: in which the weights of the encoder and decoder are shared.

Original: The increased model capacity of skip-thought, compared to language models, could explain the difference in learned representation quality—especially when these models are trained on smaller amounts of data.
Revised: The increased model capacity of skip-thought, compared to language models, could explain the difference in learned representation quality—especially when these models are trained on smaller amounts of data.

Original: Random Initialization For our randomly initialized, untrained LSTM encoders we use the default weight initialization technique in OpenNMTpy, a uniform distribution between -0.1 and 0.1; the only change we make is to set all biases to zero.
Revised: Random Initialization For our randomly initialized, untrained LSTM encoders, we use the default weight initialization technique in OpenNMT-py, a uniform distribution between -0.1 and 0.1; the only change we make is to set all biases to zero.

Original: We ﬁnd that this baseline performs quite well when using all auxiliary data, and is only 3 and 8 percentage points behind the BiLM on POS and
Revised: We ﬁnd that this baseline performs quite well when using all auxiliary data, and is only 3 and 8 percentage points behind the BiLM on POS and

Original: CCG tagging, respectively.
Revised: CCG tagging, respectively.

Original: We ﬁnd that decreasing the amount of classiﬁer data leads to a signiﬁcantly greater drop in the randomly initialized encoder performance compared to trained models.
Revised: We ﬁnd that decreasing the amount of classiﬁer data leads to a significantly greater drop in the untrained encoder performance compared to trained models.

Original: In the 1% classiﬁer data regime, the performance of untrained encoders on both tasks drops below that of all trained models and below even the wordconditional most-frequent class baseline.
Revised: In the 1% classiﬁer data regime, the performance of untrained encoders on both tasks drops below that of all trained models and below even the word-conditional most-frequent class baseline.

Original: We hypothesize that the randomly initialized baseline is able to perform well on tagging tasks with large amounts of auxiliary task training data, because the classiﬁer can learn the identity of neighboring words from a given time step’s hidden state, and simply memorize word conﬁgurations and their associated tags from the training data.
Revised: We hypothesize that the randomly initialized baseline is able to perform well on tagging tasks with large amounts of auxiliary task training data, because the classiﬁer can learn the identity of neighboring words from a given time step’s hidden state, and simply memorize word conﬁgurations and their associated tags from the training data.

Original: We test this hypothesis directly in Sectionand ﬁnd that untrained LSTM representations are in fact better at capturing neighboring word identity information than any trained model.
Revised: We test this hypothesis directly in Section 6 and ﬁnd that untrained LSTM representations are in fact better at capturing neighboring word identity information than any trained model.

Original:  Models trained on autoencoding are the only ones that do not consistently improve with the amount of training data, which is unsurprising as unregularized autoencoders are prone to learning identity mappings (Vincent et al., 2008).
Revised: Autoencoder Models trained on autoencoding are the only ones that do not consistently improve with the amount of training data, which is unsurprising as unregularized autoencoders are prone to learning identity mappings (Vincent et al., 2008).

Original: When training on 10% and 1% of the auxiliary task data, autoencoders outperform randomly initialized encoders and match the word-conditional most frequent class baseline.
Revised: When training on 10% and 1% of the auxiliary task data, autoencoders outperform randomly initialized encoders and match the word-conditional most frequent class baseline.

Original: When training on all the auxiliary data though, untrained encoders outperform autoencoders.
Revised: When training on all the auxiliary data though, untrained encoders outperform autoencoders.

Original: These results suggest that autoencoders learn some useful structure that[Figure] is useful in the low auxiliary data regime.
Revised: These results suggest that autoencoders learn some useful structure that is useful in the low auxiliary data regime.

Original: However, the representations autoencoders learn do not capture syntactically rich features, since random encoders outperform them in the high auxiliary data regime.
Revised: However, the representations autoencoders learn do not capture syntactically rich features, since random encoders outperform them in the high auxiliary data regime.

Original: This conclusion is further supported by the extremely poor performance of the second layer of an autoencoder without attention on POS tagging (almost 10 percentage points below the most frequent class baseline), as seen in Figure 4a.
Revised: This conclusion is further supported by the extremely poor performance of the second layer of an autoencoder without attention on POS tagging (almost 10 percentage points below the most frequent class baseline), as seen in Figure 4a.

Original:  Comparing Layers Embeddings (Layer 0)
Revised:  C OMPARING L AYERS

Revised: Embeddings (Layer 0)

Original: We ﬁnd that randomly initialized embeddings consistently perform as well as the word-conditional most frequent class baseline on POS and CCG tagging, which serves as an upper bound on performance for the embedding layer.
Revised: We ﬁnd that randomly initialized embeddings consistently perform as well as the word-conditional most frequent class baseline on POS and CCG tagging, which serves as an upper bound on performance for the embedding layer.

Original: As these  are untrained, the auxiliary classiﬁers are learning to memorize and classify the random vectors.
Revised: As these embeddings are untrained, the auxiliary classiﬁers are learning to memorize and classify the random vectors.

Original: When using all the auxiliary classiﬁer data, there is no signiﬁcant difference in the performance of trained and untrained embeddings on the tagging tasks.
Revised: When using all the auxiliary classiﬁer data, there is no signiﬁcant difference in the performance of trained and untrained embeddings on the tagging tasks.

Original: Only for smaller amounts of classiﬁer data do trained embeddings consistently outperform randomly initialized ones.
Revised: Only for smaller amounts of classiﬁer data do trained embeddings consistently outperform randomly initialized ones.

Original: Upper Layers Belinkov et al. (2017a)
Revised: Upper Layers Belinkov et al. (2017a)

Original: ﬁnd that, for translation models, the ﬁrst layer consistently outperforms the second on POS tagging.
Revised: ﬁnd that, for translation models, the ﬁrst layer consistently outperforms the second on POS tagging.

Original: We ﬁnd that this pattern holds for all our models, except in BiLMs, where the ﬁrst and second layers perform equivalently.
Revised: We ﬁnd that this pattern holds for all our models, except in BiLMs, for which the ﬁrst and second layers perform equivalently.

Original: The pattern holds even for untrained models, suggesting that POS information is stored on the lower layer, not  because the training task encourages this, but because offundamental properties of the deep LSTM architecture.
Revised: The pattern holds even for untrained models, suggesting that POS information is stored on the lower layer, not necessarily because the training task encourages this, but because of properties of the deep LSTM architecture.

Original: We also ﬁnd that for CCG supertagging, the ﬁrst layer also outperforms the second layer on untrained models.
Revised: We also ﬁnd that for CCG supertagging, the ﬁrst layer also outperforms the second layer on untrained models.

Original: For the trained models though,behavior is mixed, with the second layer performing better  in some cases.
Revised: For the trained models though, the second layer performs better than the ﬁrst in some cases.

Original: Which layer performs best appears to be independent of absolute performance on the supertagging task.
Revised: Which layer performs best appears to be independent of absolute performance on the supertagging task.

Original: Our layer analysis results are displayed in Figure 4.
Revised: Our layer analysis results are displayed in Figure 4.

Original:  Word Identity Prediction
Revised:  W ORD I DENTITY P REDICTION

Original: Our results on word identity prediction are summarized in Figure 5 and given in more detail in
Revised: Our results on word identity prediction are summarized in Figure 5 and given in more detail in

Original: Appendix A. We ﬁnd thatrandomly initialized LSTMs outperform all trainedmodels. We hypothesize this occurs because a kind of useful forgetting occurs during training, as the LSTMs learn that information about certain word patterns are more important to remember than others.
Revised: Appendix A. While trained encoders outperform untrained ones on both POS and CCG tagging, we ﬁnd that all trained LSTMs underperform untrained ones on word identity prediction.

Original: In this regard, randomly initialized models are less biased and process inputs more uniformly. The fact that untrained encoders outperform trained ones on word identity prediction, but underperform trained models on POS and CCG tagging, conﬁrms that trained models genuinely capture substantial syntactic features, beyond mere word identity, that the auxiliary classiﬁers can use.Effect of Depth
Revised: This ﬁnding conﬁrms that trained encoders genuinely capture substantial syntactic features, beyond mere word identity, that the auxiliary classiﬁers can use.

Original: We ﬁnd that for both trained and untrained models, the ﬁrst layer outperforms the second layer when predicting the identity of the immediate neighbors of a word.
Revised: We ﬁnd that for both trained and untrained models, the ﬁrst layer outperforms the second layer when predicting the identity of the immediate neighbors of a word.

Original: However, the second layer tends to outperform the ﬁrst at predicting the identity of more distant neighboring words.
Revised: However, the second layer tends to outperform the ﬁrst at predicting the identity of more distant neighboring words.

Original: This effect is especially apparent for the randomly initialized encoders.
Revised: This effect is especially apparent for the randomly initialized encoders.

Original: Our ﬁndings suggest that, as is the case for convolutional neural networks, depth in recur- neural networks has the effect of increasing the receptive ﬁeld and allows the upper layers to have representations that capture a larger context.
Revised: Our ﬁnding suggests that, as is the case for convolutional neural networks, depth in recurrent neural networks has the effect of increasing the receptive ﬁeld and allows the upper layers to have representations that capture a larger context.

Original: These results reﬂect the ﬁndings of Blevins et al.
Revised: These results reﬂect the ﬁndings of Blevins et al.

Original:  that for trained models, upper levels of LSTMs encode more abstract syntactic information, since more abstract information generally requires larger context information.
Revised: (2018) that for trained models, upper levels of LSTMs encode more abstract syntactic information, since more abstract information generally requires larger context information.

Original:  Conclusion
Revised:  C ONCLUSION

Original: By controlling for the genre and quantity of the training data, we make fair comparisons between several data-rich training tasks in their ability to induce syntactic information.
Revised: By controlling for the genre and quantity of the training data, we make fair comparisons between several data-rich training tasks in their ability to induce syntactic information.

Original: We ﬁnd that bidirectional language models (BiLMs) do better than translation and skip-thought encoders at extracting useful features for POS tagging and CCG supertagging.
Revised: We ﬁnd that bidirectional language models (BiLMs) do better than translation and skip-thought encoders at extracting useful features for POS tagging and CCG supertagging.

Original: Moreover, this improvement holds even when the BiLMs are trained on substantially less data than competing models.
Revised: Moreover, this improvement holds even when the BiLMs are trained on substantially less data than competing models.

Original: Although, due to limited parallel data, we could not compare BiLMs and translation encoders on more thanmillion sentences, our results suggest that for syntactic information, there is no need to compare these two models trained on more data, as BiLMs consistently outperform translation encoders in all data regimes.
Revised: Our results suggest that for transfer learning, BiLMs like ELMo (Peters et al., 2018) capture more useful features than translation encoders—and that this holds even on genres for which data is not abundant.

Original: We also ﬁnd that randomly initialized encoders extract usable features for POS and CCG tagging, at least when the auxiliary POS and CCG classiﬁers are themselves trained on reasonably large amounts of data.
Revised: We also ﬁnd that randomly initialized encoders extract usable features for POS and CCG tagging— at least when the auxiliary POS and CCG classiﬁers are themselves trained on reasonably large amounts of data.

Original: However, the performance of untrained models drops sharply relative to trained ones when using smaller amounts of the classiﬁer data.
Revised: The performance of untrained models drops sharply relative to trained ones when using smaller amounts of the classiﬁer data.

Original: We investigate further and ﬁnd that untrained models outperform trained ones on the task of neighboring word identity prediction, which conﬁrms that trained encoders do not perform well on tagging tasks because the classiﬁers are simply memorizing word identity information.
Revised: We investigate further and ﬁnd that untrained models outperform trained ones on the task of neighboring word identity prediction, which conﬁrms that trained encoders do not perform well on tagging tasks because the classiﬁers are simply memorizing word identity information.

Original: We also ﬁnd that both trained and untrained LSTMs store more local neighboring word identity information in lower layers and more distant word identity information in upper layers, which suggests that depth in LSTMs allow them to capture larger context information.
Revised: We also ﬁnd that both trained and untrained LSTMs store more local neighboring word identity information in lower layers and more distant word identity information in upper layers, which suggests that depth in LSTMs allow them to capture larger context information.

Original: Our results suggest that for transfer learning, bidirectional language models like ELMo (Peters et al., 2018) capture more useful features than translation encoders—and that this holds even on genres or languages for which data is not abundant. However, the scope of our experiments is limited, and we still know little about the representations of models trained on other supervised tasks, or precisely how the choice of training task affects the type of syntactic information that is learned. Our work also highlights the interesting behavior of randomly initialized LSTMs, which show an ability to preserve the contents of their inputs signiﬁcantly better than trained models. A Randomly Initialized Encoders
Revised: A CKNOWLEDGMENTS

Original: Training task Data Attention
Revised: We thank Nikita Nangia, Alex Wang, Phu Mon Htut, Katharina Kann, Chris Barker, and Sebastian

Original: POS L

Original: POS L

Original: POS L CCG L CCG L CCG L
Revised: Gehrmann for feedback on this project and paper.

Original: Random Init

Original: None

Original: N/A 90.

Original: 71.6 Random Init

Original: None

Original: N/A 90.

Original: Translation

Original: Parallel WMT16 (1M)

Original: Yes 95. 71.5 Translation Parallel WMT16 (1M) No 92.
Revised: We acknowledge the support of NVIDIA

Original: 71.3 LM (Bidir)

Original: Parallel WMT16 (1M)

Original: No 96.

Original: 71.6 LM (Forward)

Original: Parallel WMT16 (1M) No 94. 71.5 Skip-thought Parallel WMT16 (1M)
Revised: Corporation with the donation of the Titan X Pascal GPU used for this research.

Original: Yes 44.

Original: 71.1 Skip-thought

Original: Parallel WMT16 (1M)

Original: No 78.

Original: 71.1 Autoencoder

Original: Parallel WMT16 (1M)

Original: Yes 80. 71.2 Autoencoder Parallel WMT16 (1M) No 79.
Revised: SB acknowledges support from Google and Samsung Research.

Original: [Table]

Original: Translation

Original: Parallel WMT16 (5M)

Original: Yes 96.

Original: [Table]

Original: 71.5 Translation Parallel WMT16 (5M) No 92. 71.5 LM (Bidir)
Revised: B POS AND CCG E VALUATION F ULL R ESULTS B. T RAINING C LASSIFIERS ON A LL D ATA

Original: Parallel WMT16 (5M)

Original: No 96.

Original: [Table]
Revised: [Table]

Original: 71.6 LM (Forward)

Original: Parallel WMT16 (5M)

Original: No 94.

Original: 71.5 Skip-thought 
Revised: 71.5 Skip-thought 5M

Original: Parallel WMT16 (5M)

Original: Yes 76.
Revised: Yes 76.

Original: 71.1 Skip-thought 
Revised: 71.1 Skip-thought 5M

Original: Parallel WMT16 (5M)

Original: No 86.
Revised: No 86.

Original: 71.2 Autoencoder 
Revised: 71.2 Autoencoder 5M

Original: Parallel WMT16 (5M)

Original: Yes 88.
Revised: Yes 88.

Original: 70.8 Autoencoder 
Revised: 70.8 Autoencoder 5M

Original: Parallel WMT16 (5M)

Original: No 70.
Revised: No 70.

Original: LM (Bidir) 
Revised: LM (Bidir) 15M

Original: Parallel + Monolingual (15M)

Original: No 97.
Revised: No 97.

Original: 72.0 LM (Forward) 
Revised: 72.0 LM (Forward) 15M

Original: Parallel + Monolingual (15M)

Original: No 95.
Revised: No 95.

Original: 72.0 Skip-thought 
Revised: 72.0 Skip-thought 15M

Original: Parallel + Monolingual (15M)

Original: Yes 82.
Revised: Yes 82.

Original: 71.6 Skip-thought 
Revised: 71.6 Skip-thought 15M

Original: Parallel + Monolingual (15M)

Original: No 90.
Revised: No 90.

Original: 71.5 Autoencoder 
Revised: 71.5 Autoencoder 15M

Original: Parallel + Monolingual (15M)

Original: Yes 91.
Revised: Yes 91.

Original: 71.4 Autoencoder 
Revised: 71.4 Autoencoder 15M

Original: Parallel + Monolingual (15M)

Original: No 71.
Revised: No 71.

Original: LM (Bidir) 
Revised: LM (Bidir) 63M

Original: Parallel + Monolingual (63M)

Original: No 96.
Revised: No 96.

Original: 72.0 LM (Forward) {+
+}
Revised: 72.0 LM (Forward) {+
+}

Original: Parallel + Monolingual (63M)
Revised: M

Original: No 95.
Revised: No 95.

Original: 72.0 Skip-thought 
Revised: 72.0 Skip-thought 63M

Original: Parallel + Monolingual (63M)

Original: Yes 90.
Revised: Yes 90.

Original: 71.6 Skip-thought 
Revised: 71.6 Skip-thought 63M

Original: Parallel + Monolingual (63M)

Original: No 91.
Revised: No 91.

Original: 71.6 Autoencoder 
Revised: 71.6 Autoencoder 63M

Original: Parallel + Monolingual (63M)

Original: Yes 89.
Revised: Yes 89.

Original: 71.2 Autoencoder Parallel + Monolingual (63M) No 70.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B. Training Classiﬁers on 10% of Data

Original: Training task
Revised: Training task

Original: Data
Revised: Data

Original: Attention
Revised: Attention

Original: POS L
Revised: POS L

Original: POS L POS L
Revised: POS L POS L

Original: CCG L CCG L
Revised: CCG L CCG L

Original: CCG L
Revised: CCG L

Original: [Table]
Revised: Random Init

Original: None
Revised: None

Original: [Table]
Revised: N/A 85.

Original: [Table] [Table] [Table] [Table]
Revised: 68.3 Random Init

Original: None
Revised: None

Original: [Table] [Table] [Table] [Table]
Revised: N/A 84.

Original: Translation 
Revised: Translation 1M

Original: [Table] Yes 
Revised: Yes 93.

Original: [Table]
Revised: 69.5 Translation 1M

Original: No [Table]
Revised: No 89.

Original: 69.5 LM (Bidir) Parallel WMT16 (1M)
Revised: 69.5 LM (Bidir) 1M

Original: No 95.
Revised: No 95.

Original: 70.1 LM Forward 
Revised: 70.1 LM Forward 1M

Original: Parallel WMT16 (1M)

Original: No 93.
Revised: No 93.

Original: 69.9 Skip-thought 
Revised: 69.9 Skip-thought 1M

Original: Parallel WMT16 (1M)

Original: Yes 34.
Revised: Yes 34.

Original: 68.3 Skip-thought 
Revised: 68.3 Skip-thought 1M

Original: Parallel WMT16 (1M)

Original: No 71.
Revised: No 71.

Original: 68.1 Autoencoder 
Revised: 68.1 Autoencoder 1M

Original: Parallel WMT16 (1M)

Original: Yes 77.
Revised: Yes 77.

Original: 68.3 Autoencoder 
Revised: 68.3 Autoencoder 1M

Original: Parallel WMT16 (1M)

Original: No 71.
Revised: No 71.

Original: Translation 
Revised: Translation 5M

Original: Parallel WMT16 (5M)

Original: Yes 94.
Revised: Yes 94.

Original: 69.8 Translation 
Revised: 69.8 Translation 5M

Original: Parallel WMT16 (5M)

Original: No 89.
Revised: No 89.

Original: 69.9 LM (Bidir) 
Revised: 69.9 LM (Bidir) 5M

Original: Parallel WMT16 (5M)

Original: No 95.
Revised: No 95.

Original: 70.2 LM Forward 
Revised: 70.2 LM Forward 5M

Original: Parallel WMT16 (5M)

Original: No 93.
Revised: No 93.

Original: 70.1 Skip-thought 
Revised: 70.1 Skip-thought 5M

Original: Parallel WMT16 (5M)

Original: Yes 66.
Revised: Yes 66.

Original: 68.7 Skip-thought 
Revised: 68.7 Skip-thought 5M

Original: Parallel WMT16 (5M)

Original: No 81.
Revised: No 81.

Original: 68.7 Autoencoder Parallel WMT16 (5M)
Revised: 68.7 Autoencoder 5M

Original: Yes 84.
Revised: Yes 84.

Original: 67.9 Autoencoder 
Revised: 67.9 Autoencoder 5M

Original: Parallel WMT16 (5M)

Original: No 65.
Revised: No 65.

Original: LM (Bidir) Parallel + Monolingual (15M)
Revised: LM (Bidir) 15M

Original: No 96.
Revised: No 96.

Original: 70.6 LM Forward 
Revised: 70.6 LM Forward 15M

Original: Parallel + Monolingual (15M)

Original: No 94.
Revised: No 94.

Original: 70.6 Skip-thought 
Revised: 70.6 Skip-thought 15M

Original: Parallel + Monolingual (15M)

Original: Yes 72.
Revised: Yes 72.

Original: 68.9 Skip-thought 
Revised: 68.9 Skip-thought 15M

Original: Parallel + Monolingual (15M)

Original: No 84.
Revised: No 84.

Original: 69.1 Autoencoder Parallel + Monolingual (15M)
Revised: 69.1 Autoencoder 15M

Original: Yes 88.
Revised: Yes 88.

Original: 68.7 Autoencoder 
Revised: 68.7 Autoencoder 15M

Original: Parallel + Monolingual (15M)

Original: No 68.
Revised: No 68.

Original: LM (Bidir) Parallel + Monolingual (63M)
Revised: LM (Bidir) 63M

Original: No 96.
Revised: No 96.

Original: 70.7 LM Forward 
Revised: 70.7 LM Forward 63M

Original: Parallel + Monolingual (63M)

Original: No 94.
Revised: No 94.

Original: 70.6 Skip-thought 
Revised: 70.6 Skip-thought 63M

Original: Parallel + Monolingual (63M)

Original: Yes 85.
Revised: Yes 85.

Original: 69.4 Skip-thought 
Revised: 69.4 Skip-thought 63M

Original: Parallel + Monolingual (63M)

Original: No 88.
Revised: No 88.

Original: 69.3 Autoencoder 
Revised: 69.3 Autoencoder 63M

Original: Parallel + Monolingual (63M)

Original: Yes 82.
Revised: Yes 82.

Original: 68.4 Autoencoder 
Revised: 68.4 Autoencoder 63M

Original: Parallel + Monolingual (63M)

Original: No 67.
Revised: No 67.

Original: [Table]
Revised: [Table]

Original: B. Training Classiﬁers on 1% of Data
Revised: B. T RAINING C LASSIFIERS ON 1% OF D ATA

Original: Training task
Revised: Training task

Original: Data
Revised: Data

Original: Attention
Revised: Attn.

Original: POS L
Revised: POS L

Original: POS L POS L
Revised: POS L POS L

Original: CCG L CCG L
Revised: CCG L CCG L

Original: CCG L
Revised: CCG L

Original: Random Init
Revised: Random Init

Original: None
Revised: None

Original: N/A 68.
Revised: N/A 68.

Original: 59.3 Random Init
Revised: 59.3 Random Init

Original: None
Revised: None

Original: N/A 68.
Revised: N/A 68.

Original: Translation 
Revised: Translation 1M

Original: Parallel WMT16 (1M)

Original: Yes 90.
Revised: Yes 90.

Original: 65.4 Translation Parallel WMT16 (1M)
Revised: 65.4 Translation 1M

Original: No 82.
Revised: No 82.

Original: 65.0 LM (Bidir) Parallel WMT16 (1M)
Revised: 65.0 LM (Bidir) 1M

Original: No 93.
Revised: No 93.

Original: 67.1 LM Forward 
Revised: 67.1 LM Forward 1M

Original: Parallel WMT16 (1M)

Original: No 90.
Revised: No 90.

Original: 66.5 Skip-thought 
Revised: 66.5 Skip-thought 1M

Original: Parallel WMT16 (1M)

Original: Yes 27.
Revised: Yes 27.

Original: 60.7 Skip-thought 
Revised: 60.7 Skip-thought 1M

Original: Parallel WMT16 (1M)

Original: No 57.
Revised: No 57.

Original: 61.0 Autoencoder 
Revised: 61.0 Autoencoder 1M

Original: Parallel WMT16 (1M)

Original: Yes 71.
Revised: Yes 71.

Original: 61.9 Autoencoder 
Revised: 61.9 Autoencoder 1M

Original: Parallel WMT16 (1M)

Original: No 62.
Revised: No 62.

Original: Translation 
Revised: Translation 5M

Original: Parallel WMT16 (5M)

Original: Yes 92.
Revised: Yes 92.

Original: 65.7 Translation 
Revised: 65.7 Translation 5M

Original: Parallel WMT16 (5M)

Original: No 82.
Revised: No 82.

Original: 65.9 LM (Bidir) 
Revised: 65.9 LM (Bidir) 5M

Original: Parallel WMT16 (5M)

Original: No 93.
Revised: No 93.

Original: 67.1 LM Forward 
Revised: 67.1 LM Forward 5M

Original: Parallel WMT16 (5M)

Original: No 90.
Revised: No 90.

Original: 66.7 Skip-thought 
Revised: 66.7 Skip-thought 5M

Original: Parallel WMT16 (5M)

Original: Yes 55.
Revised: Yes 55.

Original: 63.0 Skip-thought 
Revised: 63.0 Skip-thought 5M

Original: Parallel WMT16 (5M)

Original: No 69.
Revised: No 69.

Original: 62.7 Autoencoder 
Revised: 62.7 Autoencoder 5M

Original: Parallel WMT16 (5M)

Original: Yes 67.
Revised: Yes 67.

Original: 61.0 Autoencoder 
Revised: 61.0 Autoencoder 5M

Original: Parallel WMT16 (5M)

Original: No 60.
Revised: No 60.

Original: LM (Bidir) 
Revised: LM (Bidir) 15M

Original: Parallel + Monolingual (15M)

Original: No 94.
Revised: No 94.

Original: 67.5 LM Forward 
Revised: 67.5 LM Forward 15M

Original: Parallel + Monolingual (15M)

Original: No 91.
Revised: No 91.

Original: 67.3 Skip-thought 
Revised: 67.3 Skip-thought 15M

Original: Parallel + Monolingual (15M)

Original: Yes 50.
Revised: Yes 50.

Original: 63.5 Skip-thought 
Revised: 63.5 Skip-thought 15M

Original: Parallel + Monolingual (15M)

Original: No 75.
Revised: No 75.

Original: 63.7 Autoencoder 
Revised: 63.7 Autoencoder 15M

Original: Parallel + Monolingual (15M)

Original: Yes 77.
Revised: Yes 77.

Original: 62.6 Autoencoder 
Revised: 62.6 Autoencoder 15M

Original: Parallel + Monolingual (15M)

Original: No 61.
Revised: No 61.

Original: LM (Bidir) 
Revised: LM (Bidir) 63M

Original: Parallel + Monolingual (63M)

Original: No 94.
Revised: No 94.

Original: 67.5 LM Forward 
Revised: 67.5 LM Forward 63M

Original: Parallel + Monolingual (63M)

Original: No 92.
Revised: No 92.

Original: 67.6 Skip-thought 
Revised: 67.6 Skip-thought 63M

Original: Parallel + Monolingual (63M)

Original: Yes 69.
Revised: Yes 69.

Original: 64.4 Skip-thought 
Revised: 64.4 Skip-thought 63M

Original: Parallel + Monolingual (63M)

Original: No 77.
Revised: No 77.

Original: 64.0 Autoencoder 
Revised: 64.0 Autoencoder 63M

Original: Parallel + Monolingual (63M)

Original: Yes 72.
Revised: Yes 72.

Original: 61.3 Autoencoder 
Revised: 61.3 Autoencoder 63M

Original: Parallel + Monolingual (63M)

Original: No 60.
Revised: No 60.

Original: [Table]
Revised: [Table]