L ANGUAGE M ODELING T EACHES Y OU M ORE S YNTAX THAN T RANSLATION D OES
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work using auxiliary prediction task classiﬁers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al., 2017), are so beneﬁcial for neural language understanding models.
We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn.
With this in mind, we compare four objectives—language modeling, translation, skip-thought, and autoencoding—on their ability to induce syntactic and part-of-speech information.
We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture.
We ﬁnd that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data.
These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information.
We also ﬁnd that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.
I NTRODUCTION
Representation learning with deep recurrent neural networks has revolutionized natural language processing and replaced many of the expert-designed, linguistic features used previously.
Recently, researchers have begun to investigate the properties of learned representations by training auxiliary classiﬁers that use the hidden states of frozen, pretrained models to perform other tasks.
These investigations have shown that when deep LSTMs (Hochreiter & Schmidhuber, 1997) are trained on tasks like translation, they learn substantial syntactic and semantic information about their input sentences, including part-of-speech (Shi et al., 2016; Belinkov et al., 2017a;b; Blevins et al., 2018).
These intriguing ﬁndings lead us to ask the following questions:
How does the training task affect how well models learn syntactic properties?
Which tasks are better at inducing these properties?
How does the amount of data the model is trained on affect these results?
When does training on more data help?
We investigate these questions by holding the data source and model architecture constant, while varying both the training task and the amount of training data.
Speciﬁcally, we examine models trained on English-German (En-De) translation, language modeling, skip-thought (Kiros et al., 2015), and autoencoding, and also compare to an untrained LSTM model as a baseline.
We control for the data domain by exclusively training on datasets from the 2016 Conference on Machine
Translation (WMT; Bojar et al., 2016).
We train models on all tasks using the parallel En-De corpus, which allows us to make fair comparisons across tasks.
We also train models on a subset of the this corpus to examine the effect of training data volume on learned representations.
Additionally, we augment the parallel dataset with a large monolingual corpus from WMT to examine how the performance of the unsupervised tasks (all but translation) scale with more data.
Throughout our work, we focus on the syntactic evaluation tasks of part-of-speech (POS) tagging and Combinatorial Categorical Grammar (CCG) supertagging.
Supertagging is considered a build- ing block for parsing as these tags constrain the ways in which words can compose, largely determining the parse of the sentence.
CCG supertagging thus allows us to measure the degree to which models learn syntactic structure above the word.
We focus our analysis on representations learned by language models and by the encoders of sequence-to-sequence models, as translation encoders have been found to learn richer representations of POS and morphological information than translation decoders (Belinkov et al., 2017a).
We ﬁnd that for POS and CCG tagging, bidirectional language models (BiLMs)—created by separately training forward and backward language models, and concatenating their hidden states— outperform models trained on all other tasks.
Even BiLMs trained on relatively small amounts of data (1 million sentences) outperform translation and skip-thought models trained on larger datasets (5 million and 63 million sentences respectively).
Our inclusion of an untrained LSTM baseline allows us to study the effect of training on hidden state representations of LSTMs.
We ﬁnd, surprisingly, that when we use all of the available labeled tag data to train our auxiliary task classiﬁers, our best trained models (BiLMs) only outperform the randomly initialized, untrained LSTMs by a few percentage points.
When we reduce the amount of classiﬁer training data though, the performance of the randomly initialized LSTM model drops far below those of trained models.
We hypothesize that this occurs because training the classiﬁers on large amounts of auxiliary task data allows them to memorize conﬁgurations of words seen in the training set and their associated tags.
We test this hypothesis by training classiﬁers to predict the identity of neighboring words from a given hidden state, and ﬁnd that randomly initialized models outperform all trained models on this task.
Our ﬁndings demonstrate that our best trained models do well on the tagging tasks because they are truly learning representations that conform to our notions of POS and CCG tagging, and not simply because the classiﬁers we train are able to recover neighboring word identity information.
R ELATED W ORK
Evaluating Learned Representations Adi et al.
(2016) introduce the idea of examining sentence vector representations by training auxiliary classiﬁers to take sentence encodings and predict attributes like word order.
Belinkov et al.
(2017a) build on this work by examining the hidden states of LSTMs trained on translation and ﬁnd that they learn substantial POS and morphological information without direct supervision for these linguistic properties.
Beyond translation, Blevins et al.
ﬁnd that deep LSTMs learn hierarchical syntax when trained on a variety of tasks—including semantic role labeling, language modeling, and dependency parsing.
However, the models examined by Blevins et al.
(2018) were also trained on different datasets, so it’s unclear if the differences in syntactic task performance are due to the training objectives or simply differences in the training data.
By controlling for model size and the quantity and genre of the training data, we we are able to make direct comparisons between tasks on their ability to induce syntactic information.
Transfer Learning of Representations Much of the work on sentence-level pretraining has focused on sentence-to-vector models and evaluating learned representations on how well they can be used to perform sentence-level classiﬁcation tasks.
A prominent early success in this area with unlabeled data is skip-thought (Kiros et al., 2015), the technique of training a sequence-to-sequence model to predict the sentence preceding and following each sentence in a running text.
InferSent (Conneau et al., 2017)—the technique of pretraining encoders on natural language inference data— yields strikingly better performance when such labeled data is available.
Work in transfer learning of representations has recently moved beyond strict sentence-to-vector mappings.
Newer models that incorporate LSTMs or Transformer networks pretrained on datarich tasks, like translation and language modeling, have achieved state-of-the-art results on many tasks—including semantic role labeling, natural language inference, and coreference resolution (Peters et al., 2018;
McCann et al., 2017; Howard & Ruder, 2018; Radford et al., 2018).
Although comparisons have previously been made between translation and language modeling as pretraining tasks (Peters et al., 2018; Wang et al., 2018), we investigate this issue more thoroughly by controlling for the quantity and content of the training data.
[Table]
Training
Dataset Size
The performance of neural models depends immensely on the amount of training data used.
Koehn & Knowles (2017) ﬁnd that when training machine translation models on corpora with fewer than 15 million words (English side), statistical machine translation approaches outperform neural ones.
Similarly, Hestness et al.
(2017) study the affect of training data volume on performance for several tasks—including translation and image classiﬁcation.
They ﬁnd that for small amounts of data, neural models perform about as well as chance.
After a certain threshold, model performance improves logarithmically with the amount of training data, but this eventually plateaus.
With this in mind, we also vary the amount of training data to investigate the relationship between performance and data volume for each task.
Randomly Initialized Models Conneau et al.
(2018) use randomly initialized LSTMs as a baseline when studying sentence-to-vector embedding models.
They ﬁnd that untrained models outperform many trained models on several auxiliary tasks, including predicting word content.
Similarly in vision, untrained convolutional networks have been shown to capture many low-level image statistics and can be used for image denoising (Ulyanov et al., 2017).
Our method of training auxiliary classiﬁers on randomly initialized RNNs builds on the tradition of reservoir computing, in which randomly initialized networks or “reservoirs” are ﬁxed and only “read-out” classiﬁer networks are trained (Luko ˇ sevi ˇ cius & Jaeger, 2009).
Echo state networks—reservoir computing with recurrent models—have been used for tasks like speech recognition, language modeling, and time series prediction (Verstraeten et al., 2006; Tong et al., 2007; Sun et al., 2017).
M ETHODS 3. M AIN T RAINING D ATA
We use the parallel English-German (En-De) dataset from the 2016 ACL Conference on Machine
Translation (WMT) shared task on news translation (Bojar et al., 2016).
This dataset containsmillion ordered sentence translation pairs.
We also use the 2015 English monolingual news discussion dataset from the same WMT shared task, which contains approximately 58 million ordered sentences.
To examine how the volume of training data affects learned representations, we use four corpus sizes: 1, 5, 15, and 63 million sentences (translation is only trained on the smaller two sizes).
We create the 1 million sentence corpora from the 5 million sentence dataset by sampling (i) sentence pairs for translation, (ii) English sentences for autoencoders, and (iii) ordered English sentence pairs for skip-thought and language models 1 .
Similarly, we create the 15 million sentence corpora for the unsupervised tasks by sampling sentences from the entire corpus of 63 million sentences.
We use word-level representations throughout and use the Moses package (Koehn et al., 2007) to tokenize and truecase our data.
Finally, we limit both the English and German vocabularies to the 50k most frequent tokens in the training set.
[Figure]
3. M ODEL A RCHITECTURE AND T RAINING
We train all our models using OpenNMT-py (Klein et al., 2017) and use the default options for model sizes, hyperparameters, and training procedure—except we increase the size of the LSTMs, make the encoders bidirectional, and use validation-based learning rate decay instead of a ﬁxed schedule.
Speciﬁcally, all our models (except language models) are 1000D, two-layer encoder-decoder LSTMs with bidirectional encoders (500D per direction) and 500D embeddings.
We train models both with and without attention (Bahdanau et al., 2015).
For language models, we train a 1000D forward language model and a bidirectional language model—two 500D language models (forward and backward) trained separately, whose hidden states are concatenated.
All models, including our untrained baseline, are initialized from a uniform distribution ( − 0 . 1 , 0 . 1 ), the default in OpenNMT.
We use the same training procedure for all our models.
We evaluate on the validation set every epoch when training on the 1 and 5 million sentence datasets, and evaluate approximately everymillion sentences when training on the larger datasets.
We use SGD with an initial learning rate of 1.
Whenever a model’s validation loss increases relative to the previous evaluation, we halve the learning rate and stop training when the learning rate reaches 0 .
For each training task and dataset size, we select the model with the lowest validation perplexity to perform auxiliary task evaluations on.
We report model performance in terms of perplexity and BLEU (Papineni et al., 2002) in Table 1.
For translation we use beam search ( B = 5 ) when decoding.
3. C LASSIFIER D ATA AND A RCHITECTURE
POS and CCG For Part-of-Speech (POS) tagging evaluation, we use the Wall Street Journal (WSJ) portion of the Penn Treebank (PTB; Marcus et al., 1993)
We follow the standard WSJ split (train 2-21; dev 22; test 23).
The dataset contains approximately 50k sentences and 45 tag types.
For CCG supertagging, we use CCG Bank (Hockenmaier & Steedman, 2007), which is based on
PTB WSJ.
CCG supertagging provides ﬁne-grained information about the role of each word in its larger syntactic context and is considered almost parsing, since sequences of tags map sentences to small subsets of possible parses.
The entire dataset contains approximately 50k sentences andtag types.
We display POS and CCG tags for an example sentence in Figure 1.
To study the impact of auxiliary task training data volume, for both datasets we create smaller classiﬁer training sets by sampling 10% and 1% of the sentences.
We truecase both datasets using the same truecase model trained on WMT and restrict the vocabularies to the 50k tokens used in pretraining our LSTM models.
In addition to the untrained LSTM baseline, we also compare to the word-conditional most frequent class (WC-MFC)—the most frequently assigned tag class for each distinct word in the training set.
For this baseline we restrict the vocabulary to that of our LSTM models and map all out-of-vocabulary words to a single UNK token.
Note that while PTB and WMT are both drawn from news text, there is slight genre mismatch.
Word Identity For this task, the classiﬁer takes a single LSTM hidden state as input and predicts the identity of the word at a different time step.
For example, for the sentence “I love NLP” and a time step shift of -2, we would train the classiﬁer to take the hidden state for “NLP” and predict the word “I”.
We use the WSJ dataset for this task.
Following Conneau et al.
(2018), we take all words that occur between 100 and 1000 times (about 1000 words total) as the possible targets for neighboring word prediction.
Classiﬁer Training Procedure We train multi-layer perceptron (MLP) classiﬁers that take an
LSTM hidden state (from one time step and one layer) and output a distribution over the possible labels (tags or word identities).
The MLPs we train have a single 1000D hidden layer with a
ReLU activation.
For classiﬁer training, we use the same training and learning rate decay procedure used for pretraining the LSTM encoders.
C OMPARING P RETRAINING T ASKS
In this section we discuss the main POS and CCG tagging results displayed in Figure 2.
Overall,
POS and CCG tagging accuracies tend to increase with the amount of data the LSTM encoders are trained on, but the marginal improvement decreases as the amount of training data increases.
Language Modeling and Translation For all pretraining dataset sizes, bidirectional language model (BiLM) and translation encoder representations perform best on both POS and CCG tagging.
Translation encoders, however, slightly underperform BiLMs, even when both models are trained on the same amount of data.
In fact, even BiLMs trained on the smallest amount of data (1 million sentences) outperform models trained on all other tasks and dataset sizes (up to 5 million sentences for translation, and 63 million sentences for skip-thought and autoencoding). Especially since BiLMs
[Figure] do not require aligned data to train, the superior performance of BiLM representations on these tasks suggests that BiLMs (like ELMo; Peters et al., 2018) are better than translation encoders (like CoVe;
McCann et al., 2017) for transfer learning of syntactic information.
Note that in contrast to the encoders for other tasks, LM encoders have a per-token loss.
Since our evaluation tasks also predict a single label for each token, this could be one reason that BiLMs perform so well.
For all amounts of training data, the BiLMs signiﬁcantly outperform the 1000D forward-only language models.
The gap in performance between bidirectional and forward language models is greater for CCG supertagging than for POS tagging.
When using all available auxiliary training data, there is a 2 and 8 percentage point performance gap in POS and CCG tagging respectively.
This difference in relative performance suggests that bidirectional context information is more important for identifying syntactic structure than for identifying part of speech.
Figure 2 illustrates how the best performing BiLMs and translation models tend to be more robust to decreases in classiﬁer data than models trained on other tasks.
Also, when training on less auxiliary task data, POS tagging performance tends to drop less than CCG supertagging performance.
For the best model (BiLM trained on 63 million sentences), when using 1% rather than all of the auxiliary task training data, CCG accuracy drops 9 percentage points, while POS accuracy only drops 2 points.
Further examinations of the effect of classiﬁer data volume are displayed in Figure 3.
Skip-Thought Although skip-thought encoders consistently underperform both BiLMs and translation encoders in all data regimes we examine, skip-thought models improve the most when increasing the amount of pretraining data, and are the only models whose performance does not seem to have plateaued by 63 million training sentences.
Since we train our language models on ordered sentences, as we do for skip-thought, our language models can be interpreted as a regularized versions of skip-thought, in which the weights of the encoder and decoder are shared.
The increased model capacity of skip-thought, compared to language models, could explain the difference in learned representation quality—especially when these models are trained on smaller amounts of data.
Random Initialization For our randomly initialized, untrained LSTM encoders, we use the default weight initialization technique in OpenNMT-py, a uniform distribution between -0.1 and 0.1; the only change we make is to set all biases to zero.
We ﬁnd that this baseline performs quite well when using all auxiliary data, and is only 3 and 8 percentage points behind the BiLM on POS and
CCG tagging, respectively.
We ﬁnd that decreasing the amount of classiﬁer data leads to a significantly greater drop in the untrained encoder performance compared to trained models.
In the 1% classiﬁer data regime, the performance of untrained encoders on both tasks drops below that of all trained models and below even the word-conditional most-frequent class baseline.
[Figure]
We hypothesize that the randomly initialized baseline is able to perform well on tagging tasks with large amounts of auxiliary task training data, because the classiﬁer can learn the identity of neighboring words from a given time step’s hidden state, and simply memorize word conﬁgurations and their associated tags from the training data.
We test this hypothesis directly in Section 6 and ﬁnd that untrained LSTM representations are in fact better at capturing neighboring word identity information than any trained model.
Autoencoder Models trained on autoencoding are the only ones that do not consistently improve with the amount of training data, which is unsurprising as unregularized autoencoders are prone to learning identity mappings (Vincent et al., 2008).
When training on 10% and 1% of the auxiliary task data, autoencoders outperform randomly initialized encoders and match the word-conditional most frequent class baseline.
When training on all the auxiliary data though, untrained encoders outperform autoencoders.
These results suggest that autoencoders learn some useful structure that is useful in the low auxiliary data regime.
However, the representations autoencoders learn do not capture syntactically rich features, since random encoders outperform them in the high auxiliary data regime.
This conclusion is further supported by the extremely poor performance of the second layer of an autoencoder without attention on POS tagging (almost 10 percentage points below the most frequent class baseline), as seen in Figure 4a.
C OMPARING L AYERS
Embeddings (Layer 0)
We ﬁnd that randomly initialized embeddings consistently perform as well as the word-conditional most frequent class baseline on POS and CCG tagging, which serves as an upper bound on performance for the embedding layer.
As these embeddings are untrained, the auxiliary classiﬁers are learning to memorize and classify the random vectors.
When using all the auxiliary classiﬁer data, there is no signiﬁcant difference in the performance of trained and untrained embeddings on the tagging tasks.
Only for smaller amounts of classiﬁer data do trained embeddings consistently outperform randomly initialized ones.
Upper Layers Belinkov et al. (2017a)
ﬁnd that, for translation models, the ﬁrst layer consistently outperforms the second on POS tagging.
We ﬁnd that this pattern holds for all our models, except in BiLMs, for which the ﬁrst and second layers perform equivalently.
The pattern holds even for untrained models, suggesting that POS information is stored on the lower layer, not necessarily because the training task encourages this, but because of properties of the deep LSTM architecture.
We also ﬁnd that for CCG supertagging, the ﬁrst layer also outperforms the second layer on untrained models.
For the trained models though, the second layer performs better than the ﬁrst in some cases.
Which layer performs best appears to be independent of absolute performance on the supertagging task.
Our layer analysis results are displayed in Figure 4.
W ORD I DENTITY P REDICTION
Our results on word identity prediction are summarized in Figure 5 and given in more detail in
Appendix A. While trained encoders outperform untrained ones on both POS and CCG tagging, we ﬁnd that all trained LSTMs underperform untrained ones on word identity prediction.
This ﬁnding conﬁrms that trained encoders genuinely capture substantial syntactic features, beyond mere word identity, that the auxiliary classiﬁers can use.
We ﬁnd that for both trained and untrained models, the ﬁrst layer outperforms the second layer when predicting the identity of the immediate neighbors of a word.
However, the second layer tends to outperform the ﬁrst at predicting the identity of more distant neighboring words.
This effect is especially apparent for the randomly initialized encoders.
Our ﬁnding suggests that, as is the case for convolutional neural networks, depth in recurrent neural networks has the effect of increasing the receptive ﬁeld and allows the upper layers to have representations that capture a larger context.
These results reﬂect the ﬁndings of Blevins et al.
(2018) that for trained models, upper levels of LSTMs encode more abstract syntactic information, since more abstract information generally requires larger context information.
C ONCLUSION
By controlling for the genre and quantity of the training data, we make fair comparisons between several data-rich training tasks in their ability to induce syntactic information.
We ﬁnd that bidirectional language models (BiLMs) do better than translation and skip-thought encoders at extracting useful features for POS tagging and CCG supertagging.
Moreover, this improvement holds even when the BiLMs are trained on substantially less data than competing models.
Our results suggest that for transfer learning, BiLMs like ELMo (Peters et al., 2018) capture more useful features than translation encoders—and that this holds even on genres for which data is not abundant.
We also ﬁnd that randomly initialized encoders extract usable features for POS and CCG tagging— at least when the auxiliary POS and CCG classiﬁers are themselves trained on reasonably large amounts of data.
The performance of untrained models drops sharply relative to trained ones when using smaller amounts of the classiﬁer data.
We investigate further and ﬁnd that untrained models outperform trained ones on the task of neighboring word identity prediction, which conﬁrms that trained encoders do not perform well on tagging tasks because the classiﬁers are simply memorizing word identity information.
We also ﬁnd that both trained and untrained LSTMs store more local neighboring word identity information in lower layers and more distant word identity information in upper layers, which suggests that depth in LSTMs allow them to capture larger context information.
A R ANDOMLY I NITIALIZED E NCODERS
B POS AND CCG E VALUATION F ULL R ESULTS B. T RAINING C LASSIFIERS ON A LL D ATA
[Table]
5M
No 94.
71.5 Skip-thought 5M
Yes 76.
71.1 Skip-thought 5M
No 86.
71.2 Autoencoder 5M
Yes 88.
70.8 Autoencoder 5M
No 70.
LM (Bidir) 15M
No 97.
72.0 LM (Forward) 15M
No 95.
72.0 Skip-thought 15M
Yes 82.
71.6 Skip-thought 15M
No 90.
71.5 Autoencoder 15M
Yes 91.
71.4 Autoencoder 15M
No 71.
LM (Bidir) 63M
No 96.
72.0 LM (Forward)
M
No 95.
72.0 Skip-thought 63M
Yes 90.
71.6 Skip-thought 63M
No 91.
71.6 Autoencoder 63M
Yes 89.
[Table] [Table]
Training
[Table]
Data
Attention
POS L
POS L POS L
CCG L CCG L
CCG L
Random Init
None
N/A 85.
[Table]
[Table] [Table]
68.3 Random Init
None
N/A 84.
[Table]
[Table] [Table]
[Table] [Table]
Translation 1M
Yes 93.
[Table] [Table]
[Table] [Table]
Translation 1M
No
[Table] [Table]
[Table] [Table]
LM (Bidir) 1M
No
[Table] [Table]
[Table]
LM Forward 1M
No
[Table] [Table]
Skip-thought 1M
Yes 34.
[Table]
Skip-thought 1M
No
[Table]
Yes
[Table] [Table]
Autoencoder 1M
No 71.
[Table] [Table]
[Table] [Table]
[Table]
Translation 5M
Yes
[Table] [Table]
69.8 Translation 5M
No 89.
[Table] [Table]
[Table]
LM (Bidir) 5M
No
[Table] [Table]
[Table] [Table]
[Table]
LM Forward 5M
No 93.
[Table] [Table]
[Table] [Table]
70.1 Skip-thought 5M
Yes 66.
68.7 Skip-thought 5M
No 81.
68.7 Autoencoder 5M
Yes 84.
67.9 Autoencoder 5M
No 65.
LM (Bidir) 15M
No 96.
70.6 LM Forward 15M
No 94.
70.6 Skip-thought 15M
Yes 72.
68.9 Skip-thought 15M
No 84.
69.1 Autoencoder 15M
Yes 88.
68.7 Autoencoder 15M
No 68.
LM (Bidir) 63M
No 96.
70.7 LM Forward 63M
No 94.
70.6 Skip-thought 63M
Yes 85.
69.4 Skip-thought 63M
No 88.
69.3 Autoencoder 63M
Yes 82.
68.4 Autoencoder 63M
No
[Table]
B
[Table]
T
[Table]
Training task
Data
Attn.
POS L
POS L POS L
CCG L CCG L
CCG L
Random Init
None
N/A 68.
59.3 Random Init
None
N/A 68.
Translation 1M
Yes 90.
65.4 Translation 1M
No 82.
65.0 LM (Bidir) 1M
No 93.
67.1 LM Forward 1M
No 90.
66.5 Skip-thought 1M
Yes 27.
60.7 Skip-thought 1M
No 57.
61.0 Autoencoder 1M
Yes 71.
61.9 Autoencoder 1M
No 62.
Translation 5M
Yes 92.
65.7 Translation 5M
No 82.
65.9 LM (Bidir) 5M
No 93.
67.1 LM Forward 5M
No 90.
66.7 Skip-thought 5M
Yes 55.
63.0 Skip-thought 5M
No 69.
62.7 Autoencoder 5M
Yes 67.
61.0 Autoencoder 5M
No 60.
LM (Bidir) 15M
No 94.
67.5 LM Forward 15M
No 91.
67.3 Skip-thought 15M
Yes 50.
63.5 Skip-thought 15M
No 75.
63.7 Autoencoder 15M
Yes 77.
62.6 Autoencoder 15M
No 61.
LM (Bidir) 63M
No 94.
67.5 LM Forward 63M
No 92.
67.6 Skip-thought 63M
Yes 69.
64.4 Skip-thought 63M
No 77.
64.0 Autoencoder 63M
Yes 72.
61.3 Autoencoder 63M
[Table]