Using Deep Neural Networks to Learn Syntactic Agreement (2017)

Abstract: We consider the extent to which different deep neural network (DNN) configurations can learn syntactic relations, by taking up Linzen et al.’s (2016) work on subject-verb agreement with LSTM RNNs. We test their methods on a much larger corpus than they used (a ⇠24 million example part of the WaCky corpus, instead of their ⇠1.35 million example corpus, both drawn from Wikipedia). We experiment with several different DNN architectures (LSTM RNNs, GRUs, and CNNs), and alternative parameter settings for these systems (vocabulary size, training to test ratio, number of layers, memory size, drop out rate, and lexical embedding dimension size). We also try out our own unsupervised DNN language model. Our results are broadly compatible with those that Linzen et al. report. However, we discovered some interesting, and in some cases, surprising features of DNNs and language models in their performance of the agreement learning task. In particular, we found that DNNs require large vocabularies to form substantive lexical embeddings in order to learn structural patterns. This finding has interesting consequences for our understanding of the way in which DNNs represent syntactic information. It suggests that DNNs learn syntactic patterns more efficiently through rich lexical embeddings, with semantic as well as syntactic cues, than from training on lexically impoverished strings that highlight structural patterns.

## introduction the extent to which machine learning systems in general, and neural networks in particular, can learn regularities that depend on syntactic structure is a topic of ongoing debate. @xcite subject-verb number agreement is a paradigmatic case of a syntactic feature that depends on structural properties of a sentence. linzen et al. (2016) train a long short term memory recurrent neural network (lstm rnn) on a subset of a wikipedia corpus to predict the number of a verb. as they observe, the task increases in difficulty in relation to the length of the sequence of nps with the wrong number feature that occur between a subject and the verb that it controls. they refer to such intervening nps as attractors. 2 in (1) the attractors for the the students-submit pair are indicated in boldface. linzen et. al use a dependency parser to identify the controlling subject of each verb in their corpus of examples. this identification is necessary to compute the number of attractors, but it is not used in the training for the number prediction task, because the number of the verb is morphologically manifest in the raw data. they train their lstm rnn on ⇠121,500 examples (9% of the total corpus) by supervised learning, in which the system is shown the correct number feature of the verb. they test the number predictions of their rnn on ⇠1.21 million examples (90% of their corpus). they encode the input words as vectors in 50 dimensions, and their rnn has 50 hidden units. they report that their system achieves 99% accuracy in the number prediction task for cases with 0 attractors between the subject and its verb, and declines to 83% accuracy for examples with 4 attractors. they do not report scores for examples with more than 4 attractors. in addition to the supervised number prediction task, they train a generative language model (predicting the next word) and use it to predict the number of the verb in an unsupervised manner. by contrast to their supervised lstm rnn, linzen et al.'s language model goes below chance in its predictions for 4 attractor cases. while the much larger google lm (józefowicz et al., 2016) does better, at a ⇠45% error for 4 attractors, it also performs well below their supervised rnn. they conclude that a considerable amount of syntactic structure is accessible to dnn learning if training is properly supervised. our main objective in the work that we report here is to explore the capabilities, and the limitations of dnns for learning complex syntactic relations which depend on structural properties of sentences. we used methods similar to linzen et al.'s to test several dnn models on a much larger corpus. we experimented with different dnn architectures, and with alternative values for the following parameters: our results are broadly compatible with those that linzen et al. present. however, some of the correlations that we observed between certain parameter settings and levels of performance surprised us. specifically, by working with different vocabulary sizes, lexical encodings, and embedding sizes we discovered that our supervised dnn models learn agreement patterns more effectively from rich word embeddings than from abstract syntactically annotated input. we also found that, in general, our models required larger amounts of training relative to testing than linzen et al. describe for their system, in order to reach the performance that they report. @xcite we have observed that increasing the values of hyperparameters (and thereby the number of degrees of freedom in the network itself) generally improves accuracy, even if after a certain point, overfitting is observed. but changes to any individual hyperparameter do not create dramatic effects. however, the total effect of hypeparameter optimisation is quite significant. finally, we were able to construct a language model with significantly better (unsupervised) prediction. yet, our model is much smaller than the google lm. all of our supervised dnns outperformed our language model on the number prediction task. @xcite 1 experimental design 1.1 corpus for our experiments, we used the wackypedia english corpus (baroni et al., 2009), which contains ⇠24 million example cases of present tense subjectverb agreement. @xcite the corpus is annotated for pos by treetagger (schmid, 1995), and for dependency relations by the maltparser (nivre et al., 2007). linzen et al. restricted training and testing to one agreement case per sentence in their corpus. we used the full set of number agreement relations in the sentences of our corpus for our experimental work. they also limit their test, but not their training examples to cases in which all nps intervening between the subject and the verb that it controls are attractors. we did not adopt this constraint on our test sets of examples. we included the cases in which agreeing, as well as non-agreeing nps intervened between the subject and its verb. we are interested in measuring the accuracy with which a dnn predicts verb number in complex and possibly confusing syntactic sequences. training and testing less filtered data of the kind that involves discarding the non-agreement constraint that linzen we implemented our own versions of an lstm rnn, gru rnn, and cnn, using the keras library (chollet, 2015) with a tensorflow backend. our lstm rnn and gru rnn have standard architectures. we apply a dense layer with sigmoid activation to the output of the latest rnn cell to obtain the verb-number classifier. our cnn has 6 levels with filtering successively compressing vector dimensions from 15 through 20, 15, 10 to 5. convolution at these levels yields 7, 5, 5, and 3 features, respectively. every convolution layer uses a relu activation function. the last layer is a dense layer with sigmoid activation. we represent this cnn graphically in fig. 1 . ## results we present our results in figures 2 to 6. in all the figures, except for fig. 6 , the y-axis gives the accuracy rate, and the x-axis the number of np attractors. we found that 73% of the verbs in our test sets are singular. this provides a majority choice baseline, which is indicated by a straight horizontal line in these figures. figures 2a and 2b shows that using a reduced vocabulary of 100 most common words, substituting pos tags for the remainder, consistently reduces accuracy across dnn architectures, for the supervised learning task. increasing the vocabulary to 100k words generally yields a significant improvement in performance for the lstm rnn, but gives mixed results for our cnn. fig. 3a indicates that increasing the ratio of training to testing examples from 10% to 50% significantly improves the performance of the lstm rnn (with 150 units and a vocabulary of 10,000 word embeddings). further increasing it to 90% does not make much of a difference, even degrading accuracy slightly at 6 attractors. fig. 3b reveals that the lstm and gru rnns perform at a comparable level, and both achieve significantly better accuracy than our cnn. fig. 3c suggests that increasing the number of units in an lstm rnn from 50 to 150 improves its performance on the task, while further expanding this number to 450 yields greater accuracy in relation to the number to attractors. each three-fold increase in the number of units achieves a similar improvement in percentage points for a higher number of attractors. a further increase to 1350 provides only a small overall improvement. fig. 3d indicates that increasing the number of layers for an lstm rnn from 1 150-unit layer to 2 such layers marginally improves its performance. a further increase to 4 150-unit layers makes no clear difference. fig. 3e shows that by introducing a dropout rate of 0.1 for the lstm rnn (1 layer with 150 units) we improve its performance slightly. an increase to 0.2 provides no clear benefit. furthuer increasing the dropout rate to 0.5 degrades performance. fig. 3f indicates that decreasing lexical embedding dimensions from our benchmark 50 to 17 decreases the performance of our lstm rnn. increasing dimension size to 150 improves performance, while further expanding it to 450 contributes no benefit, and, for some cases, reduces accuracy. using the summing method (inequality (1.1)) yields results comparable to linzen et al.'s lm, the verb targeted method (inequality (1.2)) achieves a far higher level of accuracy than their model, and than the results that they give for the much larger google lm (józefowicz et al., 2016). we note that linzen et al. report using the verb targeted method for their results. however the accuracy of our lm with the verb targeted method is still below our best supervised results for the lstm rnn. its performance with the verb targeted method is roughly comparable to that of our cnn. fig. 4 shows the performance of a dnn configured with the best observed parameter values, which are @xmath0, @xmath1, number of @xmath2, dropout @xmath3.1, vocabulary @xmath4, @xmath5, of the corpus, and lexical embedding @xmath6. this dnn provides the highest level of accuracy of all the systems that we tested. finally fig. 6 displays the inverse relation between the number of examples and the number of attractor nps. ## discussion our results support linzen et al.'s finding that rnns, both of the lstm and gru variety, are well suited to the task of identifying long distance syntactic dependencies, even when an extended, complex sequence of expressions that could cause confusion intervenes between a controller of a syntactic feature, and the lexical item on which it is realised. however, their success in learning subject-verb agreement scales with the size of the data set on which they train. given that rnns rely on sequential composition of layers of neural units, it is not clear how much hierarchical structure they can identify in their recog0 1 2 3 4 5 6 7 0 0 1 2 3 4 5 6 7 0.6 0.7 0.8 0.9 1 benchmark best figure 4: results for a configuration with best parameters values: lstm rnn with 2 layers of 1350 units, dropout rate 0.1, vocabulary size 100k, training on 90%, and lexical embedding dimension size 150 0 1 2 3 4 5 6 7 0.4 0.5 0.6 0.7 0.8 0.9 1 supervised summing method verb targeted method nition of these dependencies. it is also worth noting that we used fairly simple rnns. it could be that an rnn with a more structured memory that incorporates the equivalent of a stack for encoding the beginning of a dependency and a pop mechanism for releasing it later in a sequence (grefenstette et al., 2015) would yield even better results. cnns have been highly successful in image recognition, which involves constructing levels of successfully more abstract visual feature patterns. therefore it is surprising that our cnn did not perform particularly well on a task that would, at first glance, appear to require learning hierarchical phrase structure. it is important to recall that we employed a fairly simple, static cnn model. one with a dynamic memory might yield better performance for this task. in general, there is considerable room for exploring alternative architectures before drawing strong conclusions on the capabilities of the entire class of dnns for learning syntactic relations. one of the most striking results to emerge from our experiments is that training dnns on data that is lexically impoverished, but highlights the syntactic elements between which a relation is to be acquired does not, at least in the current case, facilitate learning. on the contrary, it degrades it. dnns learn better from data populated by richer lexical sequences. this suggests that dnns are not efficient at picking up abstract syntactic patterns when they are explicitly marked in the data. instead they extract them incrementally from lexical embeddings through recognition of their distributional regularities. it is also possible that they use the lexical semantic cues that larger vocabularies introduce to determine agreement preferences for a verb. it is an open question as to how dnn modes of learning resemble and diverge from human learning. we are making no cognitive claims here. however, it is interesting to note that some recent work in neurolinguistics indicates that syntactic knowledge is distributed through different language centres in the brain, and closely integrated with lexical-semantic representations (blank et al., 2016). this lexically encoded and distributed way of representing syntactic information is consistent with the role of rich lexical embeddings in dnn syntactic learning. finally our results show that a language model can achieve not entirely unreasonable results on the number agreement prediction task, if an appropriate method is applied for comparing the conditional probabilities of alternative number markings on verbs. our lm is trained on only 100 words, with pos tags substituted for the others in order to highlight the agreement dependency relations that we are seeking to model. this strategy did not improve performance for supervised learning, but it does appear to have been successful for unsupervised learning with a language model. ## conclusions and future work our experimental results strengthen linzen et al.'s conclusion that dnns are able to learn long distance syntactic relations to a fairly high degree of accuracy, across extended complex sequences of potentially distracting phrases. we also found that accuracy in the supervised version of this task scales with the amount of training data used, and with the size of the lexical embedding vocabulary. performance improves with an increase in the number of hidden units. this effect may be even more pronounced when tracking more complex syntactic relations with multiple features. this is a question that we will explore in future work. we also found that it is possible to obtain reasonable results with unsupervised learning through a comparatively small language model, when we use a targeted procedure for predicting verb number. the performance of this model, with this procedure, significantly exceeds that of the two models that linzen et al. present. in furture work we will experiment with alternative dnn architectures. we are particularly interested in incorporating a structured memory that simulates a stack and pop mechanism for handling long distance dependencies. we will also test larger language models trained on bigger vocabularies, using our targeted prediction procedure, to see if our conjecture that a smaller vocabulary produces better probabilistic predictions. this proposal did not hold for the supervised learning case, but we do not know if it will be sustained for unsupervised learning with a language model. one of our main concerns will be to explore syntactic dependencies involv-ing several agreement features. in languages in which gender, and person, as well as number are morphologically realised on verbs the agreement prediction task is more difficult. it requires accuracy across three feature dimensions rather than one. testing dnns on agreement in such languages will provide a better sense of their capacity to learn and represent syntactic information.