An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models (2020)

Abstract: We explore the utilities of explicit negative examples in training neural language models. Negative examples here are incorrect words in a sentence, such as barks in * The dogs barks . Neural language models are commonly trained only on positive examples, a set of sentences in the training data, but recent studies suggest that the models trained in this way are not capable of robustly handling complex syntactic constructions, such as long-distance agreement. In this paper, we first demonstrate that appropriately using negative examples about particular constructions (e.g., subject-verb agreement) will boost the model’s robustness on them in English, with a negligible loss of perplexity. The key to our success is an additional margin loss between the log-likelihoods of a correct word and an incorrect word. We then provide a detailed analysis of the trained models. One of our findings is the difficulty of object-relative clauses for RNNs. We find that even with our direct learning signals the models still suffer from resolving agreement across an object-relative clause. Augmentation of training sentences involving the constructions somewhat helps, but the accuracy still does not reach the level of subject-relative clauses. Although not directly cognitively appealing, our method can be a tool to analyze the true architectural limitation of neural models on challenging linguistic constructions.

## introduction despite not being exposed to explicit syntactic supervision, neural language models (lms), such as recurrent neural networks, are able to generate fluent and natural sentences, suggesting that they induce syntactic knowledge about the language to some extent. however, it is still under debate whether such induced knowledge about grammar is robust enough to deal with syntactically challenging constructions such as long-distance subjectverb agreement. so far, the results for rnn language models (rnn-lms) trained only with raw text are overall negative; prior work has reported low performance on the challenging test cases @xcite even with the massive size of the data and model @xcite , or argue the necessity of an architectural change to track the syntactic structure explicitly @xcite . here the task is to evaluate whether a model assigns a higher likelihood on a grammatically correct sentence (1a) over an incorrect sentence (1b) that is minimally different from the original one @xcite . the author that the guards like laughs. b. * the author that the guards like laugh. in this paper, to obtain a new insight into the syntactic abilities of neural lms, in particular rnn-lms, we perform a series of experiments under a different condition from the prior work. specifically, we extensively analyze the performance of the models that are exposed to explicit negative examples. in this work, negative examples are the sentences or tokens that are grammatically incorrect, such as (1b) above. since these negative examples provide a direct learning signal on the task at test time it may not be very surprising if the task performance goes up. we acknowledge this, and argue that our motivation for this setup is to deepen understanding, in particular the limitation or the capacity of the current architectures, which we expect can be reached with such strong supervision. another motivation is engineering: we could exploit negative examples in different ways, and establishing a better way will be of practical importance toward building an lm or generator that can be robust on particular linguistic constructions. the first research question we pursue is about this latter point: what is a better method to utilize negative examples that help lms to acquire robustness on the target syntactic constructions? regarding this point, we find that adding additional token-level loss trying to guarantee a margin between log-probabilities for the correct and incorrect words (e.g., log p(laughs|h) and log p(laugh|h) for (1a)) is superior to the alternatives. on the test set of @xcite , we show that lstm language models (lstm-lms) trained by this loss reach near perfect level on most syntactic constructions for which we create negative examples, with only a slight increase of perplexity about 1.0 point. past work conceptually similar to us is @xcite , which, while not directly exploiting negative examples, trains an lm with additional explicit supervision signals to the evaluation task. they hypothesize that lstms do have enough capacity to acquire robust syntactic abilities but the learning signals given by the raw text are weak, and show that multi-task learning with a binary classification task to predict the upcoming verb form (singular or plural) helps models aware of the target syntax (subject-verb agreement). our experiments basically confirm and strengthen this argument, with even stronger learning signals from negative examples, and we argue this allows us to evaluate the true capacity of the current architectures. in our experiments (section 4), we show that our margin loss achieves higher syntactic performance than their multi-task learning. another relevant work on the capacity of lstm-lms is @xcite , which shows that by distilling from syntactic lms @xcite , lstm-lms can improve their robustness on various agreement phenomena. we show that our lms with the margin loss outperform theirs in most of the aspects, further strengthening the argument about a stronger capacity of lstm-lms. the latter part of this paper is a detailed analysis of the trained models and introduced losses. our second question is about the true limitation of lstm-lms: are there still any syntactic constructions that the models cannot handle robustly even with our direct learning signals? this question can be seen as a fine-grained one raised by @xcite with a stronger tool and improved evaluation metric. among tested constructions, we find that syntactic agreement across an object relative clause (rc) is challenging. to inspect whether this is due to the architectural limitation, we train another lm on a dataset, on which we unnaturally augment sentences involving object rcs. since it is known that object rcs are relatively rare compared to subject rcs @xcite , frequency may be the main reason for the lower performance. interestingly, even when increasing the number of sentences with an object rc by eight times (more than twice of sentences with a subject rc), the accuracy does not reach the same level as agreement across a subject rc. this result suggests an inherent difficulty in tracking a syntactic state across an object rc for sequential neural architectures. we finally provide an ablation study to understand the encoded linguistic knowledge in the models learned with the help of our method. we experiment under reduced supervision at two different levels: (1) at a lexical level, by not giving negative examples on verbs that appear in the test set; (2) at a construction level, by not giving negative examples about a particular construction, e.g., verbs after a subject rc. we observe no huge score drops by both. this suggests that our learning signals at a lexical level (negative words) strengthen the abstract syntactic knowledge about the target constructions, and also that the models can generalize the knowledge acquired by negative examples to similar constructions for which negative examples are not explicitly given. the result also implies that negative examples do not have to be complete and can be noisy, which will be appealing from an engineering perspective. the most common evaluation metric of an lm is perplexity. although neural lms achieve impressive perplexity @xcite , it is an average score across all tokens and does not inform the models' behaviors on linguistically challenging structures, which are rare in the corpus. this is the primary motivation to separately evaluate the models' syntactic robustness by a different task. ## learning with negative examples now we describe four additional losses for exploiting negative examples. the first two are existing ones, proposed for a similar purpose or under a different motivation. as far as we know, the latter two have not appeared in past work. 4 we note that we create negative examples by modifying the original wikipedia training sentences, not sentences in the test set. as a running example, let us consider the case where sentence (6a) exists in a mini-batch, from which we create a negative example (6b). (6) a. an industrial park with several companies is located in the close vicinity. b. * an industrial park with several companies are located in the close vicinity. notations by a target word, we mean a word for which we create a negative example (e.g., is). we distinguish two types of negative examples: a negative token and a negative sentence; the former means a single incorrect word (e.g., are), while the latter means an entire ungrammatical sentence. ## experiments on additional losses we first see the overall performance of baseline lstm-lms as well as the effects of additional losses. throughout the experiments, for each setting, we train five models from different random seeds and report the average score and standard deviation. the code is available at https://github.com/aistairc/lm syntax negative. @xcite . k19 is the result of distilled two-layer lstm-lm from rnngs @xcite . vp: verb phrase; pp: prepositional phrase; src: subject relative clause; and orc: object-relative clause. margin values are set to 10, which works better according to figure 1 . perplexity values are calculated on the test set of the wikipedia dataset. the values of m&l18 and k19 are copied from @xcite . notice that our baseline lstm-lm (section 2.2) performs much better than marvin and linzen (2018)'s lm. a similar observation is recently made by @xcite . 8 this suggests that the original work underestimates the true syntactic ability induced by lstm-lms. the table also shows the results by their distilled lstm-lm from rnngs (section 1). ## limitations of lstm-lms in importantly, the accuracies for a subject rc are more stable, reaching 99.8% with the token-level margin loss, although the content words used in the examples are common. 9 it is known that object rcs are less frequent than subject rcs @xcite , and it could be the case that the use of negative examples still does not fully alleviate this factor. here, to understand the true limitation of the current lstm architecture, we try to eliminate such other factors as much as possible under a controlled experiment. 9 precisely, they are not the same. examples of object rcs are divided into two categories by the animacy of the main subject (animate or not), while subject rcs only contain animate cases. if we select only animate examples from object rcs the vocabularies for both rcs are the same, remaining only differences in word order and inflection, as in (7, 8). ## discussion and conclusion our results with explicit negative examples are overall positive. we have demonstrated that models exposed to these examples at training time in an appropriate way will be capable of handling the targeted constructions at near perfect level except a few cases. we found that our new token-level margin loss is superior to the other approaches and the remaining challenging cases are dependencies across an object relative clause. object relative clauses are known to be harder for a human as well, and our results may indicate some similarities in the sentence processing behaviors by a human and rnn, though other studies also find some dissimilarities between them @xcite . the difficulty of object relative clauses for rnn-lms has also been observed in the prior work @xcite . a new insight provided by our study is that this difficulty holds even after alleviating the frequency effects by augmenting the target structures along with direct supervision signals. this indicates that rnns might inherently suffer from some memory limitation like a human subject, for which the difficulty of particular constructions, including center-embedded object relative clauses, are known to be incurred due to memory limitation @xcite rather than purely frequencies of the phenomena. in terms of language acquisition, the supervision provided in our approach can be seen as direct negative evidence @xcite . since human learners are known to acquire syntax without such direct feedback we do not claim that our proposed learning method itself is cognitively plausible. one limitation of our approach is that the scope of negative examples has to be predetermined and fixed. alleviating this restriction is an important future direction. though it is challenging, we believe that our final analysis for transferability, which indicates that the negative examples do not have to be complete and can be noisy, suggests a possibility of a mechanism to induce negative examples themselves during training, perhaps relying on other linguistic cues or external knowledge.