Overestimation of Syntactic Representation in Neural Language Models (2020)

Abstract: With the advent of powerful neural language models over the last few years, research attention has increasingly focused on what aspects of language they represent that make them so successful. Several testing methodologies have been developed to probe models’ syntactic representations. One popular method for determining a model’s ability to induce syntactic structure trains a model on strings generated according to a template then tests the model’s ability to distinguish such strings from superficially similar ones with different syntax. We illustrate a fundamental problem with this approach by reproducing positive results from a recent paper with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs.

## introduction in recent years, rnn-based systems have proven excellent at a wide range of nlp tasks, sometimes achieving or even surpassing human performance on popular benchmarks. their success stems from the complex but hard to interpret, representations that they learn from data. given that syntax plays a critical role in human language competence, it is natural to ask whether part of what makes these models successful on language tasks is an ability to encode something akin to syntax. this question pertains to syntax "in the meaningful sense," that is, the latent, hierarchical, largely context-free phrase structure underpinning human language as opposed to superficial or shallow issues of word order @xcite @xcite . clearly, syntactic information can be explicitly incorporated into neural systems to great effect (e.g., @xcite . less certain is whether such systems induce something akin to hierarchical structure (henceforth, "syntax") on their own when not explicitly taught to do so. uncovering what an rnn actually represents is notoriously difficult, and several methods for probing rnns' linguistic representations have been developed to approach the problem. most directly, one can extract finite automata (e.g., @xcite from the network or measure its state as it processes inputs to determine which neurons attend to what features (e.g., @xcite @xcite . alternatively, one can present a task which only a syntactic model should be able to solve, such as grammaticality discrimination or an agreement task, and then infer if a model has syntactic representations based on its behavior @xcite @xcite . in practice, simple sentences far outnumber the ones that require syntax in any natural corpus, which may obscure evaluation @xcite . one way around this, referred to here as templatebased probing, is to either automatically generate sentences with a particular structure or extract just the relevant ones from a much larger corpus. templates have been used in a wide range of studies, including grammaticality prediction (e.g., @xcite , long-distance dependency resolution, and agreement prediction tasks (e.g., @xcite . by focusing on just relevant structures that match a given template rather than the gamut of naturally occurring sentence, templatebased probing offers a controlled setting for evaluating specific aspects of a model's representation. the crux of behavioral evaluation is the assertion that the chosen task effectively distinguishes between a model that forms syntactic representations and one which does not. this must be demonstrated for each task -if a model that does not capture syntax can pass the evaluation, then there is no conclusion to be drawn. however, this step is often omitted (but not always, e.g., @xcite . moreover, templatebased generation removes the natural sparse and diverse distribution of sentence types, increasing the chance that a system might pick up on nonsyntactic patterns in the data, further increasing the importance of a clear baseline. this problem is most clearly illustrated with an example. in the following sections, we introduce @xcite novel psycholinguisticsinspired template-based probe of relative clause types, which was taken as evidence in support of syntactic representation in lstms. we then pass pvsl's test with two non-syntactic baselines: an n-gram lm which can only capture short-distance word order of concrete types (section 3), and an lstm trained on scrambled inputs (section 4). these baselines show that a combination of collocation and lexical representation can account for pvsl's results, which highlights a critical flaw in that experimental design. following that, we argue that it is unlikely that lstms induce syntactic representations given current evidence and suggest an alternative angle for the question (section 5). ## prasad, van schijndel, & linzen 2019 prasad et al. (pvsl; 2019) leverage an analogy from psycholinguistic syntactic priming to test whether an lstm is able to distinguish between sentences with different syntactic structures. when human subjects are primed by receiving an example of some input, their expectation of receiving similar subsequent input will temporarily increase relative to their expectation of other inputs. this can be used to test questions about syntax because once one is primed with sentences with a specific structure, subsequent sentences with shared structure will tend to show decreased surprisal responses relative to those with different structures. pvsl observe that this procedure may be applied to neural networks as well. since a model's surprisal upon receiving some input decreases as it receives subsequent similar inputs, one could cumulatively "prime" a model by adapting it toward a certain class of input @xcite . as the reasoning goes, if the model can be primed for a particular syntactic structure, that implies that it is able to recognize that structure and therefore has learned a representation for it. this paradigm is used to assess an lstm's ability to distinguish between five superficially similar but structurally distinct sentences types: those con-taining an unreduced object relative clause (rc), reduced object rc, unreduced passive subject rc, unreduced passive subject rc, and active subject rc, as well as two types matched for lexical content: passive subj./obj. rc-matched coordination sentences and active subj. rc-matched coordination. (1-2) present an example object rc and subject rc sentence to illustrate the structures. 1 these are distinguished syntactically by the origin of their subjects. in the first case, the subject of the sentence, 'the cake,' is also the object of the relative clause (position indicated by underscore), but in the second case, the sentence subject, 'the baker,' is also the subject of the relative clause. as pvsl note, if a model were able to track the position of the implicit syntactic origin, it would be able to distinguish these sentence types, so one would expect the model to exhibit a greater adaptation effect (greater decrease in surprisal) when primed and tested on the same sentence type than if primed on one type and tested on the other. ## n-gram model we begin by training an n-gram language model (through 4-grams) with knesser-ney smoothing @xcite with the nltk toolkit to determine whether it could be primed to distinguish pvsl's sentence types. an n-gram lm can only learn surface collocations and so cannot capture (hierarchical) syntax, so if it produces a significant differential adaptation effect, then the experiment is not able to discriminate between models which capture syntax from those which do not. adaptation and testing were carried out with pvsl's adaptation and test sets, and lm training was modified slightly to address n-gram models' characteristics. they have no recency bias, unlike rnns, which diminishes the impact of adaptation. as such, 20 smaller models were trained on disjoint subsets of wikitext-2 rather than the full-sized wikitext-103 subsets. plotting and statistical analysis were carried out with pvsl's code foot_1 . figure 1 shows the average adaptation effect observed when the models are adapted and tested on the same sentence type or different sentence types. importantly, the same-type adaptation effect is greater than the different-type effect for six of seven sentence types (unreduced passive rc is reversed). although the adaptation effect is uniformly weaker than observed for pvsl's lstm lms, there is a statistically significant difference between the same-type and different-type effects for six of seven sentence types. figure 2 compares the adaptation effect over rcs compared to coordination sentences. the ngram models show a significantly greater sametype adaptation effect for coordination but not for rcs. a small but significant increase in voiceand reduction-matched adaptation over unmatched combinations was found (matched-passive matched reduction: 0.610, matched-passive mismatchedreduction: 0.594, mismatched-passive matchedreduction: 0.575, mismatched-passive mismatchedreduction: 0.572). ## scrambled-input model next, the same van schijndel and linzen (2018) trained lstm lms which pvsl employed were adapted on altered versions of their adaptation sets in which the word order of each sentence was scrambled to destroy the sentence's syntax while retaining its lexical content, then tested on the original non-scrambled test sets. even though pvsl minimize the amount of lexical overlap in the adaptation and test sets, it may be the case that the models pick up on lexical similarities because of the felicity constraint which was imposed on them. scrambling was random on a sentence-bysentence basis. results were averaged across all the adaptation sets and models (as they were in pvsl), so the effect of any individual accidentally grammatical scramble was diminished. figure 3 shows the average differential adaptation effects on these scrambled annotation runs. the same-type adaptation effect is significantly greater than different-type for six of seven sentence types (except subject coord.), and the largest relative difference is seen for unreduced passive rcs, the only type for which the n-gram models produced a reverse effect. overall, the adaptation effect is an order of magnitude larger than for the n-gram models' but still smaller than pvsl's. figure 4 shows differential adaptation effects for rc and coordination sentences. a backward effect is observed for sentences adapted on coordination, but a large positive effect is found for those adapted on rc sentences. this is the complement of what was found for n-gram models. a significant positive difference was found between sentence types matched and unmatched in passives and reduction (matched-passive matched reduction: 0.65, matched-passive mismatched-reduction: 0.53, mismatched-passive matched-reduction: 0.53, mismatched-passive mismatched-reduction: 0.43). ## discussion these results call into question the van schijndel and @xcite and @xcite syntactic priming paradigm's ability to distinguish models which represent syntax from those which rely on shallow phenomena by achieving a positive result with two non-syntactic baseline models. first, success in the priming paradigm is measured by whether or not adaptation reduces surprisal, but not by how much, so even though both baseline models tested here reduce surprisal by less than pvsl's models on average, they still pass the success criterion. to put it another way, pvsl report quantitative results but do not actually establish what would constitute a meaningful effect size. even though the effect sizes of both our baseline replications were smaller, pvsl could have reported the results from our baseline models instead of their actual model and drawn the same conclusions. second, the fact that our surface word order ngram model and lexical similarity-only scrambled lstm lms also show surprisal effects draws into question the basic claim that only a syntactic model would respond to adaptation: it is our hypothesis that the combined effect of word order and lexical similarity are what drive the lstm models' larger effect. this is upheld, especially when it is noted that the adaptation effects of both baselines complement each other. both alternative sources of information are well known in the community and have been tested in the past @xcite . this reiterates the need for proper baseline testing in computational linguistics and for informative evaluations. this highlights a more general problem with template-based probing, namely, that the unnatural lack of sentence diversity imposed by the templates imposes unintended regularity for models to latch onto. given the well-known observation that neural models will "take the easy way out" given the presence of this unintended surface information @xcite @xcite , and other work suggesting that lstms do not necessarily induce syntactic structure @xcite @xcite , one must take successes in template-based probing studies with a grain of salt. the evaluation of non-syntactic baselines is an easy-to-implement way to combat the tendency of these behavioral probes to overestimate language models' abilities. to improve the priming paradigm in particular, one would need to establish a success metric that discriminates between baselines and alter the experimental setup to mitigate information side channels. one possibility would be to include infelicitous "colorless green ideas" sentences with grammatical syntax (cf. @xcite , which might decrease the lexical similarity problem. removing the issue altogether could require enforcing completely lexically disjoint training, adaptation, and test sets, but we cannot reasonably expect a model to function when it has no generalizations to work with, and demanding lexically distinct sets (including function words) greatly limits the set of phenomena that could be studied.