How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context (2020)

Abstract: We study the influence of context on sentence acceptability. First we compare the acceptability ratings of sentences judged in isolation, with a relevant context, and with an irrelevant context. Our results show that context induces a cognitive load for humans, which compresses the distribution of ratings. Moreover, in relevant contexts we observe a discourse coherence effect that uniformly raises acceptability. Next, we test unidirectional and bidirectional language models in their ability to predict acceptability ratings. The bidirectional models show very promising results, with the best model achieving a new state-of-the-art for unsupervised acceptability prediction. The two sets of experiments provide insights into the cognitive aspects of sentence processing and central issues in the computational modeling of text and discourse.

## introduction sentence acceptability is the extent to which a sentence appears natural to native speakers of a language. linguists have often used this property to motivate grammatical theories. computational language processing has traditionally been more concerned with likelihood-the probability of a sentence being produced or encountered. the question of whether and how these properties are related is a fundamental one. lau et al. (2017b) experiment with unsupervised language models to predict acceptability, and they obtained an encouraging correlation with human ratings. this raises foundational questions about the nature of linguistic knowledge: if probabilistic models can acquire knowledge of sentence acceptability from raw texts, we have prima facie support for an alternative view of language acquisition that does not rely on a categorical grammaticality component. it is generally assumed that our perception of sentence acceptability is influenced by context. sentences that may appear odd in isolation can become natural in some environments, and sentences that seem perfectly well formed in some contexts are odd in others. on the computational side, much recent progress in language modeling has been achieved through the ability to incorporate more document context, using broader and deeper models (e.g., devlin et al., 2019; yang et al., 2019). while most language modeling is restricted to individual sentences, models can benefit from using additional context (khandelwal et al., 2018). however, despite the importance of context, few psycholinguistic or computational studies systematically investigate how context affects acceptability, or the ability of language models to predict human acceptability judgments. two recent studies that explore the impact of document context on acceptability judgments both identify a compression effect (bernardy et al., 2018; bizzoni and lappin, 2019). sentences perceived to be low in acceptability when judged without context receive a boost in acceptability when judged within context. conversely, those with high out-of-context acceptability see a reduction in acceptability when context is presented. it is unclear what causes this compression effect. is it a result of cognitive load, imposed by additional processing demands, or is it the consequence of an attempt to identify a discourse relation between context and sentence? we address these questions in this paper. to understand the influence of context on human perceptions, we ran three crowdsourced experiments to collect acceptability ratings from human annotators. we develop a methodology to ensure comparable ratings for each target sentence in isolation (without any context), in a relevant threesentence context, and in the context of sentences randomly sampled from another document. our results replicate the compression effect, and careful analyses reveal that both cognitive load and discourse coherence are involved. to understand the relationship between sentence acceptability and probability, we conduct experiments with unsupervised language models to predict acceptability. we explore traditional unidirectional (left-to-right) recurrent neural network models, and modern bidirectional transformer models (e.g., bert). we found that bidirectional models consistently outperform unidirectional models by a wide margin, calling into question the suitability of left-to-right bias for sentence processing. our best bidirectional model achieves simulated human performance on the prediction task, establishing a new state-of-the-art. ## modeling acceptability in this section, we explore computational models to predict human acceptability ratings. we are interested in models that do not rely on explicit supervision (i.e., we do not want to use the acceptability ratings as labels in the training data). our motivation here is to understand the extent to which sentence probability, estimated by an unsupervised model, can provide the basis for predicting sentence acceptability. to this end, we train language models (section 3.1) using unsupervised objectives (e.g., next word prediction), and use these models to infer the probabilities of our test sentences. to accommodate sentence length and lexical frequency we experiment with several simple normalization methods, converting probabilities to acceptability measures (section 3.2). the acceptability measures are the final output of our models; they are what we use to compare to human acceptability ratings. ## linguists' examples one may argue that our dataset is potentially biased, as round-trip machine translation may introduce particular types of infelicities or unusual features to the sentences (graham et al., 2019). lau et al. (2017b) addressed this by creating a dataset where they sample 50 grammatical and 50 ungrammatical sentences from adger (2003)'s syntax textbook, and run a crowdsourced experiment to collect their user ratings. lau et al. (2017b) found that their unsupervised language models (e.g., simple recurrent networks) predict the acceptability of these sentences with similar performances, providing evidence that their modeling results are robust. we test our pre-trained models using this linguist-constructed dataset, and found similar observations: gpt2, bert cs , and xlnet bi produce a penlp correlation of 0.45, 0.53, and 0.58, respectively. these results indicate that these language models are able to predict the acceptability of these sentences reliably, consistent with our modeling results with round-trip translated sentences (section 3.4). although the correlations are generally lower, we want to highlight that these linguists' examples are artificially constructed to illustrate specific syntactic phenomena, and so this constitutes a particularly strong case of outof-domain prediction. these texts are substantially different in nature from the natural text that the pre-trained language models are trained on (e.g., the linguists' examples are much shorter-less than 7 words on average-than the natural texts). ## related work acceptability is closely related to the concept of grammaticality. the latter is a theoretical construction corresponding to syntactic wellformedness, and it is typically interpreted as a binary property (i.e., a sentence is either grammatical or ungrammatical). acceptability, on the other hand, includes syntactic, semantic, pragmatic, and non-linguistic factors, such as sentence length. it is gradient, rather than binary, in nature (denison, 2004; sorace and keller, 2005; sprouse, 2007). linguists and other theorists of language have traditionally assumed that context affects our perception of both grammaticality (bolinger, 1968) and acceptability (bever, 1970), but surprisingly little work investigates this effect systematically, or on a large scale. most formal linguists rely heavily on the analysis of sentences taken in isolation. however, many linguistic frameworks seek to incorporate aspects of context-dependence. dynamic theories of semantics (heim, 1982; kamp and reyle, 1993; groenendijk and stokhof, 1990) attempt to capture intersentential coreference, binding, and scope phenomena. dynamic syntax (cann et al., 2007) uses incremental tree construction and semantic type projection to render parsing and interpretation discourse dependent. theories of discourse structure characterize sentence coherence in context through rhetorical relations (mann and thompson, 1988; asher and lascarides, 2003), or by identifying open questions and common ground (ginzburg, 2012). while these studies offer valuable insights into a variety of context related linguistic phenomena, much of it takes grammaticality and acceptability to be binary properties. moreover, it is not formulated in a way that permits fine-grained psychological experiments, or wide coverage computational modeling. psycholinguistic work can provide more experimentally grounded approaches. greenbaum (1976) found that combinations of particular syntactic constructions in context affect human judgments of acceptability, although the small scale of the experiments makes it difficult to draw general conclusions. more recent work investigates related effects, but it tends to focus on very restricted aspects of the phenomenon. for example, zlogar and davidson (2018) investigate the influence of context on the acceptability of gestures with speech, focussing on interaction with semantic content and presupposition. the priming literature shows that exposure to lexical and syntactic items leads to higher likelihood of their repetition in production (reitter et al., 2011), and to quicker processing in parsing under certain circumstances (giavazzi et al., 2018). frameworks such as act-r (anderson, 1996) explain these effects through the impact of cognitive activation on subsequent processing. most of these studies suggest that coherent or natural contexts should increase acceptability ratings, given that the linguistic expressions used in processing become more activated. warner and glass (1987) show that such syntactic contexts can indeed affect grammaticality judgments in the expected way for garden path sentences. cowart (1994) uses comparison between positive and negative contexts, investigating the effect of contexts containing alternative more or less acceptable sentences. but he restricts the test cases to specific pronoun binding phenomena. none of the psycholinguistic work investigates acceptability judgments in real textual contexts, over large numbers of test cases and human subjects. some recent computational work explores the relation of acceptability judgments to sentence probabilities. lau et al. (2015, 2017b) show that the output of unsupervised language models can correlate with human acceptability ratings. warstadt et al. (2018) treat this as a semisupervised problem, training a binary classifier on top of a pre-trained sentence encoder to predict acceptability ratings with greater accuracy. bernardy et al. (2018) explore incorporating context into such models, eliciting human judgments of sentence acceptability when the sentences were presented both in isolation and within a document context. they find a compression effect in the distribution of the human acceptability ratings. bizzoni and lappin (2019) observe a similar effect in a paraphrase acceptability task. one possible explanation for this compression effect is to take it as the expression of cognitive load. psychological research on the cognitive load effect (sweller, 1988; ito et al., 2018; causse et al., 2016; park et al., 2013) indicates that performing a secondary task can degrade or distort subjects' performance on a primary task. this could cause judgments to regress towards the mean. however, the experiments of bernardy et al. (2018) and bizzoni and lappin (2019) do not allow us to distinguish this possibility from a coherence or priming effect, as only coherent contexts were considered. our experimental setup improves on this by introducing a topic identification task and incoherent (random) contexts in order to tease the effects apart. ## conclusions and future work we found that processing context induces a cognitive load for humans, which creates a compression effect on the distribution of acceptability ratings. we also showed that if the context is relevant to the sentence, a discourse coherence effect uniformly boosts sentence acceptability. our language model experiments indicate that bidirectional models achieve better results than unidirectional models. the best bidirectional model performs at a human level, defining a new state-of-the art for this task. in future work we will explore alternative ways to present sentences for acceptability judgments. we plan to extend tdlm, incorporating a bidirectional objective, as it shows significant promise. it will also be interesting to see if our observations generalize to other languages, and to different sorts of contexts, both linguistic and non-linguistic.