Pragmatic competence of pre-trained language models through the lens of discourse connectives (2021)

Abstract: As pre-trained language models (LMs) continue to dominate NLP, it is increasingly important that we understand the depth of language capabilities in these models. In this paper, we target pre-trained LMs’ competence in pragmatics, with a focus on pragmatics relating to discourse connectives. We formulate cloze-style tests using a combination of naturally-occurring data and controlled inputs drawn from psycholinguistics. We focus on testing models’ ability to use pragmatic cues to predict discourse connectives, models’ ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives. We find that although models predict connectives reasonably well in the context of naturally-occurring data, when we control contexts to isolate high-level pragmatic cues, model sensitivity is much lower. Models also do not show substantial humanlike temporal preferences. Overall, the findings suggest that at present, dominant pre-training paradigms do not result in substantial pragmatic competence in our models.

## introduction pre-trained language models continue to display impressive performance across various domains of nlp, raising the important question of exactly what level of linguistic competence these models have acquired, particularly during the pre-training process. although models show outstanding performance on downstream tasks, there is also evidence that their handling of language is shallow @xcite . in this paper we examine aspects of pre-trained lms competence in pragmatics, with a particular focus on pragmatic reasoning surrounding discourse connectives. discourse connectives are linguistic elements that connect two neighboring events or sentences, signalling the discourse re-lation that exists between them. discourse connectives reflect and convey pragmatic information about relationships between events being described, and connectives can also be associated with pragmatically-enriched meanings that go beyond their literal meanings. if pre-trained lms have acquired competence in pragmatics of a language, we would expect them to be able to use cues in context to infer what discourse relation holds between events, and by extension which discourse connective is appropriate. additionally, we would expect them to be able to pick up on pragmatic inferences generated by the connectives themselves. these are the questions that we test in this paper. we formulate all of our tests as cloze tasks, so that we can test the pre-trained lms without finetuning. we begin with a general task of connective prediction, to gauge how well models can use surrounding sentence context to predict the appropriate discourse connective. we then move on to more controlled tests, inspired by psycholinguistics, to ask more targeted questions about models' use of contextual cues and handling of implicature. the first of these examines models' ability to use high-level pragmatic cues to infer appropriate connectives, in the absence of clear syntactic or lexical cues. the second examines the extent to which model predictions reflect understanding of temporal implicatures in settings of implicature cancellation and reinforcement. finally, we test whether models show humanlike preferences in terms of temporal dynamics of event mentions. we test a range of pre-trained lms with these analyses. our results indicate that although models show reasonable connective predictions in naturally-occurring data, they lack the pragmatic sensitivity to perform well on our controlled tests, and they don't show any strong humanlike preference in terms of temporal dynamics. the results suggest that for the moment, the currently dominant pre-training paradigms are not yielding clear pragmatic competence in nlp models, at least with respect to discourse connectives. we make all code and test data available for additional testing. foot_0 2 related work connectives have been studied in nlp models from a number of angles. some works have focused on improving automatic extraction of connectives via heuristics @xcite or dependency parses @xcite , so as to use connective prediction during model pre-training. while these works share with ours the use of discourse connective prediction, we differ in focusing not on modifying model training, but on evaluating and analyzing existing models, to better understand the extent to which existing pre-training paradigms confer sensitivity to pragmatic cues. others have studied models' competence in classifying discourse relations on the basis of connectives or contexts. @xcite show that using syntactic features along with the connectives themselves, a supervised classifier is able to identify the discourse relation that a connective represents. @xcite study competence of multilingual models in various discourse tasks by evaluating cross-lingual zero-shot transfer in a range of sentence encoders-among these tasks is classification of discourse relations, though they focus on implicit discourse relations, where connectives are not the source of discourse information. @xcite also evaluate pre-trained language models' performance on discourse level relations, including a task of discourse connective prediction: given two sentences, the task is to predict the explicit connective. they fine-tune pretrained models on this classification task, finding that for the discourse connective prediction task, different models produce similar patterns. patterson and kehler (2013) also examine connective prediction, training a classifier to predict whether a connective is present, and finding that classifiers are able to perform this prediction task on the basis of shallow linguistic features alone. our work in this paper is similar to these prior works in examining the ability of models to capture discourse information through tasks involving prediction of connectives and discourse relations. however, we differ critically from prior work in asking not whether we can train models to make these predictions-rather, our question is to what extent pre-trained lms have already developed pragmatic knowledge relating to connectives, as a byproduct of the pre-training process itself. consequently, we deviate from this prior work in that we do not fine-tune models or do any supervised training for connective predictions-instead, we formulate our tests as word prediction tasks and test whether model predictions reflect an understanding of the pragmatics surrounding connectives. our work also distinguishes itself from prior work on connectives in that we anchor our tests in insights and methods from neuro-and psycholinguistic experiments, which are well controlled and designed for assessing linguistic behavior and competence in a targeted manner. this helps us to tease out potential superficial cues that may inflate perceived levels of pragmatic competence. modifications that we make to the original psycholinguistic experimental items are furthermore grounded in established findings of pragmatic theory. ## predicting connectives with controlled context the results in section 4 give a preliminary sense of models' behavior in using context to predict connectives, but it is difficult to discern from these uncontrolled data precisely what types of cues the models may be using to inform connective predictions. in particular, if we are interested in models' ability to use high-level pragmatic information to infer the appropriate discourse relation, it is important that we control for lower-level syntactic and lexical cues that may be predictive of connectives, but that tell us less about models' sensitivity to pragmatics. previous works have shown that lexical and semantic cues can be used for predicting connectives and discourse relations @xcite , and that certain kinds of relations co-occur at rates greater than chance @xcite , supporting the possibility that non-pragmatic cues alone can likely lead to strong connective prediction performance. in order to better isolate high-level pragmatic cues, we take advantage of sentences designed by @xcite for a psycholinguistic study of human language processing. the original psycholinguistic experiment tested how different discourse connectives facilitate human language comprehension, and the extent to which connectives can elicit predictions of upcoming content. the experimental items constitute minimal pairs with nearly identical syntax and word content-but a slight difference late in the context makes it such that a causal connective is appropriate in one version, while a concessive connective is appropriate in the other. 4 taking advantage of this controlled minimal pair set-up, we adapt the items from this study to formulate a connective prediction task- ## discourse connectives and implicature section 4 suggests that models are quick to predict and in contexts that generally support a discourse connective. here we test the extent to which models understand what the connective and actually means-more specifically, we examine the extent to which models pick up on temporal implicatures that humans commonly interpret as part of the meaning of and. this section will again use controlled minimal pairs of contexts, inspired by findings in linguistics and psycholinguistics. our tests here make use of the pragmatic notion of implicature: non-literal, enriched meaning of words, phrases, or sentences generated by inferences about speaker intent. for instance, the sentence "some people have pets" literally means that "there are a non-zero number of people who have pets". in addition to this literal meaning, a common implicature in interpretation of this sentence is that "some but not all people have pets". we focus on an implicature generated by the connective and when joining two events-namely, the implicature that and actually means and then (i.e., the two events are being mentioned in the temporal order in which they actually occurred). this has been studied by @xcite , noting the oddness of sentences like "jane got into bed and brushed her teeth", which seems to carry the clear implication that jane brushed her teeth in bed. @xcite also study this implicature in children, finding that compared to younger children, older children and adults generate more and then implicatures when events are joined by and. our tests will focus first on whether models seem to be sensitive to this and then implicature. to test whether models pick up on this implicature, we cannot simply test models' aptitude at predicting the connective and in context. instead, we make use of two additional hallmarks of implicatures @xcite : 1) that they can be reinforced with, e.g., "which is to say", and 2) that they can be canceled with, e.g., "in fact". (for instance, we can reinforce the "some but not all" implicature above by saying, "some people have pets-which is to say, some, but not all, people have pets". alternatively, we can cancel the "some but not all" implicature by saying "some people have pets-in fact, all people have pets".) these tests are well established in the linguistics literature for teasing apart implied meanings from compositional truth conditions @xcite @xcite @xcite @xcite . we will leverage sentences featuring reinforcement and cancellation of the and then implicature to test our models. we create a dataset of item pairs containing events joined by and, followed by a reinforcement or cancellation. when we examine model predictions in the masked positions of these items, the critical question is the relative probability that they assign to completions of before versus after. if models generate the natural and then implicature for the connective and (and if they have the pragmatic compe- condition example item reinforcement test maggie did the paperwork by hand and the company bought new computers, which is to say, maggie did the paperwork by hand [mask] the company bought new computers. (ideal target probs: @xmath0, in fact, maggie did the paperwork by hand [mask] the company bought new computers. (ideal target probs: @xmath1: prediction accuracy in implicature test, with cancellation and reinforcement settings as in section 5, we see that although models may make correct predictions on individual items, their ability to choose the correct connective in both items of a pair is very limited. chance-level performance on this criterion is 25%, and it is clear that most models are performing substantially below chance level-and even the highest accuracies remain slightly below chance. the results indicate that this form of pragmatic competence-inferring temporal implicature and deploying it in cancellation and reinforcement environments-remains outside of models' current capacity. ## sensitivity to event order and corresponding connectives the previous section tested whether models, like humans, infer that a connective and joining two events has an implied meaning of and then. a related finding in psycholinguistics is from @xcite , who examine the effect of before and after clauses on sentence processing in both sentence-initial contexts and sentence-medial contexts. their results provide further evidence that human brains have a preference for events to be mentioned in chronological order. we leverage this existing experiment to probe whether models, in their connective predictions, show similar preferences for events to be mentioned in the order in which they occur in real life. importantly, this test differs from the above tests in that models cannot be said to behave "correctly" or "incorrectly", on the basis of the presence/absence of this preference. rather, this serves as a more general test of whether models' connective predictions reflect humanlike trends with respect to temporal implications. we adapt the politzer-ahles et al. (2017) materials to form a connective prediction task. examples are shown in as we have established above, models' lack of the target trend in this experiment cannot be considered "incorrect" in any sense-however, it does indicate that models lack yet another humanlike pattern with respect to processing of discourse connectives, where such a pattern would have suggested finer-grained sensitivity to temporal dynamics. ## discussion in the above experiments, we have studied the competence of pre-trained lms in predicting and interpreting connectives. examining connective prediction in naturally-occurring data suggests that models may have certain go-to connectives that they predict across a variety of contexts. so we turn to more controlled tests inspired by psycholinguistics, to examine the extent to which models' handling of connectives reflects pragmatic competence. the results of these controlled tests suggest that models are not yet equipped to use subtle pragmatic cues to inform connective predictions-whether the test involves distinguishing causal from concessive discourse relations in settings of high syntactic and lexical overlap, or making predictions based on inference, reinforcement, or cancellation of implicatures. we also find that models appear to be insensitive to redundancy or contradiction, and that although certain models may have a slight tendency to prefer connectives that suggest chronological event mentions, this tendency is weak at best. variation between models is fairly minor, though some roberta and albert variants at times distinguish themselves in coming closer to chancelevel performance when other models are close to 0% accuracy. this could be attributable to larger pre-training data in the case of roberta, and in the case of albert, we speculate that some benefit may be derived from the sentence order prediction loss @xcite , which may encourage sensitivity to certain discourse dynamics. however, it is important to note that in these cases the models still at best barely surpass chance-level performance, so this does not indicate any particularly strong pragmatic competence from these models. why do models perform so poorly on the controlled tests? it is clear, of course, that these models have not been trained to do these specific tasks. however, if models have competence in pragmatic reasoning, that competence should be reflected in the preferences and distinctions tested for here. our use of minimal pairs creates a particular challenge, in reducing models' capacity to succeed on the basis of shallower types of cues-syntactic, lexicalthat they are more likely to have learned to priori-tize. however, this means that these tests hopefully give us a clearer look at models' ability to use pragmatic cues per se. all in all, our results point to a situation in which sophisticated pragmatic reasoning is not yet a property of current models-at least not the aspects of pragmatics tested for here. what are the implications of these results for our approach to these models? in terms of gauging linguistic competence that emerges from pretraining, the results suggest that word prediction objectives alone may not suffice to force models to learn nuances of pragmatic reasoning. models tested here, at least, do not suggest that such pragmatic reasoning has been learned. the results on naturally-occurring data may suggest one source of pre-training limitations: models may be able to achieve reasonable prediction outcomes simply by defaulting to versatile connectives in a wide range of contexts, and by relying on lower-level syntactic and lexical cues. for models to learn pragmatics, it may be necessary to reduce reliability of shallower cues, scaffold learning with more meaning-rich supervision, or both. enriched pragmatic competence may be achievable through fine-tuning of pre-trained models, but a fine-tuning approach will be subject to the same risks of models defaulting to shallower heuristics-so the same considerations will apply. we leave the problem of improving models' pragmatic competence for future work. ## conclusion the above experiments have examined pragmatic competence in pre-trained language models, with respect to discourse connectives. results suggest that models are not yet equipped to use high-level pragmatic cues or reasoning to guide predictive behaviors, even if they show reasonable predictive accuracy in naturally-occurring data. we suggest that arriving at more pragmatically competent models may require greater control of shallow cues, or use of more meaning-rich training signal. we hope that this work will help to shed light on the linguistic competence of pre-trained lms, and ultimately contribute to advancement in the pragmatic competence of models in nlp.