Do Language Models Make Human-like Predictions about the Coreferents of I talian Anaphoric Zero Pronouns? (2022)

Abstract: Some languages allow arguments to be omitted in certain contexts. Yet human language comprehenders reliably infer the intended referents of these zero pronouns, in part because they construct expectations about which referents are more likely. We ask whether Neural Language Models also extract the same expectations. We test whether 12 contemporary language models display expectations that reflect human behavior when exposed to sentences with zero pronouns from five behavioral experiments conducted in Italian by Carminati (2005). We find that three models - XGLM 2.9B, 4.5B, and 7.5B - capture the human behavior from all the experiments, with others successfully modeling some of the results. This result suggests that human expectations about coreference can be derived from exposure to language, and also indicates features of language models that allow them to better reflect human behavior.

## introduction in italian, like other pro-drop ('pronoun-dropping') languages, verbal arguments that would usually be expressed by pronouns in languages such as english can be omitted under certain circumstances. for example, consider the sentence in (1) from @xcite . in this sentence, the referent of the 'dropped' pronoun-generally referred to as a zero or null pronoun-can be inferred from the fact that the adjective contenta is feminine; thus, maria is the most likely subject of the second clause. resolving the referents of anaphoric zero pronouns like this is a long-standing, important, and active area of research in natural language understanding (jurafsky and martin, 2021; for examples, see @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . it has been argued that aiming for human-likeness in natural language processing systems is vital if we want our natural language understanding systems to behave not only as humans do, but also as human users expect them to (see, e.g., @xcite @xcite . this is particularly true for zero anaphora resolution, and pronoun resolution more generally. as an illustration of the prominence of reference resolution, one pronoun resolution task, the winograd schema challenge @xcite ; based on work by @xcite , has been referred to as 'an alternative to the turing test' @xcite . so, how do humans resolve coreference? the evidence suggests that we use a range of cuesfor example, agreement information as in (1), but also factors such as world knowledge and common sense @xcite @xcite @xcite . in addition, pronoun resolution is shaped by our expectations about the next entity that is likely to be mentioned and what argument it should take @xcite @xcite . for example, crosslinguistic work has found a bias towards expecting a subject pronoun to refer to an antecedent subject (for discussion, see @xcite . this has been demonstrated experimentally with sentences such as those in (2). the same principles apply in zero anaphora resolution. @xcite , for example, tests human expectations by investigating how long it takes for experimental participants to read stimuli with certain linguistic features, based on the well-established finding that contextually expected words are read faster than unexpected words, demonstrating an increased processing difficulty when these expectations are violated (see, e.g. @xcite @xcite . @xcite finds that the main clauses of sentences such as (1)-that is, the part of the sentence containing the zero subject pronoun, i.e., era contenta ('[she] was happy')-are read faster when the zero pronoun co-refers with a subject antecedent, as in (1), than when it co-refers with the antecedent object, as in (3). the question, then, if we want human-like zero anaphora resolution, is how to test whether a given zero anaphora resolution system is able to reflect these human expectations. in the present study, we propose a method to do just that. the vast majority of recent pronoun resolution systems base their approach around using the representations learned by contemporary transformer language models-for example, in the zero pronoun anaphora resolution literature alone, researchers have used pretrained transformers such as monolingual @xcite @xcite @xcite @xcite and multilingual @xcite bert models @xcite , as well as xlm-r @xcite ; for an example see @xcite . for these systems, there is a clear way to test for human-like-ness. we can directly probe the extent to which the representations learned by the language models take into account the factors that lead to coreference expectations in humans by testing how similar the predictions of language models are to those of human comprehenders-if they exhibit the same pattern of predictive behavior as humans in a given context, this demonstrates that they are sensitive to the same factors as humans in this context. we do this by comparing the reading times reported by @xcite to the surprisals of 12 contemporary transformer language models @xcite @xcite @xcite @xcite for the same stimuli. a key question is whether it is possible for language models to learn any of these human-like expectations at all, given that they can only rely on the statistics of language. for this reason, the results of the present study should be of interest both from a natural language understanding perspective, as discussed above, and also from a psycholinguistics perspective. from the natural language understanding perspective, the present study presents an approach for 'pre-evaluating' a language model's suitability as a basis for a zero anaphora resolution system. specifically, if a language model can model a specific effect in human language processing-that is, if an experimental manipulation that elicits a significant difference in reading time also results in a significant difference in that language model's surprisal in the same direction-this demonstrates that it is able to take into account the relevant factors that underlie human comprehender expectation. for example, if a language model can successfully model the subject antecedent preference, this suggests that it has learned that all else being equal, subject antecedents are more likely to be the coreferents of zero subject pronouns, and thus, crucially, that this pattern is in some way represented in the contextual embeddings that can be used as the representations underlying a zero anaphora resolution system. from the psycholinguistics perspective, this study explores the extent to which it is possible that specific patterns in zero anaphora coreference expectations can be learned on the basis of the statistics of language alone. there is substantial work demonstrating that some expectations are highly correlated with language statistics, and thus may be at least partly derived from them @xcite @xcite @xcite . however, other work has suggested that coreference expectations are instead (or in addition) at least partly based on semantic knowledge, world experience, and conceptual salience @xcite @xcite @xcite . nonetheless, since the predictions of language models are derived from language statistics alone, if even one language model can successfully model a given effect (after adjusting for multiple comparisons), this provides in-principle evidence that the effect can be successfully learned using distributional information alone. ## general method the experiments reported by @xcite were self-paced reading experiments. participants were native speakers of italian asked to read italian sentences on a computer. stimuli were similar to those discussed in the previous section, with a subordinate clause (e.g. quando maria ha chiamato mario; 'when maria called mario') first presented, followed by the main clause (e.g. either era contenta '[she] was happy' or era contento '[he] was happy'). the time taken by participants to read the main clause-which includes the word that disambiguates the null subject pronoun-was recorded. to measure the language model's expectations, we used surprisal (negative log-probability) based on a large body of evidence that language model surprisal generally correlates well with reading time (see, e.g. @xcite @xcite and other metrics of processing difficulty that are thought to correlate with human expectations such as the neural n400 response @xcite @xcite . to model each effect, we compared whether specific linguistic features of the stimuli that elicited a significant difference in human reading times also led to a significant difference in language model surprisal. for example, we investigate whether, like reading time, surprisal is significantly lower when the referent of a zero subject pronoun is a subject antecedent compared to an object antecedent, among other patterns in reading time reported by @xcite . the language models were all presented with the same stimuli as the human participants, which are provided by @xcite in an appendix to the original paper. to match reading time, surprisal was calculated over the whole of the main clause in each stimulus item. this was done by calculating the sum of the surprisals of the main clauses' constituent words, which is equivalent to taking the negative logarithm of the product of their probabilities. we ran the stimuli through 12 transformer language models-5 monolingual and 7 multilingual. two of the monolingual models were autoregressive transformer networks: geppetto @xcite and the small english gpt-2 retrained on italian (de vries and nissim, 2021). the three remaining monolingual models were masked language models: umberto @xcite trained on the italian subcorpus of oscar, and the base and xxl versions of the italian bert models @xcite . the multilingual models also included autoregressive and masked language models. the autoregressive models were three different sizes of xglm @xcite : the 2.9b, 4.5b, and 7.5b parameter models. the masked language models were xlm-100 @xcite , and the base and large versions of xlm-r @xcite . the aim in using this range of models was to test whether there are any model types or characteristics made them better suited to capturing human behavior-for instance, whether the models were autoregressive or masked, or monolingual or multilingual. previous systems designed to resolve zero pronoun anaphora of the kind described here appear to be predominantly based on masked language models; however, autoregressive models such as gpt-2 have been successfully used in similar systems @xcite . we are also interested in whether monolingual or multilingual models are better suited to the task of zero pronoun anaphora resolution-while crosslingual transfer may help with some phenomena @xcite , there is also evidence that it can cause harm to model performance in others @xcite . there is currently mixed evidence with respect to zero pronoun anaphora resolution- @xcite , for example, find that a monolingual korean bert-based model performs better than the standard multilingual bert model; while @xcite finds that their model, based on xlm-r, is better than a model based on a chinese-only bert @xcite . we include both multilingual bert and xlm-r in our analyses, in addition to the base italian bert model, which has previously been evaluated in terms of its capacity to learn non-anaphoric null subject and agreement phenomena in italian @xcite . to test whether each model successfully modeled each effect, we constructed linear mixedeffects models predicting model surprisal with experimental manipulation as a main effect and a random intercept of sentence frame, where sentence frame refers to a set of stimuli that differ only by experimental condition (e.g., the previously discussed quando maria ha chiamato mario, era contenta and quando maria ha chiamato mario, era contento are two stimuli with the same sentence frame). for three of the five analyses-the two analyses in section 3.1 where the coreferent is distinguished by gender, and the analysis in section 3.2-we tested whether the relevant experimental manipulation was a significant predictor of language model surprisal by constructing a null regression with only the random intercept of sentence frame and running a likelihood ratio test investigating whether adding the experimental manipulation improved model fit. the remaining two analyses correspond to two different tests utilized by @xcite to analyze the results of a single experiment (experiment 4 of the original paper). crucially, @xcite tests whether there is an interaction between coreferent argument (whether it is the antecedent subject or object) and coreferent person (whether the coreferent is in the first or second person or in the third person), but also whether there is a main effect of each of these. to test whether there is an interaction (in section 3.3), we construct a linear mixed-effects model with and without the interaction, and run a likelihood ratio test comparing the two. in addition to the interaction, carminati (2005) finds a main effect of coreferent argument but not of person. thus, we also test for the main effect of coreferent argument, which we report in section 3.1. because we want to investigate whether the main effect of coreferent argument explains a significant amount of the variance in surprisal while also accounting for the effect of a possible interaction, instead of using a likelihood ratio test, we opt for a type iii anova with satterthwaite's method for estimating degrees of freedom @xcite . the details of the results of the statistical analyses that were run by @xcite are pro-vided in the original paper. the full results of the statistical analyses that we ran are provided in appendix a. the results of both sets of statistical analyses are summarized in figure 1 . all language models were run in python (van rossum and drake, 2009), using the pytorch @xcite implementation of each model, as provided by the transformers package @xcite . statistical analysis and data manipulation were carried out in r (r core team, 2020) using rstudio (rstudio team, 2020) and the tidyverse @xcite , lme4 @xcite , lmertest @xcite , ggsignif @xcite , and cowplot (wilke, 2020) packages. the stimuli, code used to run the models, and code used to run the statistical analyses are provided on github foot_0 . note that all p-values reported in this analysis have been corrected for multiple comparisons @xcite . ## manipulation-level results and discussion in this section, we compare the performance of the 12 language models tested with human behavior on five of the experimental manipulations carried out by @xcite . note that two additional studies from that paper focus on a different question-the effects of distractor referents on processing time. although at least one model was able to capture each of these human results, they are not included here because they address a different set of phenomena. ## conclusion we present the first study investigating whether language models make the same predictions as humans when processing zero pronoun anaphora. for each the 5 effects we investigate, we find that there are at least four models that successfully do so; and three models, xglm 2.9b, 4.5b, and 7.5b, successfully do so in all 5. this suggests that human processing of zero pronoun anaphora may at least partly rely on our statistical knowledge of language. furthermore, this approach provides a useful way to investigate how human-like the referent predictions of language models are, which is vital if we are to use their representations for zero anaphora resolution systems.