Does referent predictability affect the choice of referential form? A computational approach using masked coreference resolution (2021)

Abstract: It is often posited that more predictable parts of a speaker’s meaning tend to be made less explicit, for instance using shorter, less informative words. Studying these dynamics in the domain of referring expressions has proven difficult, with existing studies, both psycholinguistic and corpus-based, providing contradictory results. We test the hypothesis that speakers produce less informative referring expressions (e.g., pronouns vs. full noun phrases) when the context is more informative about the referent, using novel computational estimates of referent predictability. We obtain these estimates training an existing coreference resolution system for English on a new task, masked coreference resolution, giving us a probability distribution over referents that is conditioned on the context but not the referring expression. The resulting system retains standard coreference resolution performance while yielding a better estimate of human-derived referent predictability than previous attempts. A statistical analysis of the relationship between model output and mention form supports the hypothesis that predictability affects the form of a mention, both its morphosyntactic type and its length.

## introduction a long-standing hypothesis in linguistics relates predictability to linguistic form: the more predictable parts of a speaker's meaning tend to be communicated with shorter, less informative words and be pronounced more quickly and flatly @xcite @xcite . this is posited to result from a trade-off between clarity and cost: when the context already clearly conveys certain information, making it explicit would be inefficient. in the domain of referring expressions, or mentions, the prediction is that pronouns such as "they" and "her" are used for more * first two authors contributed equally. existing work on the relation between predictability and mention form paints a mixed picture (section 2). a set of studies in psycholinguistics used controlled stimuli designed to manipulate predictability @xcite ). an alternative approach uses naturally occurring corpus data to elicit judgments from human subjects @xcite . neither approach, however, scales well. we instead obtain predictability estimates from a coreference system (trained on english data); if successful, this strategy would allow to test the hypothesis in a wider set of contexts than in psycholinguistic experiments. this paper follows the long tradition of using computational models trained on large amounts of data as proxies for different aspects of human cognition; in particular, it extends to the referential level previous research that uses computational models to obtain predictability scores @xcite . standard coreference models, however, cannot be directly used to model predictability, because they are trained with access to both the context of a referring expression and the expression itself. instead, we aim at obtaining predictability scores that are only conditioned on the context, following the definition of predictability used in psycholinguistics. to this end, we minimally modify a state-of-the-art coreference system @xcite to also carry out what we call masked coreference resolution (figure 1 ): computing referent probabilities without observing the target mention. we show that the resulting model retains standard coreference resolution performance, while yielding a better estimate of human-derived referent predictability than previous attempts. statistical analysis of the relationship between model scores and mention form suggests that predictability is indeed linked to referential form. low referent surprisal tends to be associated with pronouns (as opposed to proper names and full noun phrases) and in general shorter expressions. when controlling for shallow cues that modulate the salience of an entity (e.g., recency, frequency), surprisal no longer differentiates pronouns from proper names (while still favoring longer noun phrases). this may be read as supporting the hypothesis that it is primarily mention length that is at stake, rather than morphosyntactic category, but it may also be due to stylistic factors in the data we use @xcite . 1 2 related work predictability and referential form. traditional approaches to discourse anaphora posit that the salience of referents determines the choice of referring expression (a.o., @xcite . salience, however, is notoriously hard to characterize and operationalize. later work has therefore focused on predictability, that is, how expected a specific referent is at a certain point in the discourse (see arnold and zerkle 2019 for an overview). this notion can be more easily operationalized to gather judgements from human subjects. previous work has done so using cloze tasks and discourse continuation experiments. some of this work found that more pronouns were produced for more predictable referents, suggesting an addressee-oriented strategy for pronoun production (e.g., @xcite . other work instead reported the same pronominalization rates in contexts that favored different referents, and argued for a mismatch between speaker and listener strategies (e.g., @xcite . while the aforementioned psycholinguistic studies used tightly controlled stimuli, researchers also reported contradictory results when looking at cloze task responses in corpus data. @xcite considered newspaper texts from the ontonotes corpus and found that pronouns and proper names were favoured over full nps when subjects were able to guess the upcoming referent. by contrast, @xcite did not find this ef-fect on narrative texts in the inscript corpus @xcite . computational estimates of predictability. probabilistic language models have been commonly adopted to study expectation-based human language comprehension @xcite . predictability scores obtained from language models have been shown to correlate with measures of cognitive cost at the lexical and the syntactic levels (smith and @xcite . in this work, predictability is typically measured with solely the preceding context in computational psycholinguistics (e.g., @xcite . however, more recent work has also looked at predictability calculated based on both the previous and following contexts using bidirectional networks, like we do: pimentel et al. (2020) used surprisal calculated from a cloze language model which takes both left and right pieces of context, as the operationalization of word predictability. they studied the trade-off between clarity and cost and reported a tendency for ambiguous words to appear in highly informative contexts. previous work also used computational estimates of referent predictability. in @xcite , they were used to explain referential choice as the result of an addressee-oriented strategy. their measures of predictability, however, were based on simple features like frequency or recency. @xcite built upcoming referent prediction models combining shallow linguistic features and script knowledge. this approach allowed them to disentangle the role of linguistic and common-sense knowledge, respectively, on human referent predictions. more recent research assessed the ability of autoregressive language models to mimick referential expectation biases that humans have shown in the psycholinguistic literature (e.g., @xcite . @xcite extended this assessment to non-autoregressive models, like the ones we use here, and reported results consistent with prior work in autoregressive models, showing that these models exhibited biases in line with existing evidence on human behavior, at least for english. ## evaluation results by mention type. figure 2 breaks down the antecedent precision scores of m m by mention type. from now on we look only at the setup with gold mention boundaries, though the trends are the same for predicted mentions (reported in appendix b). we distinguish between proper names (e.g., "kamala harris"), full noun phrases (nps; e.g., "the tall tree") and pronouns (e.g., "she", "my", "that"). for completeness, in appendix b we report the results considering a more fine-grained distinction. the figure shows that for predicting the antecedent of masked mentions, pronouns are the easiest (.81), followed by proper names (.71), and full nps (.66) are the hardest. put differently, pronouns are used in places where the referent is the most predictable, full nps when the referent is the least predictable. for unmasked mentions, instead, proper names are the easiest (.96; names are typically very informative of the intended referent), and full nps (.89) are only slightly more difficult than pronouns (.92). hence, the pattern we see for masked mentions cannot be a mere side-effect of pronouns being easier to resolve in general (also when unmasked), which does not seem to be the case. instead, it provides initial evidence for the expected relation between referent predictability and mention choice, which we will investigate more in the next section. comparison to human predictions. we assess how human-like our model m m behaves by comparing its outputs to human guesses in the clozetask data from @xcite . subjects were asked to guess the antecedent of a masked mention in narrative stories while seeing only the left context (182 stories, ∼3k mentions with 20 guesses each). to evaluate the model's estimates, we follow modi et al.'s approach, and compute jensen-shannon divergence to measure the dissimilarity between a model's output and the human distribution over guessed referents (the lower the better). m m achieves a divergence of .46, better than modi et al.'s best model (.50), indicating that our system better approximates human expectations. appendix b provides further results and details. ## predictability and mention form the previous section assessed the effect of our masked training method on model quality. we believe that the model predictions are of high quality enough that we can use them to test the main hypothesis regarding the relation between predictability and mention choice. following previous work (see section 2), we define predictability in terms of the information-theoretic notion of surprisal: the more predictable an event, the lower our surprisal when it actually occurs. given a masked mention x with its true referent e true , surprisal is computed from the model's output probability distribution over entities e x (eq. 3), given the context c x : surprisal ranges from 0 (if the probability assigned to the correct entity equals 1) to infinity (if this probability equals 0). surprisal depends only on the probability assigned to the correct entity, regardless of the level of uncertainty between the competitors. as @xcite note, uncertainty between competitors is expected to be relevant for mention choice, e.g., a pronoun may be safely used if no competitors are likely, but risks being ambiguous if a second entity is somewhat likely. @xcite and, following them, @xcite took this uncertainty into account in terms of entropy, i.e., expected surprisal. we report our analyses using entropy in appendix c, for reasons of space and because they support essentially the same conclusions as the analyses using just surprisal. we check whether surprisal predicts mention type (section 5.1) and whether it predicts mention length (number of tokens; section 5.2). all analyses in this section use the probabilities computed by m m with gold mention boundaries. ## discussion and conclusion in this work, we studied the relationship between referent predictability and the choice of referring expression using computational estimates of the former. to derive these, we adapted an existing coreference resolution system to operate in a setup resembling those of cloze tasks employed in psycholinguistics. using computational estimates of semantic expectations allowed us to scale and expedite analyses on a large dataset, spanning different genres and domains. our results point to a trade-off between clarity and cost, whereby shorter and possibly more ambiguous expressions are used in contexts where the referent is more predictable. we found this both when grouping mentions by their morphosyntactic type and by their length. referent predictability seems to play a partially overlapping but complementary role on referential choice with features affecting the salience of an entity, such as its recency, frequency or whether it was last mentioned as a subject. this points to the open question as to whether salience can actually be reduced to predictability @xcite . our bidirectional setup is not directly comparable to that of some of the related work as to the amount and scope of context given for prediction. referents are predicted with only the preceding context in previous work, both in psycholinguistic and computational approaches, while our model gives predictions based on both the preceding and following contexts. if one important hypothesis both previous studies and our study aim at testing is that speakers tend to avoid the redundancies between the informativeness of context and that of referring expression, our results then point to an issue that merits more attention: what kind of context influences referential choice? is it only the preceding one, or the following one? how much (on either side)? @xcite raised a similar question concerning word intelligibility in sentences and found that participants delayed the decision about a test word presented in noise until the entire sentence was encoded, and that the context after the target word was more facilitating to its intelligibility. song and kaiser (2020) also showed that comprenhenders actively utilized postpronominal information in pronoun resolution. the use of a computational model provides flexibility to compare predictions using different amounts of context, and could shed light on how the previous and following context affect mention choice. future work could also use unidirectional models, which allow for a setup more like the one adopted by prior work for ease of comparison, if requirements on the quality of performance can be met. we hope that our work will foster the use of computational models in the study of referential choice. our methodology can be applied to more languages besides english (provided the availability of coreference resources; for instance, arabic and chinese are included in ontonotes 5.0) and the study of phenomena beyond those considered here. relevant future venues are more fine-grained classifications of nps (such as indefinite vs. definite), the effect of referent predictability on processing @xcite , and the kinds of context examined in psycholinguistic experiments (e.g., different rhetorical relations, verbs with contrasting referent biases; rohde and kehler 2014; mayol 2018). ralph weischedel, martha palmer, mitchell marcus, eduard hovy, sameer pradhan, lance ramshaw, nianwen xue, ann taylor, jeff kaufman, michelle franchini, mohammed el-bachouti, robert belvin, and ann houston. 2013. ontonotes release 5.0 ldc2013t19. web download.