Computational Interpretations of Recency for the Choice of Referring Expressions in Discourse (2020)

Abstract: First, we discuss the most common linguistic perspectives on the concept of recency and propose a taxonomy of recency metrics employed in Machine Learning studies for choosing the form of referring expressions in discourse context. We then report on a Multi-Layer Perceptron study and a Sequential Forward Search experiment, followed by Bayes Factor analysis of the outcomes. The results suggest that recency metrics counting paragraphs and sentences contribute to referential choice prediction more than other recency-related metrics. Based on the results of our analysis, we argue that, sensitivity to discourse structure is important for recency metrics used in determining referring expression forms.

## introduction speakers use various linguistic forms such as pronouns, proper names, and common nouns, to refer to entities in discourse. a great number of studies have addressed the issue of referring, and the factors that play a role in speakers' choice of the form of referring expressions. these factors include grammatical function @xcite , animacy @xcite , competition @xcite , frequency @xcite and recency @xcite , among others. the focus of this article is on recency. broadly speaking, we understand recency to be the distance between the current mention of a referent and its antecedent. therefore, in this work, we employ recency metrics to predict the form of subsequent mentions, and are not interested in the choice of "first-mention" expressions. recency has received much attention in both linguistic and computational studies, but in many cases, the notion of recency itself has been left largely undefined even though, as we shall see, recency can be understood in different ways. this paper has three objectives. the first is to survey different computational "interpretations" of the notion of recency. the second goal is to determine which of these computational interpretations is most effective for predicting the form of a referring expression in discourse context. in other words, we will ask, "what is the best way to operationalize the notion of recency in computational and data-oriented studies?" and the final objective is to see to which extent the choice of recency metrics should depend on the corpus. the structure of this paper is as follows: in section 2, we summarize how recency has been used in linguistic studies. in section 3, we provide a brief overview of the notion of recency in machine learning (ml) studies, with the purpose of creating a taxonomy of recency metrics discussed in section 4. sections 5 and 6 report two new studies. the former analyzes single recency metrics, the latter takes their combination into account. finally, section 7 gives a brief summary and review of the findings. ## different interpretations of the notion of recency/distance there is a long tradition of work in linguistics considering recency as a factor influencing the salience of a referent. the general idea is that the greater the distance between the two mentions, the greater the chance of using a full noun phrase anaphor @xcite @xcite ; conversely, the shorter the distance between the two mentions, the greater the chance of pronominalization. some studies have kept the notion of recency or "distance to the previous mention" opaque by not defining what long and short distance mean; while others have presented different interpretations of the notion of distance. in this paper, we focus on the three most frequent interpretations that are found in the literature. ## recency in ml studies within natural language generation @xcite , reference production is computationally modelled in an area known as referring expression generation (reg) @xcite . reg models have various shapes and forms, with feature-based ml models playing a substantial role. grec @xcite was a series of shared task evaluation tasks that is still regarded as a natural starting point when it comes to the generation of referring expressions in context. different ml algorithms were submitted to these shared tasks, a number of which have exploited recency metrics. some of the metrics used in these algorithms are pursuant to the interpretations mentioned in section 2. for example, the recency feature in @xcite resembles the metric defined in @xcite . another example is a binary feature used by @xcite , which captures whether or not the antecedent occurs in the same sentence. this metric is similar to the interpretation discussed above under the heading "immediate context". some of the other recency metrics used in these algorithms, however, are not in accordance with the interpretations introduced in section 2. for instance, @xcite and @xcite used distance metrics measuring number of words between the two mentions. in a more recent ml study, @xcite stated that referential choice belongs to a large group of multifactorial processes. they used 7 different distance-related metrics in their study and concluded that these metrics are essential for successful prediction of referential choice, but there is no indication which metrics are the most relevant ones. further studies that include recency metrics are @xcite , @xcite and @xcite , among others. we saw that the metrics used in the ml studies are based on different units of measurement (e.g. word distance versus sentence distance). likewise, different strategies are used to encode these metrics. for instance, some distances are measured in natural numbers while others are categorized in a smaller class of broader "bins". in the following example taken from the grec-2.0 corpus @xcite , one could say that the distance between the expression "its" and its antecedent "berlin" is 21 words (a natural number). another solution would be, for instance, to follow ferreira et al. (2016) in grouping the numerical distances into five groups consisting of 0-10 words, 11-20 words, 21-30 words, 31-40 words and more than 40 words. with this approach, the distance between "its" and its antecedent falls into the third bin, 21-30 words. ## methodology this section begins with subsection 4.1 introducing recency metrics collected from different ml studies. later, subsection 4.2 presents the two corpora used in our assessments and highlights their main differences. and finally, subsection 4.3 introduces the baseline algorithm and the ml method employed in our assessments. ## assessing recency metrics using mlp this section firstly reports on the success of the baseline algorithms, and continues with the algorithms incorporating the recency metrics. ## sequential forward search in order to investigate the extent to which the combination of different recency metrics improves the performance, we run a sequential forward search (sfs) algorithm. the algorithm starts with an empty set and adds features to the model up to the point that no further improvement occurs. for this study, we used the r package mlr @xcite with the learner classif.mlp, and 5-fold cross-validation resampling strategy. the result of the msr experiment shows that the two recency metrics playing the most important roles are metric 15, distance in paragraph, and metric 9, exponential distance in sentences. retraining the mlp algorithm on the new model, the accuracy is 0.637. the bayes factor analysis provides strong evidence that the outcome of this model is statistically different from the baseline (@xmath0.11). in the wsj sfs experiment, metric 15, distance in paragraphs, and metric 8, log distance in sentences, were chosen as the two recency features whose combination produced the best result. the model trained on the combination of these two metrics had the accuracy of 0.631. the bayes factor analysis finds very strong evidence that the outcomes of the baseline and this model are coming from different distributions. what stands out in this experiment is that in the case of both msr and wsj, distance in paragraph is chosen as one of the recency metrics. the other chosen measures are exponential distance in msr and logarithmic distance in wsj. this could indicate that the algorithm is sensitive to the encoding of the sentence-based metrics. more experimentation in a more elaborated feature-based study is necessary to test this point. ## conclusion our goal was to shed light on different interpretations of recency, and to find out which of these interpretations are most effective for referential choice prediction. a subsidiary goal was to investigate whether the choice of recency metric should take corpus-specific features such as text genre and text structure into consideration. the findings of this study should be of interest to theoretical and computational linguists alike, because both groups of researchers have studied the relation between recency and referential choice. in the linguistic tradition, the notion of recency has often been studied without a clear definition being offered (section 2). in the computational tradition, by contrast, researchers have dwelt less on theoretical justification but have had to provide precise definitions, to ensure that their algorithms are able to deal with a broad range of inputs. for example, @xcite defined 7 different implementations of the notion of recency taking different units of measurement into account; while @xcite employed various implementations of sentence-related metrics. another difference is that in the linguistic tradition, researchers usually think of recency as operating solely on the sentence or paragraph levels; while in computational works, less conventional metrics such as measuring the distance in words or nps have been also practiced. we believe that the existence of a wider range of recency metrics in computational feature-based studies has the potential to open new windows into a better understanding of recency, and can encourage a re-evaluation of recency in the linguistic tradition. what is missing from many computational works is an explanation of why a certain metric or a certain way of encoding has been chosen over another. the findings from this study make the following contributions to the literature: creating a taxonomy of recency metrics after providing an overview of the most prevalent interpretations of recency in the linguistic tradition, we scrutinized the feature-based ml studies and provided, for the first time as far as we know, a taxonomy of recency metrics. the importance of this taxonomy is firstly that we do not know of any available work classifying and analyzing this notion comprehensively, so this work could be a starting point for getting deeper into the notion of recency. secondly, we have shed light on the differences between these metrics. knowing what the differences are, and where they stem from, could be the first step in dissecting various aspects of this notion and developing new, improved recency metrics.