A Linguistic Perspective on Reference: Choosing a Feature Set for Generating Referring Expressions in Context (2020)

Abstract: This paper reports on a structured evaluation of feature-based Machine Learning algorithms for selecting the form of a referring expression in discourse context. Based on this evaluation, we selected seven feature sets from the literature, amounting to 65 distinct linguistic features. The features were then grouped into 9 broad classes. After building Random Forest models, we used Feature Importance Ranking and Sequential Forward Search methods to assess the “importance” of the features. Combining the results of the two methods, we propose a consensus feature set. The 6 features in our consensus set come from 4 different classes, namely grammatical role, inherent features of the referent, antecedent form and recency.

## introduction various studies have raised the question of which factors play a role in the choice of referring expressions. one of the main ideas in this tradition (henceforth, the linguistic tradition) is that there is a direct relationship between the "prominence" (in a broad sense) of a referent at a given point in the discourse, and the form used to refer to it. if a referent is prominent, a short anaphoric form (e.g. a pronoun) suffices; if it is less prominent, longer forms with more semantic content are used. prominence has been argued to be influenced by various factors such as recency and frequency of mention @xcite , grammatical function @xcite , distance @xcite , animacy @xcite , and competition between the referents @xcite . reference production is also one of the most-studied topics in natural language generation @xcite , where it is known as referring expression generation @xcite . a key part of the reg problem is deciding which form (e.g., proper name, definite description or pronoun) to employ to refer to a referent at a given point in the discourse. henceforth, we call this task selection of referential form (srf). srf models come in many shapes and forms, with feature-based machine learning (ml) models playing a dominant role. however, the feature sets employed by these models can differ considerably from one model to the next, and although features akin to the ones employed in the linguistic tradition are often used, other types of features, which are harder to interpret linguistically, are frequent as well. our aim in this paper is to examine feature-based srf models from a linguistic perspective. we will conduct a systematic evaluation of these models, asking what features make them work best. having done this, we propose a "consensus" feature set. finally, we compare the features in our consensus feature set against the factors considered to be important in the linguistic tradition. an important question in any systematic evaluation is how the objects of study (in our case, srf feature sets) are selected. we have proceeded as follows: note that our systematic evaluation does not include deep learning methods (e.g., castro ferreira et al. (2018); cao and cheung (2019)) since, at the current state of the art, these do not yet offer much opportunity for linguistic interpretation. our focus is on interpretable linguistic features. to perform our evaluation, two further choices need to be explained and motivated: since the srf task as such can be defined in different ways, we had to define an exact task, and we had to specify a corpus. these two choices will be further explained in section 3. section 4 details the feature selection experiments and propose a consensus feature set. the paper concludes with a discussion of the extent to which the features are linguistically interpretable. ## related work data-driven models employ different features to make choices similar to those made by humans. for instance, @xcite used linguistically informed features such as recency, subjecthood, parallelism and ambiguity. in a comprehensive study, @xcite argued that reference generation is a multifactorial process governed by various linguistically motivated features. @xcite , on the other hand, focused more on existing patterns in the text and less on the use of linguistic categories. in a more recent study @xcite used features marking the syntactic position, recency and referential status of the referents to model the referential choice variations. in one of the few studies targeting feature selection, @xcite surveyed the psycholinguistics literature to decide on feature sets for their srf study. they implemented several linguistically informed rules in their prototyping system, and "examined the incorrect classifications which resulted in an attempt to discover which other factors suggested by psycholinguistic research could explain the patterns they observed." additionally, they incorporated some of the features used in @xcite and created an extensive set of features which they felt had an impact on the referential choice. afterwards, they trained c5.0 decision trees on 5 subsets of these features. the interesting result of their study is that the maximum number of features does not necessarily lead to best results. the problem with their study is that firstly, they do not propose any explanation on how they have selected different subsets. secondly, their feature selection strategy is subjective, since it mostly concerns with the features they find important, and has less to do with the features of other models. lastly, they have provided no linguistic explanation for their best performing system. @xcite also briefly talked about the feature selection in their study, highlighting the importance of recency-related features. although their study is linguistically informed, the annotation effort behind it is very intense. the question is whether they could achieve a nearly similar performance using a smaller set of features. ## feature selection experiments for assessing the features we start by briefly explaining the classification algorithm and the feature selection methods we use in our experiments (subsection 4.1). afterwards, we elaborate on the classification models trained on the ontonotes data using the proposed feature sets (subsection 4.2). the section continues with two feature selection experiments with which we assess the importance of the features (subsection 4.3 and subsection 4.4). the next step is to use different subsets of the features and to re-run the classification algorithms, to see how this will affect the accuracy of the models. ## conclusion the aim of this study was to systematically examine feature-based srf models, fleshing out what features make the models work best. by evaluating different feature sets of the computational srf studies from a linguistic perspective, we tried to build a bridge between the features employed by computational models and the linguistic explanations behind those features. based on the results of the two feature selection experiments conducted on seven feature sets, and the approach outlined in section 4.5.2, we composed a consensus set consisting of six features from 4 classes: animacy and plurality [inherent features of a referent], grammatical role of the current mention, form of the antecedent, and categorical distance in number of sentences @xcite . comparing the consensus set with the previously proposed feature sets has interesting implications for both feature-based srf research and linguistics. implications for feature-based srf studies: we found that by using a smaller set of features, models can often achieve nearly identical performance. for example, as shown in we saw in implications for linguistics: in section 3.1, we grouped the features that were used by the systems that we studied into 9 broad categories. the 6 features that we chose (above) as our consensus set are all from only 4 of these 9, namely grammatical role, inherent features of the referent, antecedent form and recency. this concludes our comparison of different feature-based computational studies of referential choice. in view of the above observations, we hope that computational and theoretical studies of language will continue to provide inspiration to each other.