What can Neural Referential Form Selectors Learn? (2021)

Abstract: Despite achieving encouraging results, neural Referring Expression Generation models are often thought to lack transparency. We probed neural Referential Form Selection (RFS) models to find out to what extent the linguistic features influencing the RE form are learned and captured by state-of-the-art RFS models. The results of 8 probing tasks show that all the defined features were learned to some extent. The probing tasks pertaining to referential status and syntactic position exhibited the highest performance. The lowest performance was achieved by the probing models designed to predict discourse structure properties beyond the sentence level.

## introduction referring expression generation (reg) is one of the main stages of classic natural language generation (nlg) pipeline @xcite @xcite . reg studies are concerned with two different tasks. the goal of the classic reg task (also called oneshot reg), is to find a set of attributes to single out a referent from a set of competing referents. the second reg task (henceforth discourse reg) is concerned with the generation of referring expressions (re) in discourse context. @xcite phrase it as follows: given an intended referent and a discourse context, how do we generate appropriate referential expressions (res) to refer to the referent at different points in the discourse? classic discourse reg was usually understood as a two-step procedure. in the first step, the referential form (rf, i.e, the syntactic type) is determined. for instance, when referring to joe biden at a given point in a discourse, the first step is to decide whether to use a proper name ("joe biden"), a * equal contribution description ("the president of the usa"), a demonstrative ("this person") or a pronoun ("he"). the second step is to determine the re content, that is, to choose between all the different ways in which a given form can be realised. for instance, to generate a description of joe biden, one needs to decide whether to only mention his job (e.g., the president entered the oval office.), or to mention the country as well (e.g., the president of the united states arrived in cornwall for the g7 summit.) in earlier works, computational linguists linked reg to linguistic theories and built discourse reg systems on the basis of linguistic features. for example, @xcite investigated the impact of 3 linguistic features namely recency, subjecthood, and discourse status on pronominalization, i.e. deciding whether the re should be realised as a pronoun. using these features, they used the notion of local focus as a criterion for detecting the set of referents that can be pronominalised. the same holds for feature-based models (see @xcite for an overview) where models are trained on linguistically encoded data. more recently, a number of neural networkbased reg models have been presented @xcite @xcite , where they propose to generate res in an end2end manner without any feature engineering. they all used a benchmark dataset called webnlg. these models generally follow the sequence-to-sequence framework @xcite , where there is an encoder for encoding the given discourse, and a decoder responsible for generating res using the encoded information. the evaluation results suggested that these neural methods perform well not only for selecting the proper rfs, but also for producing fluent res. however, it was unclear to what extent these neural models can encode linguistic features. triples: (awh engineering college, country, india), (kerala, leadername, kochi), (awh engineering college, academicstaffsize, 250), (awh engineering college, state, kerala), (awh engineering college, city, "kuttikkattoor"), (india, river, ganges) text: awh engineering college is in kuttikkattoor, india in the state of kerala. the school has 250 employees and kerala is ruled by kochi. the ganges river is also found in india. delexicialised text: pre-context: awh engineering college is in "kuttikkattoor" , india in the state of kerala . ## neural referential form selection in this section, we define the task of rfs built on the webnlg dataset, and introduce a number of neuralrfs models. ## probing rfs models we use a logistic regression classifier as our probing classifier. concretely, for each input, we first use a model discussed in section 3 to obtain its representation r. as mentioned in section 3, we ran each model five times and reported their averaged scores. for the probing tasks, we use the representations of the models with the best rfs performance on the development set. ## conclusion our aim is to understand whether neural models capture the features associated with the task of rfs. to this end, we defined 8 probing tasks in which we focused on referential status, syntactic position, recency, and discourse structure. the probing results suggest that the probe classifiers always performed better than the random and the majority baselines. the performance was consistently good in the tasks associated with referential status, syntax and local prominence. it is worth noting that probing has its own shortcomings. for instance, on the one hand, low probing performance does not always mean the feature is not encoded, but could also mean that such a feature does not matter to rfs. to mitigate this issue, we conducted a complementary ml-based variable importance analysis; in this analysis, discourse status and syntactic position came out as the factors with the highest contributions. these features were also predicted very well in the probing tasks. however, these results should still be taken with a pinch of salt: the variable importance has been conducted on the ml model and not on the neural models. we cannot be certain that the same features contribute to all the models similarly: a feature might be quite important in the machine learning model, but not as important in the neural models. on the other hand, some researches have questioned the validity of probing methods. they found out that it is difficult to distinguish between "learning the probing task" and "extracting the encoded linguistic information" @xcite for a probing classifier. this suggests that higher performance of a probing classifier does not necessarily mean more linguistic information has been encoded. this prevents us from directly quantifying how well the linguistic information has been learnt using the performance of probing classifiers and requires us to make conclusions more carefully. from our probing efforts, we conclude that: (1) all neural models have learnt some information about the features associated with the probing tasks, but how well they have learnt this information is yet to be assessed; (2) the webnlg corpus, which has often been used for the study of discourse reg, is not ideally suitable for studying discourse-related aspects of rfs, because the texts are too short and the majority of res are first mentions. this leads to bias in the evaluation of rfs and reg algorithms; in future, we plan to extend the current study from three angles. first, we plan to conduct experiments on different corpora. the webnlg corpus used in this study consists predominantly of extremely short documents with an average length of only 1.4 sentences/document; consequently the majority of res are first mentions. we hope to find a more representative distribution of uses of referring expressions in other corpora such as ontonotes @xcite , which contain longer texts. secondly, we plan to conduct experiments on other languages than english, in particular ones that favour zero pronouns (e.g., chinese @xcite ), because these pose new challenges for the task of rfs. thirdly, we plan to design new probing tasks on the basis of other factors that could influence rfs, such as animacy, competition and positional attributes (see @xcite for an overview).