Models of reference production: How do they withstand the test of time? (2023)

Abstract: In recent years, many NLP studies have focused solely on performance improvement. In this work, we focus on the linguistic and scientific aspects of NLP. We use the task of generating referring expressions in context (REG-in-context) as a case study and start our analysis from GREC, a comprehensive set of shared tasks in English that addressed this topic over a decade ago. We ask what the performance of models would be if we assessed them (1) on more realistic datasets, and (2) using more advanced methods. We test the models using different evaluation metrics and feature selection experiments. We conclude that GREC can no longer be regarded as offering a reliable assessment of models’ ability to mimic human reference production, because the results are highly impacted by the choice of corpus and evaluation metrics. Our results also suggest that pre-trained language models are less dependent on the choice of corpus than classic Machine Learning models, and therefore make more robust class predictions.

## introduction nlp research can have different aims. some nlp research focuses on developing new algorithms or building practical nlp applications. another line of nlp work constructs computational models that aim to explain human language and language use; this line of work has been dubbed nlp-as-science @xcite . among other things, nlp-as-science demands that we ask ourselves to what extent nlp research findings generalise along a range of dimensions. in addition to the practical applications of referring expression generation @xcite , reg is also one of the typical tasks in nlpas-science, where reg algorithms are built to model and explain the reference production of human beings @xcite . in the computational linguis-tics and cognitive science community, reg can be divided into two distinct tasks: one-shot reg, finding a referring expression (re) to single out a referent from a set, and reg-in-context, generating an re to refer to a referent at a given point in a discourse. in a classic setup, reg-in-context is often approached in two steps: the first is to decide on the form of an re at a given point in the discourse, and the second is to decide on its content. many researchers have been interested in the first subtask, referential form selection: the task to decide which referential form (e.g., pronoun, proper name, description, etc.) an re takes @xcite @xcite . nearly 15 years ago, @xcite introduced the grec shared tasks and a number of english reg corpora with two goals: (1) assessing the performance of computational models of reference production @xcite , and (2) understanding the contribution of linguistically-inspired factors to the choice of referential form @xcite @xcite . to this end, we pursue the following objectives: (1) we extend grec by testing its reg algorithms not only on the grec corpora but also on a corpus that was not originally considered and that has a different genre, namely the wall street journal (wsj) portion of ontonotes @xcite ; (2) we fine-tune pretrained language models on the task of reg-incontext and assess them in the grec framework. in section 2, we detail the grec shared tasks and introduce the corpora used in grec. section 3 spells out our research questions. in section 4 and section 5, we introduce the algorithms and corpora that we use. section 6 reports the performance of each algorithm on each corpus, followed by analyses in section 7. section 8 will discuss our findings and draw some lessons. ## the grec shared tasks in this section, we summarise the grec task, the corpora used by grec, and its conclusions. ## reg algorithms in what follows, we introduce the reg algorithms that are considered in this study. ## reg corpora in the following, we explain the corpora used in this work. these corpora are english-language corpora. ## evaluation in this section, we introduce the evaluation protocol and report the performance of the models. ## analysis to further compare the different models and investigate the impact of the choice of corpus, we conduct (1) a bayes factor (bf) analysis to determine whether the accuracy rates reported in section 6 come from similar or different distributions, (2) a per-class evaluation of predictions to assess the success of each model in predicting individual classes, (3) a correlation analysis to quantify how the evaluation results change with respect to the choice of a corpus, and (4) a feature selection study to check how the importance of each feature changes as a function of the choice of corpus. ## discussion in this paper, we have conducted a series of reproductions, evaluations, and analyses to check whether the conclusions of grec are still true after 15 years. below, we summarise and discuss our findings in accordance with our three research questions in section 3. we also report our post-hoc observations on the choice of evaluation metric. ## conclusion we are now in a position to address the question that we raised in the introduction: can the conclusions from the grec shared tasks still be trusted? by examining a wider class of corpora, models, and evaluation metrics than before, we found that the answer to this question is essentially negative since the grec conclusions are prone to drastic change once a different corpus or a different metric is employed. perhaps this should come as no surprise. according to a widely accepted view of scientific progress (e.g., @xcite ; applied to nlp in (van deemter, 2023)), theories should be updated again and again in light of new data (i.e., indirect support), and when new models are proposed, the plausibility of existing models should be compared against the plausibility of these new models (as well as pre-existing ones). new metrics deserve a place in this story as well, even though they are often overlooked. in other words, what we have seen in the present study is nothing more than science in progress -something we are bound to see more of as the enterprise called nlp-as-science matures. ethics statement: regarding potential biases, in addition to the biases present in text-based datasets, biases can also be introduced by the pre-trained language models @xcite used in this work. in other words, the reg algorithms we developed in this study may make different predictions with respect to different genders, for instance. in the future, we plan to investigate this phenomenon and find ways to mitigate it.