Intrinsic Task-based Evaluation for Referring Expression Generation (2024)

Abstract: Recently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on WEBNLG, Referring Expressions (REs) generated by the state-of-the-art neural models were not only indistinguishable from the REs in WEBNLG but also from the REs generated by a simple rule-based system. Here, we argue that this limitation could stem from the use of a purely ratings-based human evaluation (which is a common practice in Natural Language Generation). To investigate these issues, we propose an intrinsic task-based evaluation for REG models, in which, in addition to rating the quality of REs, participants were asked to accomplish two meta-level tasks. One of these tasks concerns the referential success of each RE; the other task asks participants to suggest a better alternative for each RE. The outcomes suggest that, in comparison to previous evaluations, the new evaluation protocol assesses the performance of each REG model more comprehensively and makes the participants’ ratings more reliable and discriminable.

## introduction referring expression generation (reg) is a key aspect of natural language generation (nlg) and a vital step in the classic natural language generation pipeline (nlg, reiter and dale, 2000; gatt and krahmer, 2018). reg produces referring expressions (res) that refer to referents at different points in a discourse (belz and varges, 2007). it has great practical value for commercial nlg (reiter, 2017) and is actively studied by theoretical linguists and psycholinguists (van deemter, 2016). conventional reg systems have two steps. the first step determines the rough form of the re. for instance, given a certain context, the system needs to decide whether a given reference to homer simpson should be a proper name, a pronoun, or a description. the second step involves choosing a con-crete noun phrase. for example, if step 1 says a proper name must be chosen, then the second step involves choosing between "homer" or "homer simpson"; if a description must be chosen, then many options exist, including "the protagonist of the american animated sitcom the simpsons". castro ferreira et al. (2018a) redefined the task of reg on the webnlg corpus (gardent et al., 2017a; castro ferreira et al., 2018b), which has been widely used in nlg research, and tackled it in an end2end manner, i.e. addressing the two above-mentioned steps simultaneously using deep neural networks. later, many follow-ups were carried out to strengthen the "neuralreg" model (e.g. cao and cheung (2019); cunha et al. (2020)). nonetheless, in a large-scale human evaluation of neuralreg models, same et al. (2022) surprisingly found that, for human readers, the res generated by neuralreg models are not only indistinguishable from the res in a corpus but also from the res generated by a simple rule-based model. for this reason, they questioned the usefulness of neural models in reg, as well as the suitability of the webnlg corpus as a testing ground for the strengths of the models. an alternative take on these counter-intuitive results is that the method of asking participants to rate models' outputs may not have been sufficiently sensitive. as previous studies have pointed out (thomson et al., 2023; goyal et al., 2023), pragmatic appropriateness is difficult to assess, especially for res, since the use of res is highly context-dependent and diverse. note that previous human evaluations for reg models (castro ferreira et al., 2018a; cunha et al., 2020; same et al., 2022) always asked participants to rate an entire discourse, which may have failed to make participants aware of the use of res (even though their instructions asked them to focus on res). in this paper, we use intrinsic task-based human evaluation. in addition to rating each dis- we believe that this new approach to evaluation has much to add to existing approaches because (1) answering the questions about res and optimising res by hand helps participants zoom in on the use of res in each discourse during the human evaluation, making their score more reliable and discriminable; (2) the participants' responses allow us to evaluate reg models in a more comprehensive way and from multiple angles; and (3) on the basis of rewritings from participants, we can gain further insights about human language use. in the current study, based on the above protocol, we carried out a human evaluation experiment on the models' outputs from same et al. (2022). building on participants' responses, we first analysed the performance of each reg model comprehensively; we then compared our results to those of same et al. (2022) to see how accomplishing the abovementioned tasks has impacted participants' rating behaviour. finally, we looked into the contents of these responses and summarised our observations. ## background first, we explain the reg task and the corpus it is based on. then, we outline the experiments of same et al. (2022). ## research questions as motivated in the introduction, our enhanced experiment rests on three key tasks. the first of these was the same ratings-based task that had been used in earlier studies (including same et al. 2022), namely, to rate the clarity, grammaticality, and coherence of the item in which a given coreference chain occurs (see figure 1 ). the second task was to judge the referential success of the res. the third was for participants to suggest a rewriting for each re wherever they felt it was necessary. @xcite our principal aim was to find out how the outcomes of the new experiment differ from earlier experiments; in particular, we were curious whether the indistinguishability results of same et al. (2022) may have been caused by the limitations of a human evaluation method that is based solely on ratings. since our experiment produced three different ways of assessing the quality of an re (namely its rating, its referential success, and the frequency of rewriting), the question comes up about how these three assessments are related to each other: do they tend to point in the same direction or not? note that even if these assessments proved to be very closely aligned, it would not follow that two of the three are superfluous because the fact that this plurality of tasks forced participants to immerse themselves deeply in the texts may have improved the quality of all their responses, including those for the familiar ratings-based task. finally, by studying the corpus, it occurred to us that referents differed sharply from each other along a potentially important dimension, namely whether a participant was previously familiar with the referent or not. it has been pointed out that, in some settings, both speakers and hearers use/interpret res differently if they are familiar with the referent (kutlak et al., 2011; staliūnaitė et al., 2018). we, therefore, asked participants, for each referent, whether or not they were familiar with this referent before they read the text, and we investigated how participants' familiarity with a referent affected their responses on the key tasks. based on the assumptions mentioned above and the questions raised by these assumptions, we put forward the following hypotheses: hypothesis 1 (h 1 ) the meta-level tasks (i.e. the referential success task and rewritings) help participants make more informed ratings. therefore, unlike the findings by same et al. (2022), we expect to observe significant distinguishable differences in the ratings of the models. hypothesis 2 (h 2 ) since the tasks help participants identify inappropriate res in each text, we expect that the scores in this experiment would be lower than those in same et al. (2022). hypothesis 3 (h 3 ) regarding the referential success of res, we expect that the more frequently res are marked as successful, the higher the discourse would be rated. hypothesis 4 (h 4 ) we expect that the more often the res are re-written, the lower the scores the discourse would receive. ## intrinsic task-based human evaluation here, we describe the setup of the new experiment. we consider this to be an "intrinsic" nlg evaluation, as opposed to the non-real-world ("extrinsic") tasks more commonly associated with task-based nlg evaluation (reiter, 2011). ## results in this section, we introduce the data we obtained from the experiment, explain how we postprocessed the data, and report the results. ## further observations additionally, we made the following observations while annotating the data. ## conclusion focusing on some surprising results from same et al. (2022), namely that reg models are indistinguishable from each other and from the res in webnlg in a rating-based human evaluation, this paper has introduced a new type of intrinsic taskbased reg evaluation. in parallel with rating, we designed two tasks: one asks participants about the referential success of each re, and the other asks participants to suggest a rewriting for each re if possible. in this way, we had a better understanding of reg models' performance from different perspectives. meanwhile, we confirmed that accomplishing these meta-level tasks helped participants rate in a more reliable and discriminable way. this comprehensive evaluation suggests that, on webnlg, the machine learning based reg model performed best of all the models we tested. compared to the rule-based and neural models, it had a remarkably high rate of producing referentially successful res. additionally, it received the best clarity, grammaticality and coherence scores, and its generated content was rewritten the least frequently. same et al. (2022) highlighted the usefulness of non-neural models because the classical rating-based human evaluation suggested that neural and non-neural models perform at comparable levels, at least in the area of referring expression generation. the results from our new intrinsic task-based evaluation reinforce earlier conclusions because they suggest that, in fact, non-neural models outperform neural models in this area. we think our design of the intrinsic task-based human evaluation protocol can serve as a reference for the evaluation of other nlg tasks that are complex enough so that simple evaluation cannot offer all the answers. we also think that the data from the experiment can help linguists understand the use of reference better and help computer scientists build better reg models. in future, on the one hand, we plan to conduct more in-depth analyses of the issues we discussed in section 6, including qualitative and quantitative analyses of the rewritings and the factors that could affect the use of re. on the other hand, our results revealed that neural models are good at processing seen data but not unseen data. this problem might be addressed by the most recent large language models (llms). @xcite for this reason, in future research, we plan to apply mainstream llms to reg and assess these models using the protocol proposed and investigated in this paper.