Non-neural Models Matter: a Re-evaluation of Neural Referring Expression Generation Systems (2022)

Abstract: In recent years, neural models have often outperformed rule-based and classic Machine Learning approaches in NLG. These classic approaches are now often disregarded, for example when new neural models are evaluated. We argue that they should not be overlooked, since, for some tasks, well-designed non-neural approaches achieve better performance than neural ones. In this paper, the task of generating referring expressions in linguistic context is used as an example. We examined two very different English datasets (WEBNLG and WSJ), and evaluated each algorithm using both automatic and human evaluations. Overall, the results of these evaluations suggest that rule-based systems with simple rule sets achieve on-par or better performance on both datasets compared to state-of-the-art neural REG systems. In the case of the more realistic dataset, WSJ, a machine learning-based system with well-designed linguistic features performed best. We hope that our work can encourage researchers to consider non-neural models in future.

## introduction natural language generation (nlg) is concerned with the generation of natural language text from non-linguistic input @xcite . one step in a classic generation pipeline @xcite is referring expression generation (reg, @xcite for an overview). reg has important practical value for commercial natural language generation (reiter, 2017), computer vision @xcite , and robotics @xcite , for example. it has also been used as a tool to understand human language use @xcite . reg contains two different problems. one is to find a set of attributes to single out a referent from a set (also called one-shot reg). the other is to generate referring expressions (res) to refer to a referent at different points in a discourse @xcite . we will focus on the latter task. we call this the reg-in-context task. in earlier works, reg is often tackled in two steps @xcite . the first step decides the form of an re. for example, whether a reference should be a proper name ("marie skłodowska-curie"), a description ("the physicist"), or a pronoun ("she") at a given point in the context. the second step is concerned with content selection, i.e., the different ways in which a referential form can be realised. for example, to generate a description of marie curie, the reg system decides whether it is sufficient to mention her profession (i.e., "the physicist") or whether it is better to mention her nationality as well (i.e., "a polish-french physicist"). thanks to the rapid development of deep learning techniques, recent nlg models are able to generate re in an end2end (e2e) manner, i.e., to tackle the selection of form and content simultaneously @xcite @xcite . the task of end2end (e2e) reg was proposed by @xcite , who extracted a corresponding corpus from the webnlg corpus @xcite 1 . grounding on the webnlg dataset, they proposed a neural reg system built on a sequence-to-sequence with attention model. their automatic and human evaluation results suggested that neural reg systems significantly outperform rule-based and feature-based machine learning (ml) baselines. however, it can be argued that castro ferreira et al. did not use very strong baselines for their comparison: onlyname is a rule-based system that always generates a proper name given an entity, and ferreira is a feature-based model that uses naive bayes with only 3 simple features foot_1 . we present several rule-based and feature-based baselines to examine how neural models perform against "well-designed" non-neural alternatives. note that a well-designed model is not necessarily complex. for example, it can be a rule-based system with one or two simple, "well-designed" rules. since one of the advantages of neural e2e models is that they require little effort for feature engineering, we used two types of baselines, namely models that require minimal expert effort and models that use more demanding (but linguistically wellestablished) rules or features. therefore, our main research question is: do state-of-the-art neural reg models always perform better than rule-based and machine learning-based models? to answer this question fairly, we consider the amount of resources used by each model. for example, the neural models require fewer human resources when it comes to linguistic expertise and annotation, but they require input from deep learning experts. resources such as computing power and data needs should also be considered. another issue with previous studies concerns the datasets that were used: in webnlg, approximately 99.34% of entities in the test set also appear in the training set; consequently, evaluations using webnlg do not take unseen entities into consideration. furthermore, since many sentences in webnlg are paraphrases of one another, evaluating neural models on webnlg alone may overestimate their performance. castro @xcite recently extended webnlg to include unseen domains that contain many unseen entities foot_2 , and @xcite have developed new models to handle them. their test set has two subsets: one consists of documents 99.34% of whose entities are seen, while the other consists of documents 92.81% of whose entities are unseen. this arguably makes the data in webnlg unrealistic (see §2 for discussion). therefore, we created what we believe to be a more realistic dataset based on the wall street journal (wsj) portion of the ontonotes corpus @xcite 4 . we evaluate all models on both webnlg and wsj, using automatic and human evaluation experiments. the human experiments included a total of 240 participants and 16920 judgments. this paper is structured as follows: in §2 and §3, we describe the datasets used and the reg models. in §4, we provide a detailed description of our automatic and human evaluations. in §5 and §6, we compare the results across different dimensions and make suggestions for future studies. the code for reproducing the results in this article can be found at: https://github.com/ a-quei/neuralreg-re-evaluation . ## task and datasets this section explains the reg-in-context task and the two english datasets used to conduct the experiments. ## reg models in this section, we introduce the rule-based, mlbased, and the sota neural reg models. the term ml-based here refers to models that require feature engineering and follow a pipeline architecture. ## evaluation we evaluated all the systems described in §3 on both webnlg and wsj using automatic and human evaluations. we implemented the neural models based on the code of @xcite and @xcite 7 . for webnlg, we used their original parameter setting, while for wsj, we tuned the parameters on the development set and used the best parameter set. to determine the optimal context length k of wsj, we varied k from 1 to 5 sentences before and after the target sentence, then tested att+meta on the development set with the different k contexts. it reaches the best performance when @xmath0. ## discussion why does neural reg not defeat rule-based reg? received wisdom has it that although neural models may be inferior to other models in terms of interpretability, they are nonetheless superior in terms of performance. although it is possible that future neural models will perform better than the ones examined here, our results call into question whether this received wisdom is correct. one possible explanation is the observation that neural nlg systems tend to perform very well on surface realisation tasks, but less well on tasks that focus on semantic content (see e.g., @xcite on hallucinations in the data2text generation tasks). reg, after all, is a task that focuses in large part on semantic content. there may be other reasons, which should be investigated in future work. ## conclusion in this work, we have re-evaluated state-of-theart neural reg systems by considering four well-designed rule-and ml-based baselines. in addition to the existing webnlg corpus, we built a new dataset for the task of reg-in-context on the basis of the wsj corpus, arguing that this dataset may be more appropriate for the task. in the reevaluation, we examined both our baselines and sota neural reg systems on both datasets, using automatic and human evaluations. the results suggest that the simplest rule-based baseline rreg-s achieves equally good or better performance compared to sota neural models. our results on the wsj suggest that, on that corpus, the linguisticallyinformed ml-based model (ml-l) is best. we hope these results will encourage further research into the comparative strengths and weaknesses of neural, non-neural and hybrid methods in nlp. in future, we have 4 items on our todo list: (1) investigate bottleneck features for neural based models based on the feature set of ml-l; (2) explore other neural architectures (e.g., testing models that leverage pre-trained language models) and construct larger realistic reg corpora; (3) explore better human evaluation methods for longer documents that are better suited for evaluating the task of generating referring expressions in context; (4) extend our research to other languages, especially in other language families, including languages that are morphological very rich or very poor and languages that frequently use zero pronouns (e.g., chinese @xcite ). participants needed, their remuneration turned out to be on the low side. in future crowd-sourcing experiments, we will base our remuneration on a more generous estimate of the duration per experimental task. we asked for demographic information, age, gender and english proficiency level, explicitly stating in the experiment that "your information will be used for research purposes only. all your data will be held anonymously." these fields were not marked as mandatory fields. the demographic information will not be made publicly available.