Underreporting of errors in NLG output, and what to do about it (2021)

Abstract: We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by ‘state-of-the-art’ research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

## introduction this paper turned out very differently from the one we had initially intended to write. our original intention was to write an overview of the different kinds of errors that appear in the output of different kinds of natural language generation (nlg) systems, and to develop a general taxonomy of nlg output errors, based on the publications that have appeared at previous inlg conferences (similar to @xcite ). this, however, turned out to be impossible. the reason? there is a severe under-reporting of the different kinds of errors that nlg systems make. by this assertion, we mean that authors neither include any error analysis nor provide any examples of errors made by the system, and they do not make reference to different kinds of errors that may appear in the output. the latter is a lower bar than carrying out an error analysis, which requires a more systematic approach where several outputs are sampled and analysed for the presence of errors, which are then categorised (ideally through a formal procedure with multiple annotators). section 3 provides more detailed statistics about error reporting in different years of inlg (and enlg), and the amount of papers that discuss the kinds of errors that may appear in nlg output. the fact that errors are under-reported in the nlg literature is probably unsurprising to experienced researchers in this area. the lack of reporting of negative results in ai has been a well-known issue for many years @xcite . with the classic nlg example being the reporting of negative results for the stop project on smoking cessation @xcite . but even going in with (relatively) low expectations, it was confronting just to see how little we as a community look at the mistakes that our systems make. we believe that it is both necessary and possible to improve our ways. one of the reasons why it is necessary to provide more error analyses (see §2.2 for more), is that otherwise, it is unclear what are the strengths and weaknesses of current nlg systems. in what follows, we provide guidance on how to gain more insight into system behavior. this paper provides a general framework to carry out error analyses. first we cover the terminology and related literature ( §2), after which we quantify the problem of under-reporting ( §3). following up on this, we provide recommendations on how to carry out an error analysis ( §4). we acknowledge that there are barriers to a more widespread adoption of error analyses, and discuss some ways to overcome them ( §5). our code and data are provided as supplementary materials. ## under-reporting of errors we examined different *nlg conferences to determine the amount of papers that describe (types of) output errors, and the amount of papers that actually provide a manual error analysis. ## recommendations for error analysis we provide general recommendations for carrying out an error analysis, summarized in figure 1 . ## (overcoming) barriers to adoption one reason why authors may feel hesitant about providing an error analysis is that it takes up significantly more space than the inclusion of some overall performance statistics. the current page limits in our field may be too tight to include an error analysis. relegating error analyses to the appendix does not feel right, considering the amount of work that goes into providing such an analysis. given the effort that goes into an error analysis, authors have to make trade-offs in their time spent doing research. if papers can easily get accepted without any error analysis, it is understandable that this additional step is often avoided. how can we encourage other nlg researchers to provide more error analyses, or even just examples of errors? improving our standards we should adopt reporting guidelines that stress the importance of error analysis in papers reporting nlg experiments. the nlp community is already adopting such guidelines to improve the reproducibility of published work (see @xcite reproducibility checklist that authors for emnlp2020 need to fill in). we should also stress the importance of error reporting in our reviewing forms; authors should be rewarded for providing insightful analyses of the outputs of their systems. one notable example here is coling 2018, which explicitly asked about error analyses in their reviewing form for nlp engineering experiments, and had a 'best error analysis' award. 20,21 making space for error analyses we should make space for error analyses. the page limit in *acl conferences is already expanding to incorporate ethics statements, to describe the broader impact of our research. this suggests that we have reached the limits of what fits inside standard papers, and an expansion is warranted. an alternative is to publish more journal papers, where there is more space to fit an error analysis, but then we as a community also need to encourage and increase our appreciation of journal submissions. spreading the word finally, we should inform others about how to carry out a proper error analysis. if this is a problem of exposure, then we should have a conversation about the importance of error reporting. this paper is an attempt to get the conversation started. ## follow-up work what should you do after you have carried out an error analysis? we identify three directions for follow-up studies. ## conclusion having found that nlg papers tend to underreport errors, we have motivated why authors should carry out error analyses, and provided a guide on how to carry out such analyses. we hope that this paper paves the way for more in-depth discussions of errors in nlg output. 1. @xcite investigate the use of different seed features for controlled neural nlg. they analysed all the outputs of their model, and categorised them based on existing lists of common grammatical errors and drafting errors. a notable exception is the paper by @xcite , who carry out an error analysis of existing output data from three different systems. this paper was not considered amenable, because it does not present an nlg system of its own, and thus it was not included in our counts. but even if we were to count this paper among the error analyses, the trend remains the same: very few papers discuss errors in nlg output.