Towards a Consensus Taxonomy for Annotating Errors in Automatically Generated Text (2023)

Abstract: Error analysis aims to provide insights into system errors at different levels of granularity. NLP as a field has a long-standing tradition of analysing and reporting errors which is generally considered good practice. There are existing error taxonomies tailored for different types of NLP task. In this paper, we report our work reviewing existing research on meaning/content error types in generated text, attempt to identify emerging consensus among existing meaning/content error taxonomies, and propose a standardised error taxonomy on this basis. We find that there is virtually complete agreement at the highest taxonomic level where errors of meaning/content divide into (1) Content Omission, (2) Content Addition, and (3) Content Substitution. Consensus in the lower levels is less pronounced, but a compact standardised consensus taxonomy can nevertheless be derived that works across generation tasks and application domains.

## introduction error analysis and error type annotation are widely considered important for diverse natural language processing (nlp) tasks @xcite . nlp has a long-standing track record in error analysis and error type annotation @xcite @xcite , not only for directly improving system performance but also for providing guidance in improving evaluation methods. errors of content (as opposed to errors of form such as grammatical or lexical-choice errors) are becoming more common in current language generation outputs, given the growing dominance of neural methods which are more prone to such errors than previous rule-based and statistical systems. documenting and analysing what types of errors different systems make can help improve the semantic correctness (known as adequacy in mt) of generated text. however, a large variety of different annotation schemes have been created @xcite , often task and/or domain-specific, which makes comparison between output annotations and thus incremental progress difficult. a standardised, task-agnostic error annotation taxonomy would not only help in comparing different nlp system outputs for performance analysis, but it would also aid in developing automatic or semiautomatic error metrics for various nlp tasks @xcite . in this paper, we explore to what extent a standard has evolved in current error annotation schemes, and whether or not enough consensus is present to turn into a standardised consensus taxonomy for errors of content/meaning. our exploration has resulted in the following contributions: the paper is organised as follows. section 2 describes the paper selection and filtering pro-cess, section 3 provides summaries of the selected papers, section 4 presents the general meaning/content error concepts and definitions we use, section 5 presents the minimally merged error taxonomy, and the maximally merged standardised version of the latter (i.e. our proposed consensus error taxonomy), section 6 discusses our findings, and section 7 concludes with a summary and future directions. ## paper selection and filtering our aim was to identify a set of papers reporting content error annotation schemes of any size and depth as a basis for deriving a consensus taxonomy. we followed the following selection/filtering process. first, we selected all papers from an existing survey on error types in machine and human-generated text @xcite that described error taxonomies or error annotation schemes comprising errors of content/meaning. this gave us seven papers. second, to further expand the selection of papers, we searched the acl anthology foot_1 for papers that contained the terms "accuracy error" and "taxonomy" which yielded 15 results. we manually examined and selected five papers reporting work on content/meaning errors for generated text. three of these papers used the same taxonomy, namely scate @xcite ; we therefore included only the main paper on the scate taxonomy @xcite . in total, we obtained three further papers from this second step. third, we added one paper @xcite ) from the related work cited by al sharou and specia (2022), and four relevant papers we were already aware of @xcite @xcite , the last of these as a (rare) example of work using the top-level content/meaning error type (adequacy, accuracy, see section) in annotation. the number of leaf nodes is simply the number of terminal nodes in a taxonomy, 4 in the above example. note that in some cases, both internal and leaf nodes are used in annotation, in other cases just leaf nodes. the depth of the tree is the longest path from the root to a leaf. in the above example, the depth is 2. if there is no underlying hierarchical structure, then @xmath1, even if an explicit one is not included). ## summaries of papers this section presents high-level summaries of the papers that directly fed into our consensus error taxonomy, focusing on content/meaning aspects. @xcite provide a corpus of 6,000 questions that have been manually translated into portuguese. error annotation addresses two types of errors that arose during the manual translation: semantic-level errors and structure-level errors. @xcite propose a statistical framework to analyse the impact of different error types, employing linear-mixed models. the experiments are designed for english as the source language and languages that are distant from english as the target language. the paper uses a set of four error classes which partially overlap with those used by @xcite : reordering errors, lexicon errors, missing words, morphological errors. @xcite introduce a linguistically motivated taxonomy of errors in machine-translated text. the taxonomy has five high-level error categories: orthography, lexis, grammar, semantic, and discourse. @xcite present a large-scale machine translation (mt) dataset that combines various degrees of human annotation with automatically recorded productivity features. errors are annotated using the multidimensional quality metrics (mqm) error annotation framework @xcite . the errors are broadly categorised into three main categories: accuracy, fluency and terminology. additionally, these errors are populated with detailed error categories from mqm. rors (relating to the wellformedness of the target sentence, regardless of content or meaning). caseli and inácio (2020) address error analysis of neural mt (nmt) system outputs for brazilian portuguese, comparing the errors made by the nmt system with those made by a phrase-based machine translation (pbsmt) system. the error analysis adopted by the paper extends the taxonomy put forward by @xcite , which consists of four broad error categories: syntactic errors, lexical errors, n-gram, reordering errors. popović (2020) introduce a manual evaluation method for mt outputs which marks up errors in the translated text. the proposed method uses two quality criteria: comprehensibility and adequacy. comprehensibility refers to the degree to which a translated text can be understood (as distinct from fluency). adequacy refers to the degree to which the translation conveys the meaning of the original source text. these error types each subdivide into major and minor. @xcite introduce polytope, a set of eight metrics for accuracy and fluency error types, designed to quantify primary errors for 10 representative models for text summarisation. accuracytype errors occur when a target summarisation does not match or accurately reflect the source text, while fluency-type errors relate to linguistic properties of the text that are independent of how source and target relate. these categories subdivide into three levels of severity: critical, minor and major. @xcite propose a methodology for gold-standard accuracy evaluations in texts generated by data-to-text systems. there are six main categories: incorrect number, incorrect named entity, incorrect word, context error, not checkable and other error. @xcite report the wmt 2021 shared task on quality estimation, where the aim is to predict the quality of outputs of neural machine translation (mt) systems at the word and sentence levels. three main categories of meaning deviation are involved: mistranslation, omission and hallucination. for each meaning deviation category, there are five critical errors. annotators are instructed to ignore minor grammatical or typographical errors. mahmud et al. (2021a) report a qualitative and quantitative comparative analysis of recently proposed source code summarisation models. a taxonomy of different error types across various models is used, with seven top-level categories: missing context, missing information, incorrect semantic information, incorrect construction, consistent with ground truth, extraneous/unnecessary, and over-generalisation. zou et al. ( 2022 ) explore the effect of translation briefs and search conditions on the quality of post-editing performed by participants with varying levels of translation expertise, using the error categorisation scheme adopted by the ata. 4 mistranslations and addition/omission errors fall under as single accuracy error type, while usage, grammar and others fall under fluency. each category has two levels of severity: accuracy critical, accuracy minor, fluency critical and fluency minor. note that errors of omission and addition are (unusually) treated as the same error type, rather than two different types, in this study. al @xcite adds two new categories of critical errors to that defined by @xcite : deviation in instructions (ins) and other critical meaning deviation (oth). @xcite investigate factual errors in summarisation system outputs, in the context of which they unify nine existing factual error annotation schemes into a single, non-hierarchical typology. the latter distinguishes errors on a number of different dimensions, of which however just two are used in the reported work: intrinsic (misrepresented words from the source text) vs. extrinsic (added words not in the source text) errors, involving a noun phrase vs. a predicate. @xcite present a zero-shot alternative for data-to-text generation using ordering, aggregation, and paragraph compression. a manual error analysis is performed using five error types: hallucination, incorrect fact merging, omissions, redundancy, grammar error, and disfluency. ## general error concepts the consensus error taxonomy we propose is intended for input-controlled text generation, rather than free text generation (see also footnote 1). in the case of the former, only content/meaning from the input must be present in the output, and all content in the input must be present in the output, except in contexts where only task-relevant parts of the input are required (e.g. in summarisation and arguably also in paraphrasing). what constitutes an error is therefore relatively clear in inputcontrolled text generation. if we think of the output as rendering the input, errors in input-controlled text generation are mismatches between input and output, where the input (1) is missing something (often referred to as an error of omission), adds something it shouldn't (error of addition), or renders something from the input wrongly (error of substitution). definitions of these and other error types are provided in the following section. it is much less clear what constitutes an error in free text generation. factual incorrectness and faulty common-sense reasoning are at the clearer end of the spectrum, but deviation from an intended reference continuation and relevance to the prompt are less clear to judge or measure. the term 'hallucination' is often used as something of a coverall term for anything that is undesirable in the output in free text generation. in contrast, in input-controlled text generation, factual incorrectness or common-sense faults have no relevance; what matters is whether what is in the output can be justified by (a) overlap between input and output content, and (b) whether the given nlp task requires all content in the input to be rendered, or just part of it. in other fields such as psychology, the term 'hallucination' is defined e.g. as "a percept, experienced by a waking individual, in the absence of an appropriate stimulus from the extracorporeal world" blom (2010). because of its association with mental health conditions, using the term for errors made by a computational system is controversial, and we prefer to use the more sober 'addition error' or just addition. omission errors are also a recognised phenomenon in neuroscience, defined (perri et al., 2017) e.g. as "infrequent errors consisting in missing responses to the target stimuli," which is fairly close to how the concept is used in nlp error assessment. section 5.1 describes the first of these steps, sec-tion 5.2 the second. section 5.3 outlines how the final consensus taxonomy is used in practice. the error taxonomies that form the starting point for our process of consensus identification often address errors of content/meaning and errors of form both. we only use the former, although the orthogonal error types below (section 5.2) can in principle apply to errors of either form or content/meaning. we draw the line between the two as follows. errors of content/meaning (in input-controlled nlg) refer to cases where the information conveyed by the output differs from the information conveyed by the input. they are defined relative to the input, hence can only be identified with reference to the input. errors of form in nlg in general refer to flaws or mistakes in how the word sequence in the output is put togehter (rather than what it means), e.g. grammatical errors, disfluencies, or inappropriate style. ## discussion error analysis identifying different types of errors plays an important role in nlp system development, providing information about specific strengths and weaknesses and their frequencies of occurrence, for different approaches, rather than a global quality assessment. for this, whether manually or automatically carried out, error categories need to be defined, at multiple levels of granularity. the current situation is that many different sets of error categories are in use, certainly for different application tasks @xcite , but very much also the same tasks, as can be e.g. seen from the ten different mt sets we have included in this paper. creating a consensus taxonomy incorporating and standardising existing taxonomies means both being able to create annotations and counts that are directly comparable across different research efforts, and, through maximising consensus increasing the taxonomy's acceptability. the consensus taxonomy as presented incorporates only error categories as used in previous work. the taxonomy can be expanded in various ways at the leaf nodes to increase granularity, notably in the substitution category, and particularly to reflect domain and task-specific distinctions. in principle, the taxonomy can be used for both manual and automatic error annotation. in standardising the error categories we have tried to make them applicable across all inputcontrolled forms of text generation. however, the judgment in particular of whether there is an omission is a different one in tasks where not all of the input needs to be rendered in the output, such as summarisation. the task we will use the taxonomy for is data-to-text generation as indicated below. ## conclusion we have presented work where we took 15 papers with error annotation schemes and derived a consensus taxonomy from them in two stages. the first was directly forming a taxonomy from error categories and hierarchical relations between them in their original forms; the second stage was maximally standardising and merging error categories and identifying and treating separately what we called orthogonal error categories that are not in any subsumption relations with other categories. an important aim is to create a basis for error annotation that is comparable across different research efforts through a single, standardised taxonomy, moreover enhancing acceptability to different practitioners by maximising the consensus embodied in the taxonomy. in our own future work, we will next use the consensus taxonomy to annotate system outputs from the webnlg 2020 shared task, and then create automatic methods for performing the annotation task. assessing inter-annotator agreement as part of the manual annotation and performance in automatic annotation will serve as two aspects of testing the taxonomy in action. craig thomson and ehud reiter. 2020. a gold standard methodology for evaluating accuracy in data-to-text systems. in proceedings of the 13th international conference on natural language generation, pages 158-168, dublin, ireland. association for computational linguistics. david vilar, jia xu, d'haro luis fernando, and hermann ney. 2006. error analysis of statistical machine translation output. in lrec, pages 697-702. deyan zou. 2022. multi-dimensional consideration of cognitive effort in translation and interpreting process studies. in proceedings of the 15th biennial conference of the association for machine translation in the americas (volume 2: users and providers track and government track), pages 416-426, orlando, usa. association for machine translation in the americas. longhui zou, michael carl, masaru yamada, and takanori mizowaki. 2022. proficiency and external aides: impact of translation brief and search conditions on post-editing quality. in proceedings of the 15th biennial conference of the association for machine translation in the americas (workshop 1: empirical translation process research), pages 60-74.