A Survey of Recent Error Annotation Schemes for Automatically Generated Text (2022)

Abstract: While automatically computing numerical scores remains the dominant paradigm in NLP system evaluation, error analysis is receiving increasing attention, with numerous error annotation schemes being proposed for automatically generated text. However, there is little agreement about what error annotation schemes should look like, how many different types of errors should be distinguished and at what level of granularity. In this paper, our aim is to map out recent work on annotating errors in automatically generated text, with a particular focus on error taxonomies. We describe our systematic paper selection process, and survey the error annotation schemes reported in the papers, drawing out similarities and differences between them. Finally, we characterise the issues that would make it difficult to move from the current situation to a standardised error taxonomy for annotating errors in automatically generated text.

## introduction error analysis and reporting is commonly encouraged in the natural language processing (nlp) field to aid in understanding system weaknesses, most recently those that are exhibited by state-of-the-art neural systems @xcite , which have led to renewed calls in nlp for error analysis and building error taxonomies @xcite . with the advancement of neural networks and growing interest beyond pipeline-based approaches, semantic errors are increasingly observed in generation scenarios. in data-to-text generation, for example, about 40% of the e2e generation challenge system outputs contained erroneously omitted or added semantic content @xcite . ideally, the data-to-text systems that we develop produce outputs that convey all and only the input content (not omitting or arbitrarily adding any content) @xcite , so it is important to identify and understand what kinds of semantic errors occur and for what reasons, for which error annotation and subsequent analysis provides a basis. however, there is currently little agreement on how the annotation part of this should be done. in this paper, we present a survey of different error annotation schemes, with a particular focus on error taxonomies, that have been proposed in nlp. our paper selection process yielded a set of 22 papers reporting error type annotations and error taxonomies from the acl anthology. the scope of this paper is limited to error annotation schemes that include semantic errors (as well as, possibly, other types of errors, e.g. syntactic and commonsense errors). the paper is organised as follows: section 2 describes the paper selection and filtering process. section 3 provides summaries of the research presented in each paper. section 4 presents a comparative survey of the papers in terms of shared properties, section 5 discusses our findings, and section 6 concludes with a summary and future directions. ## paper selection and filtering to select papers for our survey, we searched the acl anthology foot_0 with the query terms "error taxonomy" and "nlp," and "error type annotation" and "nlp" which yielded 84 results. after removing non-paper results and duplicates, foot_1 we were left with 27 papers. we manually examined the remaining papers keeping only those that actually reported an error taxonomy or error annotation scheme including semantic errors, which left 18 papers. we added four relevant papers from the related work sections of three of the 18 papers. our survey reviews the resulting 22 papers. ## paper summaries in this section, we provide high-level summaries of each of the 22 papers in our survey. @xcite propose a linguistically motivated taxonomy for machine translation (mt) errors, classifying mt errors for english to european portuguese translation. at the top level, the taxonomy is divided into five categories: orthography, lexis, grammar, semantic and discourse. it has five levels and 25 leaf nodes. the taxonomy is shown in full in figure 3 , alongside examples extracted from the paper, in appendix a. extend earlier work @xcite , al sharou and @xcite propose a an error annotation scheme with seven categories of critical errors, i.e. those with potential negative impact on users where the meaning of the target text deviates drastically from the source text. the work distinguishes three ways in which meaning can deviate from the source sentence: mistranslation, hallucination and deletion. a few examples extracted from the paper are in appendix a. caseli and inácio (2020) report an error analysis for neural machine translation (nmt) system outputs for brazilian portuguese in which errors by the nmt system are compared with those by a pbsmt system trained on the same corpus. the paper adopts the error taxonomy from martins and @xcite which divides errors at the top level into four broad categories: syntactic errors, lexi-cal errors, errors involving n-grams and reordering. the taxonomy has three levels and 12 leaf nodes. the taxonomy can be found in appendix a. @xcite propose a statistical framework for analysing the impact of different error types based on the results from mt evaluation metrics and human perceptions with linear mixed effects models. experiments are carried out for english as the source and other languages that are distant from english as the target. this paper uses a set of four general error classes: (i) reordering errors (ii) lexicon errors, (iii) missing words and (iv) morphology errors. @xcite report an error-annotated dataset called tgea which has comprehensive annotations for texts generated with pretrained language models. it is also intended as a benchmark dataset for automatic diagnostic tasks such as error detection, error classification etc. the error taxonomy covers 25 error types in a 3-level hierarchy reflecting linguistic knowledge as shown in figure 4 in the appendix. @xcite report an automatic error type annotation system called areta for modern standard arabic. areta aims to annotate and evaluate the quality of system outputs. first, it performs word alignment of the source and the target sentence. second, the alignment is fed to the automatic error type annotation where the system tries to extract the error type. the areta taxonomy is based on the arabic learner corpus (alc) error tagset @xcite with extended merge and split classes. the latter includes 29 error tags for arabic of which 26 are used for areta. the areta taxonomy has three levels and 26 leaf nodes. the taxonomy can be found in figure 5 in appendix a. @xcite introduce polytope which quantifies primary sources of errors for 10 representative models for text summarisation. while this is not an error taxonomy paper, it reports primary sources of errors with 8 'fluency' and 'accuracy' type metrics. for polytope, (i) accuracy-related issues are defined as the summarisation not matching or accurately reflecting the source text, whereas (ii) fluency-related issues are defined as problems with the linguistic qualities of the text. level of severity is additionally marked as minor, major or critical. the paper uses the cnn/dm dataset (in its non-anonymous version) for experiments. the taxonomy has three levels and eight leaf nodes, as error types definition(s) hallucination a (h) an example definition in the context of nlp is "generated content that is nonsensical or unfaithful to the source content." this is a widely accepted term @xcite to refer to content in the output that does not have corresponding content in the input. the term comes from the field of psychology where e.g. (blom, 2010) defines hallucination as "a percept, experienced by waking individual, in the absence of an appropriate stimulus from extracorporeal world." some other terms used for errors very similar to hallucination are addition, insertion, extra words, unnecessary information. omission (o) used so commonly in mt that a definition is not usually given, this term refers to content in the input that should be rendered in the output not having corresponding content in the output @xcite some alternative terms in use are deletion, absent word/n-gram, missing context/information. replacement (re) we use this term to refer to a range of error phenomena (given a variety of names in the literature) where some content in the output is clearly intended to convey some part of the input, but does so incorrectly @xcite @xcite . we can also look at replacement errors from the perspective of a combination of omission and addition, in the special case where what is added is the incorrect version of what is omitted. some alternative terms also in use are substitution, mistranslation, transposition. an example definition is "occurrence of the same words several times or syntactically similar units unintentionally or on purpose" @xcite . some alternative terms in use are duplication, redundancy. showin in figure 6 . @xcite report a detailed analysis of errors from four morphological inflection systems for tibetan, using datasets developed by @xcite , and the error taxonomy reported by @xcite for target errors and prediction errors with a more detailed analysis on (i) errors due to words that violate lexicographic or morphophonetic constraints of the language, and (ii) allomorphy errors. this latter taxonomy has three levels and three leaf nodes, and can be found in appendix a. @xcite report a qualitative investigation of errors made by neural models fro which they create a taxonomy which consists of seven top level categories each with multiple lower level subcategories as shown in figure 7 in the appendix. altogether there are three levels and 31 leaf nodes. @xcite report a corpus of about 6,000 questions manually translated into portuguese. they provide translation guidelines which discuss two types of problems: semantic level issues and structure level issues. in addition, they report an error taxonomy with four broad error categories and carry out an error analysis. the taxonomy has three levels and nine leaf nodes, and can be found in appendix a. @xcite introduces an error taxonomy to help with post-editing operations. the error taxonomy distinguishes three broad categories at the top level: (i) morphology, (ii) source language analysis and (iii) transfer and generation. altogether there are three levels and 19 leaf nodes. figure 9 shows the taxonomy extracted from the paper. @xcite address automatic translation error correction (tec) where the goal is to produce an improved translation by correcting errors found in a translation. the paper proposes a pretraining approach for tec and also introduces a human-in-the-loop user study where it was found that higher quality translations were achieved when corrections are assisted by the tec model. the taxonomy used has three levels and five leaf nodes. it can be found in appendix b. @xcite describe an error taxonomy for lexical normalisation replacements. the work makes a clear distinction between intentional and unintentional anomalies, and the taxonomy has four levels and 14 leaf nodes. ng et al. ( 2014 ) provide an error annotation scheme for grammar error types. the paper's goal is to evaluate algorithms and systems for automatically detecting and correcting grammatical errors present in english essays written by second language learners of english. the error annotation scheme has a set of 28 categories of grammatical error corrections as a part of the conll-2014 shared task. the authors report that it is often acceptable to have multiple and different corrections in grammatical error correction. the dataset used for training is the nucle corpus, the nus corpus for learner english @xcite , and the test data is collected as written essays from 25 nus students who are non-native speakers of english where each student was asked to write two essays. figure 12 in appendix b shows the error categories from the paper. @xcite report a computeraided language learning (icall) system for beginner-level learners of russian. the goal of the system is to provide exercises supporting basic grammar learning with contextualisation for morphological errors. considering the nature of the learner's language, an error taxonomy with four broad categories for russian verbal morphology is reported. it has three levels and nine leaf nodes. more details of the taxonomy can be found in appendix b. @xcite reports work on generating linguistically informed morphological errors for russian. an error taxonomy is reported that helps in the error generation process. it has four levels and 10 leaf nodes, and can be found at figure 13 in appendix b. @xcite explore the influence of spelling errors on lexical variation measures like type-token ratio (ttr) and yule's k for learner english. the error annotation scheme reported presents two ways of spelling error correction: (i) it identifies 13 errors in the corpora. (ii) it classifies them into three groups: corrected, not corrected and not counted. the scheme has 13 error types. figure 14 in appendix b shows the list of errors from the paper. @xcite propose the cople2 corpus which is a new learner corpus for portuguese. three different linguistic error types are defined for error tagging: orthographic, grammatical and lexical. the first error type covers spelling errors, with errors here restricted to word form and punctuation marks. the second error type is for when the student has produced an ungrammatical utterance, thus going beyond individual words and considering syntactic structures. the third error type covers lexical/semantic errors. the work is mainly concerned with errors that affects meaning. these error types help in visualising the same text progressing through corrections at different stages, from the version closest to original (orthographic corrections) to the most modified one (orthographical, grammatical and lexical corrections). note that the sub-error types provided for this paper are unclear, and are not counted in tables below. @xcite present a collection of es-says called cita corpus written by italian l1 learners (corpus italiano di apprendenti l1) from the first and second years of lower secondary school. in addition, they report a three-level error annotation scheme for errors made by l1 italian learners: (i) macro-class of error (grammatical, othrographic and lexical); (ii) class of error (verb, prepositions, monosyllables); and (iii) type of modification (misuse of verb with respect to verbal tense). there are therefore four levels in the underlying taxonomy, and a total of 21 leaf nodes. figure 15 in appendix b shows the taxonomy from the paper. himoro and pareja-lora (2020) propose a spelling error taxonomy for zamboanga chabacano (zc) formalised as an ontology and an adaptive spell checking approach using character-based statistical mt. first, an iterative process is applied to samples of the cwzcc corpus for categorising different spelling errors. second, the errors are classified to create an error taxonomy. it is observed that spelling errors get more complex as one goes deeper down the tree. the taxonomy has eight levels and 14 leaf nodes. and is shown in figure 16 in appendix b. @xcite introduce a corpus of oneto-one online chatroom conversations from lessons between teachers and learners of english which is known as the teacher-student chatroom corpus (tscc). a set of 24 error types is determined on the basis of the grammatical error correction of texts in the corpus. the set is shown in figure 17 in appendix b. @xcite introduce errant which is a toolkit that annotates texts and offers error typing with detailed feedback for l2 learners of greek. annotation is based on a rule-based error type framework that distinguishes (i) error description (unnecessary, replacement and missing), and (ii) error type. the latter disinguishes 16 error types. ## properties of error annotation schemes in order to be able to compare different error annotation schemes and draw conclusions about their similarities and differences, we labeled each scheme in terms of (i) whether it was designed for machine or human generated text; (ii) whether it contained error types related to semantic accuracy, fluency or both; (iii) nlp system task; (iv) purpose of the annotation that was carried out; and (v) de-tails of how many error labels and how many hierarchical levels there are in the scheme. we describe the first two in section 4.1 below, the third and fourth in section 4.3, and the last in section 4.2. the results of labelling the 22 annotation schemes with these labels are presented in section 4.4. ## conclusion we conducted a structured survey of work on error type annotation schemes (with a focus on error taxonomies), as reported in papers from the acl anthology. we observed a number of issues while analysing the papers in our survey which we characterised in terms of (1) lack of standardisation, (2) differing/incompatible error names and definitions across different papers, and (3) borderline error types which resist being classified as either fluency or accuracy related. we found that the latter is mostly observed in error annotation for humangenerated text rather than machine-generated. we conducted our study from the perspective of semantic error annotation, as we plan to build on it in future work on developing an error taxonomy of semantic error types for data-to-text generation.