ERRANT : Assessing and Improving Grammatical Error Type Classification (2020)

Abstract: Grammatical Error Correction (GEC) is the task of correcting different types of errors in written texts. To manage this task, large amounts of annotated data that contain erroneous sentences are required. This data, however, is usually annotated according to each annotator’s standards, making it difficult to manage multiple sets of data at the same time. The recently introduced Error Annotation Toolkit (ERRANT) tackled this problem by presenting a way to automatically annotate data that contain grammatical errors, while also providing a standardisation for annotation. ERRANT extracts the errors and classifies them into error types, in the form of an edit that can be used in the creation of GEC systems, as well as for grammatical error analysis. However, we observe that certain errors are falsely or ambiguously classified. This could obstruct any qualitative or quantitative grammatical error type analysis, as the results would be inaccurate. In this work, we use a sample of the FCE coprus (Yannakoudakis et al., 2011) for secondary error type annotation and we show that up to 39% of the annotations of the most frequent type should be re-classified. Our corrections will be publicly released, so that they can serve as the starting point of a broader, collaborative, ongoing correction process.

## introduction grammatical error correction (gec) is the task of correcting different types of errors in written texts, usually by taking erroneous sentences as input and transforming them into correct ones. this can be achieved with a variety or even combination of techniques, such as language modeling @xcite , statistical machine translation @xcite , and neural machine translation @xcite ). an important step that is usually taken in these techniques is error tagging, namely "when all errors in the corpus have been annotated with the help of a standardized system of error tags" @xcite . error tagging (or error classification) is of utmost importance as it contributes to sentence transformations in a gec system, when the error is mapped to the correction through special tags, such as in @xcite . the most popular error tagger to date is the grammatical error annotation toolkit (errant), which automatically extracts and categorizes errors from parallel original and corrected texts @xcite . by employing a rule-based classifier, errant is able to expand to other languages, such as german @xcite , spanish @xcite and czech @xcite . this fact makes it particularly important for second language (l2) learning, where it can provide automatic evaluation of gec systems in several languages @xcite @xcite . this work suggests an errant improvement, by observing a major shortcoming that currently applies and suggesting the way for it to be addressed. more specific, the contributions of this work are summarised to the following: input incorrect: "i was very disappointed after this show" correct: "i was very disappointed after the show" linguisticallyenhanced alignment algorithm edit extraction: "this" rule-based error type classification: "r:det" s i was very disappointed after this show. a 5 6 ||| r:det ||| the ||| -none-||| 0 figure 1: errant system demonstration. after the input, the linguistically enhanced-algorithm aligns the two parallel sentences by making sure that items with similar linguistic properties are aligned. r:det means that the determiner 'this' needs to be replaced with the determiner 'the'. we will first present our approach to analysing the mis-classification problem. then we will discuss our observations on mis-classification frequencies and patterns, along with possible implications in gec. ## methodology for the purposes of this study, we are only concerned with the fce corpus @xcite . we used the fce data file from the bea-2019 shared task which was in m2 format and included all the extracted edits, error types and corrections. a thorough exploratory data analysis showed that the most frequent error type was r:other (see figure 2 ), meaning that something in the sentence needs to be replaced with something else that does not fit into a certain category. also, there were errors of type m:other and u:other, i.e. something is missing and something is unnecessary, respectively. we focused our analysis only on sentences containing the most frequent error type, namely other. we code meaning description example adj:form adjective form comparative/superlative adjective errors more easy (easier) orth orthography case and/or whitespace errors bestfriend (instead of best friend) verb:infl verb inflection missaplication of tense morphology getted (got), fliped (flipped) ## results & discussion according to our re-classification, 39% of the errors could have been placed in other categories (i.e., 39 errors out of the sample of 100 sentences with one error each). given that other is the most frequent error type, a large number of sentences of the fce corpus could potentially be re-classified to other categories. if this percentage applied to the whole fce dataset, this would mean that 2724 out of the 6984 other errors, are currently mistakenly tagged as other. the most frequent error type that was classified as other was r:verb, namely a word in the sentence has to be replaced with a verb. spelling mistakes (r:spell) were also very common, accounting for about 15% of the sample. preposition replacements (r:prep) comprised about 10% of the sample. there were errors that were placed in other categories, as well, but as the figure shows, they account for smaller percentages. a more qualitative demonstration is presented in in the examples, the words 'because' and 'and' are conjunctions and need to be replaced with the prepositions 'for' and 'at' respectively. therefore, we are dealing with different pos. errant ignores the option to classify the errors as r:prep (our suggestion), and classifies them as r:other instead. the specific mis-classification could be explained if we take into consideration the linguistically-enhanced alignment algorithm, which aligns linguistically similar items (see figure 1 ). because conjunctions and prepositions are different pos, errant fails to assign the correct error type. this is not the case for example 3 where the wrong preposition is replaced with a correct preposition, yet errant does not provide the correct classification again. errant seems to be also neglecting grammatical rules which have possibly not been implemented during the creation of errant (see figure 1 for the annotation process). for example, in sentence 2, the original sentence contains a wrong determiner 'a' in 'a person' and needs to be substituted with 'one'. in this case, the cardinal number 'one' becomes the determiner, hence the suggested error classification r:det. example 5 clearly con-tains a spelling mistake, but has been overlooked by errant and has been put in the r:other category. the error in example 6 was re-classified from r:other to r:verb. a hypothesis for the initial misclassification could be that 'put in' is a phrasal verb, and again the linguistically-enhanced alignment algorithm prevented the correct classification. the last example could be re-classified either as r:pron or as r:spell. the inability of the tool to choose between the annotation could be the reason behind the mis-classificaion. issues like the aforementioned must not be ignored. a more robust categorization might possibly lead to a more accurate grammatical error detection and, consequently, more efficient grammatical error correction systems. errant was used in the most recent grammatical error correction shared task (bea-2019), where all system output was automatically annotated with the scorer of the toolkit @xcite . then, the automatically inferred error type was used by the participants to evaluate their performance per type. what this means, however, is that the participants are now misjudging their systems. if we assume the existence of an oracle system that always detects correctly the error type in a (fce) sentence, then approx. 20% of the correctly detected r:verb errors (see fig. 3 ) would be considered as other errors that were miss-classified, hindering the true performance of the system for the r:verb category. ## conclusion errant has definitely provided an alternative, and to some degree, efficient way of annotating datasets for gec. this is particularly important for gec systems to be able to assess their own performance and be improved. however, we show that there is still much room for improvement regarding error type classification. although standardizing corpora can alleviate the annotators from some of the time-consuming labour, incorrect automatic classification might deprive a gec system from useful information. especially, in the case of teaching, where automatic feedback is gradually gaining ground, a precise error type classification is mandatory. in the foreground, more grammar rules should be introduced during the configuration of errant. this will allow a more thorough classification, and therefore more efficient error detection and correction systems. in addition, a qualitative evaluation by linguists could ensure the quality of the classification and provide professional feedback. we release our sample of second order fce annotations, to pose the ground for the development of a larger reference dataset. potentially, this could be used either as a ground truth evaluation set (e.g., by rule-based systems) or as a training set by more robust machine learning classifiers. our next research step would be to delve into the issue of false or ambiguous error type classification further by examining and evaluating more types of errors extracted with errant. we would also like to design a more systematic and thorough error classification system, by employing transfer learning and deep learning approaches.