Classifying Syntactic Errors in Learner Language (2020)

Abstract: We present a method for classifying syntactic errors in learner language, namely errors whose correction alters the morphosyntactic structure of a sentence. The methodology builds on the established Universal Dependencies syntactic representation scheme, and provides complementary information to other error-classification systems. Unlike existing error classification methods, our method is applicable across languages, which we showcase by producing a detailed picture of syntactic errors in learner English and learner Russian. We further demonstrate the utility of the methodology for analyzing the outputs of leading Grammatical Error Correction (GEC) systems.

## introduction taxonomies of grammatical errors are important for linguistic and computational analysis of learner language, as well as for grammatical error correction (gec) systems. 1 such taxonomies divide the complex space of errors into meaningful categories and enable characterizing their distribution in learner productions. this information can be beneficial for gec: it can support the development of systems that focus on specific error types, serve as a form of inductive bias (for example, by regularizing the system's output to have a desired distribution over correction types), and guide data augmentation and data filtering by controlling the distribution of error types. error taxonomies can also improve the interpretability of system outputs for error analysis and learner feedback. a number of annotation efforts for learner language developed error taxonomies @xcite , and statistical classifiers we propose sercl, a taxonomy of syntactic errors (ses) and an automatic classification. inspired by a longstanding tradition in machine translation (mt) which analyses divergences between source and translated texts based on syntactic structure @xcite , sercl is based on divergences between ungrammatical sentences and their corrections. we define ses as errors whose correction involves changing morphological features, pos labels or the syntactic structure labels. sercl takes as input edits, i.e., grammatically incorrect text spans and their corrections, and compares their labels. for example, the error in fig. 1 is an adjective replaced with an adverb (adj adv) in pos terms, and an xcomp advmod in edge-label terms. thus, ses are defined by changes in form, rather than by the principles governing the choice of a correct form. sercl is the first taxonomy derived from a syntactic representation framework, and it uses the universal dependencies formalism (ud; @xcite . this approach provides three ma-jor advantages over prior learner error taxonomies. first, the sercl taxonomy is derived automatically from ud annotations, circumventing the need for constructing ad-hoc manually defined error categories. second, using the ud formalism makes the method applicable across languages, allowing for consistent analyses and comparisons of learner errors across different languages within one unified framework. third, sercl is compatible with standard representations and tools in nlp. further, the ud based approach to error classification can yield finer distinctions compared to existing schemes. for example, it divides the commonly used class of adposition errors into errors in the use of prepositions as nominal modifiers (e.g., "a mention to of previous work"), and the use of prepositions in prepositional objects or adjuncts (e.g., "referring for to previous work"). pos tags alone cannot distinguish them, but the ud trees expose this distinction straightforwardly. ud can also help classify agreement and case-assignment errors thanks to its morphological-feature layer containing information about case, number, gender, and other features relevant for inflection. we validate sercl's reliability by showing (1) ses based on automatic parses are similar to ones based on manual parses. ( §3.1); (2) sercl types map well to nucle's manually curated taxonomy ( §3.2); (3) sercl is complementary to the standard type classifier errant: 60% of the errors not classified by errant are classified by sercl. we demonstrate sercl's unique features, notably cross-linguistic applicability, by analyzing se distributions in available corpora for learner english ( §4.1) and learner russian ( §4.2). finally, we find in gec systems (1) certain ses are harder to correct (2) ses are harder than non-ses (c.f. 5) (3) the granular types can help devising rules to improve products (e.g. grammarly, §5.2). ## methodology this section defines our taxonomy of ses and how sercl classifies into it. given a parsed learner sentence and its correction, and given an edit @xmath0, e c ), i.e., a sub-string of the source sentence e s that contains a grammatical error and its reference correction e c , we define its class in the following way. we select a representative token in e s and in e c . specifically, each sub-string defines a sub-forest of the dependency parse, and the representative is taken to be the node closest acronym + ref. ## cross-linguistic corpus studies in this section, we apply sercl to available datasets, comparing between different english datasets, originally annotated in different taxonomies, and between english and russian datasets. ## analyzing gec system outputs to further showcase the utility of sercl, we demonstrate gec system analysis with it. ## related work error types are often used to improve performance and evaluation in gec. taxonomies have been used to construct classifiers and rule-based engines to correct specific error types (e.g., @xcite @xcite . when using end-to-end systems, balancing the distribution of errors in the train and test sets has been shown to improve results @xcite . ensembling black-box systems relying on per-type performance has been shown superior to each system's performance and over average ensembling @xcite . augmenting the training data with synthetic errors of a particular type is effective for improving performance on that type @xcite . the classification of grammatical error types is also used to analyze system performance (e.g., @xcite . @xcite showed that current systems and evaluation measures essentially ignore some error types, suggesting that targeted evaluation of these types may be needed. to date, several error taxonomies have been proposed and applied for annotating errors in major english learner-language corpora @xcite @xcite . there has been interest lately in other languages, for which different datasets and taxonomies were created @xcite @xcite . however, different taxonomies are used by different corpora, based on commonly observed error types in the target domain and language, which impedes direct comparison across corpora. moreover, these taxonomies are not formulated based on a specific theory or annotation scheme for morphosyntactic representation, which may promote accessibility to nonexperts but often leads to non-uniform terminology and difficulty in leveraging available nlp tools. another automatic type classification was suggested apart from errant. @xcite trained a log-linear model to predict types defined by @xcite . this taxonomy resembles ours in that it uses grammatical categories (pos tags), but differs in that it only distinguishes types based on the pos tag of the correction and not of the source sentence. moreover, relying solely on pos tags yields difficulties in classifying constructions that involve more than a single word. for such cases, it defines specialized error types, such as incorrect argument structure, which serves as a residual category for argument structure errors that cannot be accounted for by adposition or agreement errors. however, unlike sercl, it does not provide any information as to what particular incorrect argument structure was used or how it should be corrected. choshen and abend (2018c ) used a semantic annotation @xcite to show semantics, unlike syntax is kept upon changes. ud was previously used in gec in the tle corpus and in a learner language parser (e.g., @xcite (we do not apply their parser, as it is made specifically for english, and might alter the origin parse). ## conclusion we presented sercl, a novel method for classifying ses based on ud parses of learner text and its correction. we show that sercl provides a detailed picture of the prevalence of different ses in two languages, and can be straightforwardly automated. we further show that the method manages to classify about 60% of the unclassified edits by errant, the standard tool for error classification, and provides useful complementary information for many of the classified edits. future work will combine sercl and errant into a single tool for english error classification (work in this direction has already begun). the experiments we presented show that several leading gec systems of different types make errors of types that are not well-addressed by current systems. these results can inform the future development of tailored solutions for these cases.