ELERRANT : Automatic Grammatical Error Type Classification for G reek (2021)

Abstract: In this paper, we introduce the Greek version of the automatic annotation tool ERRANT (Bryant et al., 2017), which we named ELERRANT. ERRANT functions as a rule-based error type classifier and was used as the main evaluation tool of the systems participating in the BEA-2019 (Bryant et al., 2019) shared task. Here, we discuss grammatical and morphological differences between English and Greek and how these differences affected the development of ELERRANT. We also introduce the first Greek Native Corpus (GNC) and the Greek WikiEdits Corpus (GWE), two new evaluation datasets with errors from native Greek learners and Wikipedia Talk Pages edits respectively. These two datasets are used for the evaluation of ELERRANT. This paper is a sole fragment of a bigger picture which illustrates the attempt to solve the problem of low-resource languages in NLP, in our case Greek.

## introduction grammatical error correction (gec) is the task of automatically correcting language mistakes in written texts. these mistakes can vary from grammatical mistakes to punctuation, spelling and morphology of a word. the development of a gec system usually involves the transformation of an erroneous sentence into its correct version, while also keeping the initial meaning intact. developing those systems requires error annotated data, which can be either learner data or artificial. highresource languages, such as english, present a variety of learner data that cover a relatively wide spectrum of language proficiency levels, native language and topics. some notable examples are the cambridge english write & improve corpus @xcite , the locness corpus @xcite , and the nucle corpus @xcite . low-resource languages, on the other hand, are characterized by a scarcity of such corpora, as well as of other gec resources and natural language processing (nlp) tools. that is also the case of greek. although, greek is only spoken by approx. 13.5 million people (native), foot_0 the fact that greece has a high immigrant population underlines the need for learning greek as a second language (gsl). 2 therefore, and as technology is being integrated in education @xcite , the need for more gec and nlp tools for greek becomes evident. in this paper, we present elerrant, an automatic annotation tool, which is based on errant @xcite . errant produces an annotation mainly consisting of the error location, the error type and the correction of the error, by using an original erroneous sentence along with its correction as input. errant is the first toolkit that not only annotates texts but also provides automatic error typing, offering detailed feedback to second language (l2) learners and useful information for language analysis @xcite . most importantly, the annotator's workload is relieved and all learner corpora, regardless of size, level and other factors can be annotated in a standardized manner. we believe that its easy application and versatility can encourage the generation of more error annotated datasets in the greek language, thus tackling the scarcity of resources. for the evaluation of elerrant we developed two datasets: the greek native corpus (gnc) and the greek wiki edits (gwe). gnc comprises native greek student essays, while gwe comprises sentences extracted from wikiconv @xcite . by evaluating elerrant on the latter corpus, we also show that the tool has the potential to detect edits and alterations on wikipedia talk pages that are due to grammatical error correction, which can then be automatically whiteflagged from being moderated for misinformation (e.g., for words introducing bias). it also paves the way for further analysis of such edits. both datasets are shared for public use. 3 the rest of the paper is structured as follows: first, we discuss related work on error annotated data and the original errant. then, we describe the development of elerrant. section 4 introduces the two new datasets, demonstrates our method of evaluation and presents the findings. section 5 is concerned with the use and implications of elerrant. finally, we conclude by discussing limitations and future work. ## related work error annotated data -what for? error annotated data can be useful in multiple domains ranging from real-life teaching and educational research to nlp tasks, especially in gec. more specifically, recent advances in gec often require large amounts of annotated data both for development and evaluation of any given systems. @xcite underline the need for cross-corpora evaluation when it comes to gec systems, given that the task difficulty depends on factors such as proficiency level and essay topic. consequently, there is a demand for standardized error-annotated corpora, while also reducing the annotator's workload @xcite . in addition, error annotated corpora can play a major role in error analysis, which has slowly started to step into call (computer assisted language learning). until very recently, the staple technique to nlg (natural language generation) for language learning purposes, was to train models on large bodies of correct english @xcite . although this technique has proven to be effective, a more recent one seems to take into account more parameters when it comes to non-native speakers. this new technique involves relying on two kinds of corpora: a source corpus from non-native texts, and a target corpus, which, in reality, is a corrected version of the source corpus. @xcite summarizes the benefits on the analysis of learner corpora claiming that the annotation of learner corpora can point out learner language properties thus supporting the aim of improving our understanding of second language acquisition (sla) and developing instructional methods and materials for sla purposes. @xcite attempt to solve the issue of corpora standardization by presenting errant, an automatic annotation tool which serves as both annotator and system-output scorer. errant only needs an erroneous sentence along with its correction to produce an annotation essentially consisting of the location of the errors, the error type, and the correction. errant has paved the way for a new annotation framework, and has therefore been used in the most recent shared task, the bea-2019 @xcite , both for annotating the datasets used for the task and for evaluating the system output of the participants per error. the convenience and versatility of errant has led to the adaptation of the tool in more languages, such as german @xcite , spanish @xcite , czech (náplava and straka, 2019), and romanian @xcite . in this paper, we present our greek version. task difficulty inter-annotator agreement, although a staple in computational linguistics procedures when it comes to evaluation, has been quite controversial regarding gec. traditionally, corpora for gec purposes would be annotated by solely one native annotator providing one gold standard annotation, a practice which automatically renders the research highly biased and even uninformative @xcite . the obvious solution to the problem would be to recruit multiple annotators and estimate the degree to which they agree on the correction of an error. yet, this method also proves insufficient, since annotators usually agree up to 70%, a percentage inadequate for system evaluation. the same holds for intra-annotator agreement, where the same annotator does not always agree with themselves (κ scores of about 60%) @xcite . ## elerrant to adapt errant in the greek language, we used the original errant classifier as blueprint. our version uses the greek hunspell spellchecker dictionary 4 to detect spelling errors and the greek error type meaning description example ad:form adverb form errors concerning the form an adverb. καλός → καλώς adj:form* adjective form errors concerning the form of an adjective καλός → καλύτερος noun:form noun form errors concerning the number,the case or the suffix of a noun. του νους → τoυ νoυ pron:form pronoun form errors concerning the number, the case or the suffix of a pronoun. κάποια → κάποιας verb:form verb form errors concerning the disposition, the voice, the inflection, the tense,the number or the person of a verb. (εσείς) πηγαίνεται → (εσείς) πηγαίνετε conj conjunction errors concerning conjunctions. και → αλλά prep preposition errors concerning prepositions. από → σε det* determiner errors concerning articles or determiners. το → του τον → έναν spell spelling spelling errors. ευχέρια → ευχέρεια fn final -ν/nu final -ν/nu addition or removal. την → τη / μη → μην punct punctuation errors concerning the punctuation. . → ; other other errors an error that does not fit into any other category but can still be corrected. καμία → για κανένα acc accentuation accentuation addition or removal. καθηκοντα → καθήκοντα unk unknown error type an error that can be detected but not corrected. usually long error spans wo words order error in words order. όταν φεύγω έρθεις → όταν έρθεις φεύγω orth* orthography spacing errors γιασένα → για σένα part:form participle form errors concerning the number,the case or the person of a participle. (πήγε) τρεχόμενος → (πήγε) τρέχοντας verb:sva subject verb agreement the subject and the verb to be in person agreement. (εγώ θα) φύγει → (εγώ θα) φύγω spacy foot_4 as the main pos tagger. due to morphological differences between the two languages (english and greek), we removed some error categories that exist in the original errant, while adding some new ones. due to the fact that greek is a highly inflectional language and most pos have some sort of inflection, we "merged" some error types in order to include as much information about the error as possible. this decision can be regarded as a compromise, since many errors might have more than one overlapping error types (e.g., των γάτα → των γατιών, wrong case and number), therefore by merging the sub-types into the form type we preserve the ambiguity and multifacedness of the error. the main alterations are the following: we added the error type ad:form (adverb form), to convey errors that mainly concern the comparative and superlative degree of the adverb. the contr (contraction) category has been removed temporarily due to the fact that contractions in greek can happen in any word starting or ending with a vowel under certain conditions. we are currently developing a dictionary that assembles the most frequent cases of contractions in greek and we plan to integrate it in future versions. noun:infl (noun inflection), noun:poss (noun possessive) and noun:num (noun number) are all captured in noun:form (noun form). part (particles) were also dropped as in the greek language they have a different function, mainly in tense construction. pron:form (pronoun form) was also added, since pronouns are also inflectional. from the verb categories, only verb, verb:form, and verb:tense have been preserved. ## empirical evaluation we consider this work an opportunity to introduce two novel datasets: greek wikiedits (gwe), and the greek native corpus (gnc), which we use as gold standards in our elerrant evaluation. this section first describes our two datasets, their annotation and development process, and presents some statistics and inter-annotator agreement results. then, the evaluation is discussed and the experimental results are reported, evaluating eler-rant on both datasets. ## discussion our experimental results showed that elerrant performs better when it comes to actual error type classification (gnc) than edit classification (gwe). in terms of scores, the elerrant classifier works adequately (83% accuracy), especially when compared to the evaluation of the original errant, for which a manual evaluation rated 86% of the output error types as "good". there is still, however, room for improvement. as mentioned in section 4, the most important issue is greek spacy which also hinders the development of a more detailed elerrant, i.e., with more error types such as noun-gender agreement, case, etc. as far as the gwe is concerned, a detection and edit classification by elerrant is possible, provided that both the edit and the proper form of the text exist. moreover, the addition of further edit categories is necessary because the current version of elerrant is based on error type classification and not edit classification. categories such as syn-onym and name might give a better insight into how the edits affect the text. rant performs better, as we can see in the second example. finally, there is the case where eler-rant is right and the human-annotator is wrong, which also indicates that an automatic annotation tool such as errant and elerrant can provide more accurate, less biased annotation. ## conclusion this paper presented elerrant, the greek version of the automatic grammatical error type annotation tool errant. with this work we also introduced two new datasets: the gnc and gwe, which can be used for gec and edit classification purposes. both our datasets are released for public use. in gnc, our findings showed that elerrant achieves an accuracy of 77.30 %-83.91%, confirming that it can be an effective tool, reducing the scarcity problem of low-resource languages, such as greek. in gwe, the overall performance was much lower, mainly because wikipedia edits are not necessarily due to gec. however, despite this low performance, we observe that elerrant can still be helpful to wikipedia moderators who can use it to shortlist edits that are likely due to gec and thus white-list them. furthermore, elerrant could be updated to capture more error types that are common in gwe, which we will consider in future work, along with the expansion of our datasets.