Automatic Error Type Annotation for A rabic (2021)

Abstract: We present ARETA, an automatic error type annotation system for Modern Standard Arabic. We design ARETA to address Arabic’s morphological richness and orthographic ambiguity. We base our error taxonomy on the Arabic Learner Corpus (ALC) Error Tagset with some modifications. ARETA achieves a performance of 85.8% (micro average F1 score) on a manually annotated blind test portion of ALC. We also demonstrate ARETA’s usability by applying it to a number of submissions from the QALB 2014 shared task for Arabic grammatical error correction. The resulting analyses give helpful insights on the strengths and weaknesses of different submissions, which is more useful than the opaque M2 scoring metrics used in the shared task. ARETA employs a large Arabic morphological analyzer, but is completely unsupervised otherwise. We make ARETA publicly available.

## introduction there has been a lot of interest recently in automatic error evaluation for many languages. many specialized shared tasks in grammatical error correction (gec) and text normalization have used tools like m 2 scorer @xcite and errant @xcite . in contrast with the opaque results of m 2 scorer based on extracted edits, errant, designed primarily for english, allows for deep interpretation of gec error types since it gives more detailed explanations. error type explainability is helpful for many nlp applications, including second language learning. arabic is a morphologically rich and complex language with a high degree of ambiguity at the orthographic, morphological, syntactic, lexical and semantic levels @xcite . figure 1 presents a motivating example for the complexity of arabic error type annotation (discussion in section 2). previous arabic text correction shared tasks like qalb 2014 @xcite and qalb 2015 ! ‫ﺎ‬ ‫ﻟ‬ ‫ﺴ‬ % ‫ﺎ‬ ‫ر‬ ‫ە‬ ‫أ‬ ‫ﺣ‬ ‫ﻤ‬ ‫ﺮ‬ , -. ‫ا‬ ‫ﻟ‬ ‫ﺴ‬ % ‫ﺎ‬ ‫ر‬ ‫ة‬ ‫ا‬ ‫ﻟ‬ ‫ﺤ‬ ‫ﻤ‬ ‫ﺮ‬ ‫ا‬ ‫ء‬ balsyarh âhmr fy alsyarħ alhmra' by-the-car fs red ms in the-car fs the-red fs morph changes ortho change semantic change figure 1 : an example of aligned erroneous source and reference phrases with three different error types. @xcite evaluated submissions using the m 2 scorer. @xcite proposed a taxonomy for arabic error types and annotated the arabic learner corpus (alc) using it; but, he does not provide an error classification tool. in this paper, we present areta, a system for the extraction and annotation of error types in arabic. areta is inspired by errant, but addresses the unique and complex challenges of arabic. we base our error taxonomy on the alc error tagset @xcite with some modifications. areta reaches a micro average f1 score of 85.8% on an alc blind test. we also demonstrate areta's usability on a number of submissions from qalb 2014 @xcite shared task. while areta employs a large arabic morphological analyzer, it is completely unsupervised otherwise. to our knowledge, this is the first system of its kind for arabic. we make areta publicly available. 1 the remainder of this paper is organized as follows. sections 2 and 3 present a motivating example, and related work, respectively. section 4 describes our approach. section 5 presents experimental results and discussions. ## a motivating example figure 1 shows an example of an erroneous arabic source phrase balsyarh âhmr foot_1 and its correct reference fy alsyarh alhmra'. the phrase, meaning 'in the red car', includes three error types. a simple levenshtein edit distance @xcite between the source and reference phrases suggests the reference is modified through two word substitutions and one word deletion, or three character substitutions and five character deletions. in contrast, a linguistically motivated error type classification is more insightful. from the point of view of the source phrase, there are two words, and they each get two error tags according to the alc error taxonomy ( ## related work while m 2 scorer @xcite has been used for automatic evaluation of gec shared tasks in different languages @xcite @xcite , a lot of attention has been paid to annotating and evaluating the output of english text correction systems, e.g., errant @xcite . there is still a lack of tools that allow such utility for other languages, including arabic. in the rest of this section, we present the main tools for evaluating and annotating error types, some of the challenges of arabic processing, and the arabic error taxonomy which we modify. ## approach in this section, we present our approach to developing areta, an automatic error annotation framework for arabic. we organize this section in three parts: basic word alignment, automatic error annotation, and error-type-based evaluation. given a raw input sequence (s raw ), a system output sequence (hypothesis) (s hyp ), and a reference sequence (s ref ), we want to be able to annotate and evaluate the quality of the system output (hypothesis) (s hyp ). ## experiments we present next two sets of experiments. first, we evaluate the quality of error type annotation in comparison to the alc. second, we calculate the correlation between an evaluation metric based on our error type prediction system and the m 2 scorer. ## conclusion and future work in this paper, we presented areta, a publicly available automatic error type annotation system for modern standard arabic targeting a modified error taxonomy based on the alc error tagset. we validated areta's performance using a manually annotated blind test, where it achieved 85.8% (micro avg f1 score). we also demonstrated areta's usability in providing insightful error analyses over the submissions of the qalb 2014 shared task on arabic text correction. in the future, we plan to develop a new taxonomy that resolves overlapping and ambiguous error types in the alc error tagset, and that includes more error types such as syntactic agreement and reordering operations. we also plan to use syntactic parsers, such as @xcite , to model long distance dependency errors. naturally, we will continue to improve the various components of areta, e.g., extending the handling of many-to-many word pairs, and improving specific error types.