Towards Automatic Grammatical Error Type Classification for T urkish (2023)

Abstract: Automatic error type classification is an important process in both learner corpora creation and evaluation of large-scale grammatical error correction systems. Rule-based classifier approaches such as ERRANT have been widely used to classify edits between correct-erroneous sentence pairs into predefined error categories. However, the used error categories are far from being universal yielding many language specific variants of ERRANT.In this paper, we discuss the applicability of the previously introduced grammatical error types to an agglutinative language, Turkish. We suggest changes on current error categories and discuss a hierarchical structure to better suit the inflectional and derivational properties of this morphologically highly rich language. We also introduce ERRANT-TR, the first automatic error type classification toolkit for Turkish. ERRANT-TR currently uses a rule-based error type classification pipeline which relies on word level morphological information. Due to unavailability of learner corpora in Turkish, the proposed system is evaluated on a small set of 106 annotated sentences and its performance is measured as 77.04% F0.5 score. The next step is to use ERRANT-TR for the development of a Turkish learner corpus.

## introduction automatic error type classification is the task of assigning error type classes to predetermined grammatical errors. the building educational applications (bea) 2019 shared task on grammatical error correction (gec) @xcite emphasizes the importance of error correcting systems for educational applications. a total of 24 teams have developed gec systems for the task and er-rant @xcite (a rule-based error type classifier) was used for automatic evaluation of these systems on 5 different datasets. automatic evaluation in gec is the process of error classification on parallel data consisting of erroneous and correct sentence pairs. automatic error classification is an important process while evaluating gec systems, since the direct approach of exact match precision and recall scores is not intuitive enough to correctly analyze the strengths and weaknesses of these systems. due to the advances in deep learning based language processing in recent years, considerable performance improvements have been observed in gec systems. in parallel with this, the need for bigger and labeled datasets increased even more. usually, learner corpora foot_1 are used to create gec datasets since foreign-language learners are the ones who make such grammatical errors the most. for resource-rich languages like english, there exist many such corpora; e.g., cambridge learner corpus @xcite , nucle @xcite and w&i+locness @xcite . collecting and annotating learner corpora are pretty costly and time-consuming tasks. the erroneous sentences need to be corrected by professionals, and the inter-annotator agreement should be high (annotators should use a universal set of error categories as much as possible) so that a sufficient amount of useful samples could be obtained. the challenges of creating a learner corpus have also led researchers to find alternative data resources such as extracting edit history of comments from the web @xcite . errant-like systems help professionals annotate a vast amount of data quickly in a semi-automatic way. although there exist no prior complete gec tools nor datasets for turkish, there exist some related works (e.g., datasets related to social media errors of native speakers @xcite or to some specific error type @xcite ). these datasets are not directly usable in a general gec system since they mostly focus on social media specific error types such as intentional repetition of the same characters for exclamation purposes or misuse of diacritics due to wrong keyboard choices. text normalization and spelling correction methods which have been developed with these datasets are evaluated according to the exact match scores. however, due to the rich morphological properties of turkish, the exact match does not provide much insight on the type of errors where the system produces good and bad results. in turkish, most of grammatical errors occur in suffixes as the learners tend to make inflectional and derivational errors başak karakoç öztürk (2017). and in theory, since turkish is a highly agglutinative language, it is possible to apply some derivations recursively and result in an infinite number of possible word derivations in turkish. a single word in turkish contains more syntactic information compared to english and corresponds to several english words most of the time. therefore, while evaluating a turkish gec system output, a simple surface form matching approach loses more valuable information than it would in english. in this paper, we discuss the applicability of the previously introduced errant's grammatical error types to an agglutinative language, turkish. we suggest changes on current error categories and discuss a hierarchical structure to better suit the inflectional and derivational properties of this morphologically highly rich language. the paper introduces errant-tr, the first automatic error type classification toolkit for turkish. errant-tr currently uses a rule-based error type classification pipeline which relies on word level morphological information. due to unavailability of learner corpora in turkish, the proposed system is evaluated on a small set of 106 annotated sentences and its performance is measured as 77.04% f 0.5 score. ## related work automatic error annotation has been a popular topic in computational linguistics for a long time @xcite . researchers tried to come up with standard error categories to cover possible error types in mono or multilingual settings and tried to develop automatic annotator systems. however, it is a challenging prob-lem to come up with a unified solution and error categories due to the big structural differences between languages. there have been many attempts to develop errant-like algorithms for different languages such as greek @xcite , czech @xcite , german @xcite , spanish @xcite , russian @xcite and korean @xcite . each work considers the original error types @xcite for the corresponding language and update them according to new needs. the need for developing particular annotation methods, even for languages that fall under the same language families, proves the ongoing challenge of developing a universal error annotation scheme. learner corpora have been the main focus when it comes to creating a gec learning dataset recently. synthetically generated data has been used prior to learner corpora and are still being used as additional data during the development of gec systems @xcite @xcite @xcite @xcite . there has been other semi-automatic alternatives for creating learning datasets such as extracting wikipedia edits @xcite as correct-erroneous sentence pairs. both synthetically generated and semiautomatic datasets can be found in large amounts. however, they do not contain the natural distribution of real world errors. therefore, learner corpora have become the de-facto data source for gec systems, and many works focused on collecting and annotating these corpora @xcite @xcite . english has been the main subject of learner corpora studies as the language with the highest number of foreign students from all over the world. however for many other languages, these resources are still missing. for turkish, by the time of this writing, there is an ongoing research on collecting and manually annotating the first turkish learner corpus. in the future, we plan on evaluating our system on this corpus when the data is publicly available. the morphological structure of turkish. there are 25 error types introduced with errant. the list of categories and possible examples can be seen in in order to categorize the error types, we make use of the morphological information obtained through an automatic morphological analyzer and disambiguator @xcite . after the disambiguation process, we extract pos tags for each word from its morphological properties. the edits between correct and erroneous sentences can be grouped under 3 main categories at token level and the error types can be prefixed with "r", "m" and "u" labels, indicating "replacement", "missing" and "unnecessary" errors respectively: a necessary token may be missing or an unnecessary token may be added in the erroneous sentence. but the most common, correct token(s) are replaced with erroneous token(s). thus, not all error types are paired with "m" and "u". the possible combinations of labels can be seen in @xcite 's ## discussion in this work, we developed the first error annotation tool for turkish using the error types introduced in errant. even though the main purpose of these types is meant to cover the most common error types in a parallel corpus, during the development of errant-tr and the annotation of the validation dataset, we observed that they are not completely applicable to agglutinative languages like turkish. some common errors in turkish are not exactly covered with these types and some error types (e.g. morph, verb:infl) are too wide. especially the morphological ones need to be expanded to cover a wide range of inflectional and derivational errors. on the other hand, the advantage of an agglutinative language is that the suffixes are usually added to a word stem in an order (see @xcite and part 2 of @xcite for the case of turkish). this phenomenon helps to classify error types into hierarchical classes and makes it relatively easier to implement a decision making algorithm. @xcite proposed a hierarchical relationship between error types. for instance, a noun:poss error is also a noun error or a verb:tense error is also a verb error. knowing these relationships prior to developing a classifier will help in classifying sub-types and will provide more information during the test phase. in turkish, even more detailed hierarchical relationship between error types can be established due to the rich derivational morphology. to further illustrate this, we provide a simple example: the verb lemma yap-has the following transformations: it is derived to a participle (an adjectiveverb); the modified noun (third person singular) is dropped and the participle becomes a noun; then it is inflected with a second person singular possessive; lastly it is inflected with an accusative case. let us suppose that the learner made an error in possessive and used a third person singular possessive suffix "-ı". classifying this error as a noun:poss type loses a valuable information of a common possessive error case in participles @xcite . moreover, knowing that a verb can not be inflected with a possessive suffix, even though the lemma of the word is a verb we can safely discard the verb and its sub-types while classifying this error. therefore, a more precise classification system and an improved labeling scheme can be created by establishing hierarchical relationships among error types in a more intricate manner. for nouns and verbs, "infl" sub-categories mostly cover other sub-categories. for example, a "noun:num" or a "noun:poss" error in turkish can also be considered a "noun:infl" error as the number and possessive properties are provided with inflectional suffixes. furthermore, in the specific case of turkish, a possessive error can also be considered a genitive construction error. as can be seen in the example below, a possession suffix -ı is appended to the head (modified noun) in a genitive construction and the modifier is appended with a tamlayan (modifier) suffix -ın. ## future work in this section, we aim to address the issues concerning the labeling and classification of grammatical error types that we have discussed earlier. we also outline possible areas for further research and suggest potential enhancements for errant-tr. categorizing certain errors based solely on morphological analysis can be difficult, particularly when dealing with multi-token edits. nonetheless, supplementing morphological analysis with dependency parsing can assist in precisely aligning and categorizing multi-token errors. our intention is to use these techniques to enhance the system's performance on existing error types and address the turkish-specific errors discussed in section-5.1. the evaluation of an automatic annotation toolkit (detection, alignment and classification capabilities) is a time and resource consuming process. one needs a big amount of already-annotated parallel data with high inter-annotator agreement. as we did not possess such data we only evaluated the classifier. in this work, the ground truth data is considered properly aligned and corrected. therefore, the evaluation process has still room for improvement. we plan to test the system on a real world learner corpus which is being collected at the moment as part of an ongoing research. error type classification relies on the accuracy of the morphological analysis and disambiguation. therefore, a potential mistake in these steps may yield incorrect classifications. in order to improve the system in the future, a better morphological analyzer can be used. there is also a room for improvement in the decision making pipeline of the classifier. the detection of certain error types is not trivial with only the morphological analysis as discussed earlier. ## conclusion in this paper, we introduced errant-tr a grammatical error annotation and automatic evaluation toolkit for turkish. it automatically annotates the error types in a parallel corpus. we designed a decision making pipeline for turkish based on morphological analysis information and hand-crafted specific vocabularies (for conj, prep, pron, det types). we discussed and proposed potential changes to the error categories (which have been introduced mainly for english) in order to cover inflectional and derivational properties of turkish, an agglutinative language. we created a small evaluation dataset consisting of 106 erroneous-corrected sentence pairs collected from academic studies. errant-tr achieves an average 77.04% f 0.5 score on this dataset. we discussed the strengths and weaknesses of the system based on this evaluation. in the future, we will evaluate and improve the system on the first turkish learner corpus. errant-tr will also help learners and teachers by being used as a semi-automatic annotation toolkit while annotating erroneous-correct sentence pairs and reduce the time required to create parallel corpora drastically.