C zech Grammar Error Correction with a Large and Diverse Corpus (2022)

Abstract: We introduce a large and diverse Czech corpus annotated for grammatical error correction (GEC) with the aim to contribute to the still scarce data resources in this domain for languages other than English. The Grammar Error Correction Corpus for Czech (GECCC) offers a variety of four domains, covering error distributions ranging from high error density essays written by non-native speakers, to website texts, where errors are expected to be much less common. We compare several Czech GEC systems, including several Transformer-based ones, setting a strong baseline to future research. Finally, we meta-evaluate common GEC metrics against human judgments on our data. We make the new Czech GEC corpus publicly available under the CC BY-SA 4.0 license at http://hdl.handle.net/11234/1-4639 .

## introduction representative data both in terms of size and domain coverage are vital for nlp systems development. however, in the field of grammar error correction (gec), most gec corpora are limited to corrections of mistakes made by foreign or second language learners even in the case of english @xcite @xcite @xcite . at the same time, as recently pointed out by @xcite , learner corpora are only a part of the full spectrum of gec applications. to alleviate the skewed perspective, the authors released a corpus of website texts. despite recent efforts aimed to mitigate the notorious shortage of national gec-annotated corpora @xcite @xcite @xcite , the lack of adequate data is even more acute in languages other than english. we aim to address both the issue of scarcity of non-english data and the ubiquitous need for broad domain coverage by presenting a new, large and diverse czech corpus, expertly annotated for gec. grammar error correction corpus for czech (geccc) includes texts from multiple domains in a total of 83 058 sentences, being, to our knowledge, the largest non-english gec corpus, as well as being one of the largest gec corpora overall. in order to represent a diversity of writing styles and origins, besides essays of both native and non-native speakers from czech learner corpora, we also scraped website texts to complement the learner domain with supposedly lower error density texts, encompassing a representation of the following four domains: using the presented data, we compare several state-of-the-art czech gec systems, including some transformer-based. finally, we conduct a meta-evaluation of gec metrics against human judgments to select the most appropriate metric for evaluating corrections on the new dataset. the analysis is performed across domains, in line with @xcite . our contributions include (i) a large and diverse czech gec corpus, covering learner corpora and website texts, with unified and, in some domains, completely new gec annotations, (ii) a comparison of czech gec systems, and (iii) a meta-evaluation of common gec metrics against human judgment on the released corpus. ## model in this section, we describe five systems for automatic error correction in czech and analyze their performance on the new dataset. four of these systems represent previously published czech work @xcite @xcite and one is our new implementation. the first system is a pre-neural approach, published and available for czech @xcite , included for historical reasons as a previously known and available czech gec tool; the following four systems represent the current state of the art in gec: they are all neural network architectures based on transformers, differing in the training procedure, training data, or training objective. a comparison of systems, trained and evaluated on english, czech, german, and russian, with state of the art, is given in ## meta-evaluation of metrics there are several automatic metrics used for evaluating system performance on gec dataset, although it is not clear which of them is preferable in terms of high correlation with human judgments on our dataset. the most popular gec metrics are the max-match (m 2 ) scorer @xcite and the errant scorer @xcite . the maxmatch (m 2 ) scorer reports the f-score over the optimal phrasal alignment between a source sentence and a system hypothesis reaching the highest overlap with the gold standard annotation. it was used as the official metric for the @xcite @xcite and is also used on various other datasets such as the german falko-merlin gec @xcite or russian rulec-gec @xcite . the errant scorer was used as the official metric of the recent building educational application 2019 shared task on gec @xcite . the errant scorer also contains a set of rules operating over a set of linguistic annotations to construct the alignment and extract individual edits. other popular automatic metrics are the general language evaluation understanding (gleu) metric @xcite , which additionally measures text fluency, and i-measure @xcite , which calculates weighted accuracy of both error detection and correction. ## conclusions we release a new czech gec corpus, the grammar error correction corpus for czech (geccc). this large corpus with 83 058 sentences covers four diverse domains, including essays written by native students, informal website texts, essays written by romani ethnic minority children and teenagers and essays written by non-native speakers. all domains are professionally annotated for gec errors in a unified manner, and errors were automatically categorized with a czech-specific version of errant released at https://github.com/ufal/errant czech . we compare several strong czech gec systems, and finally, we provide a meta-evaluation of common gec metrics across domains in our data. we conclude that m 2 and errant scores with β = 0.5 are the measures most correlating with human judgments on our dataset, and we choose the m 2 0.5 as the preferred metric for the geccc dataset. the corpus is publicly available under the cc by-sa 4.0 license at http://hdl .handle.net/11234/1-4639 .