Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses (2020)

Abstract: Evaluation of grammatical error correction (GEC) systems has primarily focused on essays written by non-native learners of English, which however is only part of the full spectrum of GEC applications. We aim to broaden the target domain of GEC and release CWEB, a new benchmark for GEC consisting of website text generated by English speakers of varying levels of proficiency. Website data is a common and important domain that contains far fewer grammatical errors than learner essays, which we show presents a challenge to state-of-the-art GEC systems. We demonstrate that a factor behind this is the inability of systems to rely on a strong internal language model in low error density domains. We hope this work shall facilitate the development of open-domain GEC models that generalize to different topics and genres.

## introduction grammatical error correction (gec) is the task of automatically editing text to remove grammatical errors; for example: [a link to registration can also be found at on the same page.]. gec systems so far have primarily focused on correcting essays produced by english-as-a-second-language (esl) learners, providing fast and inexpensive feedback to facilitate language learning. however, this is only one target domain in the full spectrum of gec applications. gec models can also help to improve written communication outside of the formal education setting. today the largest medium of written communication is the internet, with approximately 380 new websites created every minute. @xcite ensuring grammatical correctness of websites helps facilitate clear communication and a professional commercial presentation. therefore, it is important that gec models perform well in the open-domain setting and generalize, not only to writing produced in the educational context, but also to language production "in the wild". website data specifically represent a broad and diverse range of writing and constitute a major part of what people read and write on an everyday basis. this work highlights two major prevailing challenges of current approaches to gec: domain adaptation and low precision in texts with low error density. previous work has primarily targeted essaystyle text with high error density (see figure 1 ); however, this lack of diversity means that it is not clear how systems perform on other domains and under different error distributions (sakaguchi et al., 2017). @xcite current publicly available datasets are restricted to non-native english essays [e.g. fce (yannakoudakis et al., 2011); conll14 (ng et al., ## cweb dataset we create a new dataset of english texts from randomly sampled websites, and annotate it for grammatical errors. the source texts are randomly selected from the first 18 dumps of the common-crawl foot_1 dataset and represent a wide range of data seen online such as blogs, magazines, corporate or educational websites. these include texts written by native or non-native english speakers and professional as well as amateur online writers. text extraction to ensure english content, we exclude websites with country-code top-level domains; e.g., .fr, .de. we use the justext foot_2 tool to retrieve the content from html pages (removing boilerplate elements and splitting the content into paragraphs). we heavily filter the data by removing paragraphs which contain non-english foot_3 and incomplete sentences. to ensure diversity of the data, we also remove duplicate sentences. among the million sentences gathered, we select paragraphs randomly. annotation the data is corrected for errors by two expert annotators, trained for correcting grammatical errors in english text: not attempting to rewrite the text nor make fluency edits, but rather to make minimal edits -minimum number of edits to make the text grammatical. during error annotation, the annotators have access to the entire paragraph in which a sentence belongs, therefore using the context of a sentence to help them in the correction. examples of erroneous sentences from our data are shown in the table also includes a number of other statistics, and the different datasets are further analyzed, compared and contrasted in section 5. the texts are tokenized using spacy 9 and automatically labeled for error types (and converted into the m2 format) using the error annotation toolkit (errant) (bryant et al., 2017). release for each dataset, we release a development and a test set: we propose a roughly equal division of the data into the two splits, which presents a fair amount of errors to evaluate on (see to avoid copyright restrictions, we split the collected paragraphs into sentences and shuffle all sentences in order to break the original and coherent structure that would be needed to reproduce the copyrighted material. this approach has successfully been used in previous work for devising web-based corpora (schäfer, 2015; biemann et al., 2007). the data is available at https: //github.com/simonhfl/cweb. ## gec corpora we compare our data with existing gec corpora which cover a range of domains and proficiency levels. fce (yannakoudakis et al., 2011) consists of 1, 244 error corrected texts produced by learners taking the first certificate in english exam, which assesses english at an upper-intermediate level. we use the data split made available for the bea gec shared task 2019 (bryant et al., 2019). conll14 (ng et al., 2014) consists of (mostly argumentative) essays written by esl learners from the national university of singapore, which are annotated for grammatical errors by two native speakers of english. write&improve (w&i) (bryant et al., 2019) cambridge english write & improve (yannakoudakis et al., 2018) is an online web platform that automatically provides diagnostic feedback to non-native english-language learners, including an overall language proficiency score based on the common european framework of reference for languages (cefr). @xcite the w&i corpus contains 3, 600 texts across 3 different cefr levels -a (beginner), b (intermediate), and c (advanced) -that have been annotated for errors. @xcite 3.2 other corpora locness (bryant et al., 2019; granger, 1998) the locness corpus consists of essays written by native english students. a sample of 100 essays has been annotated for errors with a 50:50 development/test split. @xcite gmeg wiki (napoles et al., 2019) is devised based on edits in the wikipedia revision history, 11 https://www.cambridgeenglish.org/ exams-and-tests/cefr/ @xcite since error corrections on test sets are not publicly available, we carry out our analyses on the development sets. note that collecting sentences based on edits in the wikipedia revision history introduces a substantial bias. @xcite this means that evaluation results on this benchmark are not truly representative of how a system would perform when applied to realistic online data and full-length articles. gmeg yahoo (napoles et al., 2019) comprises paragraphs of informal web posts gathered from answers in the yahoo! answers platform. the style is informal, and contains slang terms and nonconventional mechanics. aesw (daudaravicius et al., 2016) was released as part of the automated evaluation of scientific writing shared task. it is a collection of text extracts from published journal articles (mostly in physics and mathematics) pre-trained on synthetic errors and fine-tuned on learner data from the train section of fce (yannakoudakis et al., 2011), lang-8 (mizumoto et al., 2011), and nucle (dahlmeier et al., 2013) and for gec-pseudodata additionally on the w&i train split (bryant et al., 2019). performance is evaluated using the f 0.5 metric calculated by errant (bryant et al., 2017). @xcite however, the more annotators a dataset has, the higher score a system will get on this data (bryant and ng, 2015). in order to perform a fair comparison of systems across datasets with a different number of annotators, we calculate the errant score against each individual annotator and then take the average to get the final score. evaluation results are presented in using the gec-pseudodata system, on average, we find a higher f 0.5 on esl corpora (jf-leg, fce, conll, w&i) compared to non-esl ones (47.4 vs. 29.0). this demonstrates that gec systems trained on language learning data do not perform as well on other domains and further work is needed to improve their generalization. ## analysis in order to assess the impact our new dataset can have on the gec field, we carry out analyses to show 1) to what degree the domain of our data is different from existing gec corpora, and how existing gec systems are affected by the domain shift; and 2) that a factor behind the performance drop on cweb data is the inability of systems to rely on a strong internal language model in low error density domains. ## conclusion we release a new gec benchmark, cweb, consisting of website text generated by english speakers at varying levels of proficiency. comparisons against existing benchmarks demonstrate that cweb differs in many respects: 1) in the distribution of sentences (higher vocabulary variation and named entity frequency); 2) in error density (lower); and 3)