Unsupervised Grammatical Error Correction Rivaling Supervised Methods (2023)

Abstract: State-of-the-art grammatical error correction (GEC) systems rely on parallel training data (ungrammatical sentences and their manually corrected counterparts), which are expensive to construct. In this paper, we employ the Break-It-Fix-It (BIFI) method to build an unsupervised GEC system. The BIFI framework generates parallel data from unlabeled text using a fixer to transform ungrammatical sentences into grammatical ones, and a critic to predict sentence grammaticality. We present an unsupervised approach to build the fixer and the critic, and an algorithm that allows them to iteratively improve each other. We evaluate our unsupervised GEC system on English and Chinese GEC. Empirical results show that our GEC system outperforms previous unsupervised GEC systems, and achieves performance comparable to supervised GEC systems without ensemble. Furthermore, when combined with labeled training data, our system achieves new state-of-the-art results on the CoNLL-2014 and NLPCC-2018 test sets.

## introduction grammatical error correction (gec) @xcite @xcite is the task of correcting errors in a source sentence and generating a grammatically correct target sentence. current state-of-the-art (sota) systems @xcite have reached good performance using sequence-tosequence (seq2seq) models. however, a common drawback of these systems is their extensive reliance on a significant quantity of labeled data. for instance, @xcite utilized over 2 million sentence pairs, which are time-consuming and costly to obtain as they require human manual correction. unsupervised gec systems aim to over-come this limitation. however, the current performance of unsupervised gec systems @xcite is much lower than supervised systems. moreover, they still require manually defined or extracted confusion sets to generate synthetic data and assess sentence grammaticality. as a result, this greatly hinders the applicability of unsupervised gec systems. the sota unsupervised gec system, lm-critic @xcite , uses the break-it-fix-it (bifi) framework @xcite to extract realistic parallel data from unlabeled data. specifically, the bifi framework utilizes a fixer and a critic. the fixer is designed to perform the gec task, while the critic is designed for the grammatical error detection (ged) task, which classifies an input sentence as grammatical or ungrammatical. given a critic which classifies each unlabeled sentence as grammatical or ungrammatical, bifi generates parallel data to train a better fixer by the following four steps. (1) correct ungrammatical sentences with the existing fixer and collect outputs that are classified as grammatical by the critic. in this paper, we propose a novel method for generating synthetic data and building a critic, with the aim of building an unsupervised gec system that can rival supervised systems. by examining the grammatical errors in labeled data, we identified several language-independent error patterns. using these patterns, we propose a synthetic data generation method based on a masked language model (mlm) to build a fixer. subsequently, we use this fixer as a basis for building our critic. the critic is trained using grammaticality labels obtained from high-confidence fixer predictions. to address the data scarcity problem that arises from high-confidence filtering, we propose a maskingbased approach and a self-knowledge distillation method for data augmentation. the unsupervised gec system is trained using the bifi framework, with the fixer and the critic being refined repeatedly in iterations. we evaluate the performance of our system on both english and chinese gec tasks. specifically, we evaluate our system on the conll-2014 @xcite and bea-2019 @xcite test sets for english gec, and on the nlpcc-2018 @xcite test set for chinese gec. our unsupervised system outperforms the prior unsupervised sota by 12.5 f 0.5 and 13.8 f 0.5 on the conll-2014 and bea-2019 test sets, respectively. our unsupervised system also compares favorably with the best-performing supervised systems for both languages. furthermore, when we further train our system with labeled data, we surpass the sota results on both conll-2014 and nlpcc-2018 test sets. the contributions of our paper are as follows: ## related work unsupervised grammatical error correction. prior research @xcite builds an unsupervised gec system by leveraging manually constructed confusion sets to provide possible corrections, and uses language models (lms) to validate these corrections. @xcite utilize the confusion sets and lm in a different way. instead of constructing a gec model directly, yasunaga et al. (2021) use them to create a ged model. this ged model is then combined with the bifi method to build an unsupervised gec system. in contrast to these works, our method does not rely on any manually constructed confusion sets, making it easy to extend to low-resource languages. synthetic data generation. synthetic data generation for gec commonly adopts two strategies: backtranslation-based corruption methods using labeled data @xcite @xcite , and error injection corruption methods via edit pairs or confusion sets extracted from labeled data @xcite @xcite . methods that do not require labeled gec data have been explored by @xcite and @xcite . the former utilizes spellcheckerbased confusion sets to generate erroneous sentences, while the latter applies machine translation pairs and a pre-trained cross-lingual language model (xlm) for sentence corruption. our method avoids external dependencies, such as confusion sets, spellcheckers, or translation pairs. text evaluation. prior work in gec @xcite @xcite assesses sentence grammaticality through reference text or syntactic information, such as partof-speech tags. @xcite mitigate this reliance with an lm-based method, yet it still needs pre-defined confusion sets. our method constructs a critic using high-confidence predictions from the fixer model, thereby completely eliminating the need for external information. 3 method figure 1 illustrates our method to build an unsupervised gec system. it contains two key components: initial fixer 2 construction ( §3.2) and the critic construction ( §3.3). ## conclusion in this paper, we present innovative unsupervised techniques to produce synthetic parallel data and train a critic to evaluate the grammaticality of sentences. by combining our methods with bifi, we develop an unsupervised gec system that achieves results comparable to models utilizing substantial labeled data. the core idea is to employ languageindependent erroneous models to construct realistic synthetic data, and then create an unsupervised critic utilizing high-confidence predictions from the fixer model. our system does not require any manually defined or extracted confusion sets, making it an ideal solution for developing gec models for low-resource languages.