Heterogeneous Recycle Generation for C hinese Grammatical Error Correction (2020)

Abstract: Most recent works in the field of grammatical error correction (GEC) rely on neural machine translation-based models. Although these models boast impressive performance, they require a massive amount of data to properly train. Furthermore, NMT-based systems treat GEC purely as a translation task and overlook the editing aspect of it. In this work we propose a heterogeneous approach to Chinese GEC, composed of a NMT-based model, a sequence editing model, and a spell checker. Our methodology not only achieves a new state-of-the-art performance for Chinese GEC, but also does so without relying on data augmentation or GEC-specific architecture changes. We further experiment with all possible configurations of our system with respect to model composition order and number of rounds of correction. A detailed analysis of each model and their contributions to the correction process is performed by adapting the ERRANT scorer to be able to score Chinese sentences.

## introduction grammatical error correction (gec) is the task of correcting grammatical and spelling errors that appear in a sentence. an example of chinese gec is correcting the word-choice error in the following sentence: in recent years, there has been a great deal of gec related research for english, most notably with the conll 2014 shared task @xcite and the bea-2019 shared task @xcite . chinese gec has a much shorter history, with the nlpcc 2018 shared task @xcite being the first to focus on this research topic. most work prior to the nlpcc 2018 shared task focused on correcting only one type of error, such as preposition errors @xcite or chinese spelling error correction @xcite . most recent work in gec formulate correction as a translation task, and use neural machine translation (nmt) based models. that is, models are trained to translate an erroneous source sentence into a corrected target sentence. a considerable disadvantage of this approach is that nmt-based systems require an enormous amount of training data to achieve good results, while the availability of parallel correction data is limited in many languages. the current leading methods for english gec both rely on pre-training models with a large amount of artificially generated data @xcite . in this work, we aim to avoid this issue by combining several different models that perform corrections in different ways. another challenge of gec is that sentences can have multiple errors. sometimes a model is not able to correct all of the errors present in a sentence in one pass, resulting in only a partial correction. one of the methods used to resolve this issue is recycle generation, also known as iterative decoding @xcite . in this method, a system performs multiple iterations of correction on an erroneous sentence. the current state-of-the-art method for chinese gec @xcite also employs recycle generation to solve this issue. a drawback of current recycle generation methods is that for each round of correction the same model is re-applied, resulting in a limited coverage of error corrections. we postulate that larger performance gains can be had if different kinds of models perform each round of correction. in this work, we propose applying recycle generation to chinese gec using a heterogeneous system composed of different kinds of models, covering more diverse error types. our idea is to leverage the advantages of both machine translation models and sequence editing editing models. machine translation models are capable of rewriting the entire sentence, making large scale corrections such as re-ordering or performing multi-word substitutions possible. in contrast, sequence editing models focus on smaller scale corrections, such as removing a word or adding in a missing punctuation mark. each family of model is adept at correcting different kinds of errors. by integrating these two kinds of models using recycle generation, a wider range of errors can be effectively corrected for each round of correction. in this work, we also discuss the performance metrics of chinese gec. the maxmatch (m2) scorer @xcite that has been extensively used for english gec, and also for the nlpcc 2018 chinese gec task can only report overall model performance. this problem has been solved in english gec, with the introduction of the errant scorer @xcite . the er-rant scorer can provide model performance in terms of edit-level operation as well as specific english grammatical error types. we extend the idea of the errant scorer to deal with chinese sentences. this will allow chinese gec researchers to be able to get more detailed analysis of model performance. in summary, our contributions are threefold as follows. ## methodology our gec system is composed of three separate components: a neural machine translation system, a sequence editing system, and a spell-checker. each model performs one or several rounds of correction on the input sentence to produce the final corrected output. each component has separate strengths and weaknesses in terms of the errors it can correct. composing the different models in this way allows (1) one model to correct any errors that another model has missed and (2) a model can fix any errors accidentally introduced by another model earlier in the pipeline. an example of the entire correction process applied to a sentence can be seen in ## experiments the results of our recycle generation experiments are summarized in ## errant for chinese gec in the previous section, we saw that heterogeneous recycle generation was superior to homogeneous recycle generation. this was because heterogeneous recycle generation can significantly increase the recall of the gec system, while only slightly sacrificing precision. this increase in recall leads us to believe that each of our models are attempting to correct different errors that are present in the sentences. our precision not significantly decreasing, while simultaneously increasing recall means that our heterogeneous recycle generation system has vastly reduces false negatives while only slightly increasing false positives. to verify our hypothesis that each model is more adept at correcting certain error types, we have to be able to calculate model performance specific to each of our four error types: redundant (r), missing (m), word selection (s), and word ordering (w). unfortunately, the official maxmatch (m2) scorer @xcite does not have this capability. it can only provide the overall score of the system. in order to rectify this situation, we develop a scorer that can report the precision, recall, and f 0.5 score of our models with respect to our four error types. another issue with the maxmatch scorer worth mentioning is that it is known to overestimate model performance @xcite . the problem of error-type specific performance evaluation has already been solved for english gec. the errant scorer @xcite provides very detailed performance results for english gec systems, by reporting the precision, recall, and f 0.5 scores with respect to over twenty-five different error categories. the errant scorer works in two main steps, (1) edit extraction (annotation) and (2) scoring. the edit extraction step works in the following way. first, the source and target sentences are tokenized and the part-of-speech for each token is computed. next, the resulting tokenized sentences are aligned and labeled with edit-level operations: (m)atch, (d)elete, (i)nsert, and (s)ubstitute. after this, adjacent edit operations are merged using a rule-based approach. finally, the resulting edits are classified into a specific error type, using the part of speech and edit-level operation information computed in steps (1) and (2). after annotation is performed, the system edits are compared with ground-truth edits, and the precision, recall, and f 0.5 score is computed. ## conclusion in this work, we propose a system for chinese gec that uses three different models: a nmt-based model, a sequence editing model, and a spell checker. we showed how these models can be composed using heterogeneous recycle generation into a system that achieves state of the art performance for chinese gec. furthermore, we extended an automatic annotator and scorer for parallel error corpora to be able to handle chinese sentences. we use this scorer to evaluate each models performance in terms of the four error types, and show how each model is adept at correcting different types of errors. in the future, there is potential to combine the nmt-based model and the sequence editing model into a single model with both capabilities. our first attempt at adapting errant for chinese gec can also be improved upon by switching to word-level tokenization and adding pos tagging. this would allow for error classification to be performed, allowing each error to be classified into more distinct error types instead of just the four edit-level types that we currently use.