Frustratingly Easy System Combination for Grammatical Error Correction (2022)

Abstract: In this paper, we formulate system combination for grammatical error correction (GEC) as a simple machine learning task: binary classification. We demonstrate that with the right problem formulation, a simple logistic regression algorithm can be highly effective for combining GEC models. Our method successfully increases the F0.5 score from the highest base GEC system by 4.2 points on the CoNLL-2014 test set and 7.2 points on the BEA-2019 test set. Furthermore, our method outperforms the state of the art by 4.0 points on the BEA-2019 test set, 1.2 points on the CoNLL-2014 test set with original annotation, and 3.4 points on the CoNLL-2014 test set with alternative annotation. We also show that our system combination generates better corrections with higher F0.5 scores than the conventional ensemble.

## introduction grammatical error correction (gec) is the task of detecting and correcting grammatical errors present in a text @xcite . grammatical error correction has achieved remarkable progress since the late 2000s, and has flourished more along with the development of sequence-to-sequence architecture. grammatical error correction shared tasks such as @xcite , @xcite , and bea-2019 @xcite also contribute to popularizing the task. with the success of transformer @xcite architecture in sequence-to-sequence tasks, most recent state-of-the-art grammatical error correction systems use transformer-based architecture. even though they all use a transformer-based architecture, there are still some variations to the models, especially in the task formulation and pre-training data. generally, we can divide the recent state-ofthe-art systems into a sequence tagging approach 1 source code of this paper is publicly available at https: //github.com/nusnlp/esc . @xcite that usually uses a large pre-trained masked language model, and a sequence-to-sequence approach @xcite @xcite ) that usually pre-trains a transformer architecture with synthetic data. the differences in the synthetic data generation methods and seed corpora used also contribute to more diverse gec systems. with these differences, each model has its own strengths and weaknesses (figure 1 ). @xcite has demonstrated that the differences in the strengths of the gec models can be utilized to generate better grammatical error corrections by combining them through a system combination method. in this paper, we present our simple yet effective system combination method for grammatical error correction that outperforms all prior state-of-theart systems on both conll-2014 and bea-2019 shared tasks. the contributions of this paper are as follows: ## related work in this section, we discuss some prior work on gec system combination. one approach (memt) only uses the output sentences without relying on any edit type, while the other two approaches (ibm, gec-ip) only use the edit type and which hypotheses (output sentences of component systems) propose an edit type while not using the output sentences at all. in this section, we also discuss another system combination method (ddc) that introduces diversity to the base systems, which is complementary to our system combination method. ## method in this section, we describe how we formulate the task and present our method, which gathers all possible edits from the hypotheses (i.e., output sentences of individual base systems) and for each edit, predicts whether it should be kept or discarded to generate the final output sentence of the combined system. our method is named esc (edit-based system combination). ## results we combine up to six strong gec base systems, namely t5-large @xcite , gector xlnet @xcite , gector roberta @xcite , riken&tohoku @xcite , uedin-ms @xcite , and kakao&brain @xcite . for the conll-2014 combination, we do not include the kakao&brain system as a base system since its conll-2014 score (61.15) is significantly lower than the other systems (≥ 64.00). for the bea-2019 combination, our main results combine all 6 aforementioned base gec systems. in addition, our bea-2019 f 0.5 score outperforms the state of the art by 4.0 points ( ## analysis in this section, we analyze the capability of our esc method in comparison with other gec system combination methods. our method, which is a simple logistic regression, chooses whether to include an edit based on its appearance and the edit type in each component system. with edit type set t and k hypotheses, the model is a function of k|t |+1 weight parameters: @xmath0. if we assume the edit type for the same edit span and correction between the hypotheses to always be the same, we can decompose the model into |t | independent functions that accept k inputs, one function for each edit type @xmath1, as ## discussions 7.1 effect of the number of base systems in this section, we discuss the effect of using different number of gec base systems in our combination. we conduct an experiment of gec system combination with increasing number of base systems from 2 to 6 ( we find that even with combining only three systems, our method already outperforms the state of the art (75.9). we also find that adding more base systems, even though an additional system is weaker than the already combined base systems, is still beneficial to improve the f 0.5 score. we find that adding more base systems allows the model to have more information in picking the correct edits, hence improving the precision. ## conclusion and future work in this work, we present a novel gec system combination method using logistic regression by formulating the task as binary classification. our method increases the f 0.5 score by 4.17 points from the highest base gec system on conll-2014 and 7.24 points on bea-2019. this shows that our method does not overfit to the dataset even though it is trained on the bea-2019 development data. in addition, our combined system outperforms the best published gec system by 4.0 points on the bea-2019 test set, 1.21 points on the conll-2014 test set with original annotation, and 3.41 points on the conll-2014 test set with alternative annotation. we also compare our system combination method to the other gec system combination methods, including memt @xcite , ibm @xcite , and gec-ip @xcite , and show that our method has better expressive power compared to ibm and gec-ip. we also show that our method produces a higher f 0.5 score than the conventional ensemble. this work highlights the importance of edit type information in system combination for grammatical error correction. in recent gec research, most systems just employ end-to-end learning without making use of any syntactic knowledge. we show that syntactic information, such as edit type, can also be useful to improve neural grammatical error correction models. even though we only evaluate this method on english gec, we believe our method can also be applied to other languages when a similar error type annotation toolkit is available.