Are Pre-trained Language Models Useful for Model Ensemble in C hinese Grammatical Error Correction? (2023)

Abstract: Model ensemble has been in widespread use for Grammatical Error Correction (GEC), boosting model performance. We hypothesize that model ensemble based on the perplexity (PPL) computed by pre-trained language models (PLMs) should benefit the GEC system. To this end, we explore several ensemble strategies based on strong PLMs with four sophisticated single models. However, the performance does not improve but even gets worse after the PLM-based ensemble. This surprising result sets us doing a detailed analysis on the data and coming up with some insights on GEC. The human references of correct sentences is far from sufficient in the test data, and the gap between a correct sentence and an idiomatic one is worth our attention. Moreover, the PLM-based ensemble strategies provide an effective way to extend and improve GEC benchmark data. Our source code is available at https://github.com/JamyDon/PLM-based-CGEC-Model-Ensemble .

## introduction grammatical error correction (gec) is the task of automatically detecting and correcting errors in text @xcite . nowadays, there are two mainstream gec approaches. the first is treating gec as a low-resource machine translation task @xcite , where sequence-tosequence models like bart @xcite are used. this approach simply inputs the incorrect text to the encoder and gets the corrected result from the decoder. the second is treating gec as a sequence tagging task, where the incorrect text is still taken as the input, but the output is edit tags (keep, delete, add, replace, etc.) for each token. after applying all the edits to the input text, the corrected result is then generated. the model used in this approach is also known as sequence-to-edit * corresponding author. models and gector @xcite ) is a typical one. however, most researches on gec focus on english while chinese gec (cgec) has just started up. the chinese language is different from english in many ways and its gec is thus much harder. instead of word inflection in many western languages, the chinese grammar is expressed by function words and word order, making cgec more difficult and complex for that we can't take word form as a handle. in addition, unlike english, we have very few datasets for training and testing cgec, which sets us exploring training-free methods like model ensemble to further improve the performance of cgec systems. because of the nature of gec that corrections can be represented as several independent edits, model ensemble has been a popular way to improve gec systems. in cgec, @xcite , @xcite and @xcite ensemble their models by majority voting on edits and achieve considerable improvement. besides, @xcite adopt language models to improve neural language correction, following whom junczys-dowmunt et al. (2018) ensemble their gec models using a language model probability. today, transformer-based @xcite pre-trained language models (plms) have been in predominant use in nlp. however, we find few works on model ensemble using plms in cgec. in this work, we hypothesize that choosing the best ensemble output with the help of perplexity (ppl) computed by plms should boost the final performance of cgec. we experiment on ensemble of four cgec models, including two sequenceto-sequence ones and two sequence-to-edit ones. we try four ensemble strategies: traditional voting, sentence-level ensemble, edit-level ensemble, and edit-combination ensemble, the last three exploiting the power of plms. to our surprise, the results of model ensemble with plms do not exceed those of traditional voting and are even worse than most of the single models. to find out why a low ppl cannot lead to a better gec performance, we carry out a detailed analysis on the ensemble results and get some insights on gec: ## ensemble strategy with the source sentence and the outputs of four single models as the input, we present four ensemble strategies. the diagram of our plm-based ensamble strategies is shown in figure 1 . ## analysis and discussion we design three ensemble strategies to choose the sequence with the lowest ppl as the final output, but why does f 0.5 score drop? in our work, all single models are made up of their own plms, which means ensembling them exploiting another plm is just like using plms to judge plms, so the performance may benefit little. this is in line with the work of @xcite , where pre-trained single models gain little and even have worse performance after plm-based ensemble while other simple single models benefit a lot. besides this, are there any other reasons? ## conclusion this paper introduces novel ensemble strategies for the gec task by leveraging the power of pretrained language models (plms). we compare different strategies of model ensemble in cgec. surprisingly, plm-based ensemble strategies do not benefit the system. this suggests that ppl and f 0.5 have diverging goals. according to our analysis, the insufficiency of references in gec remains a major problem, which should be continuously improved in future work. the national natural science foundation of china (62076008) and the key project of natural science foundation of china (61936012).