System Combination via Quality Estimation for Grammatical Error Correction (2023)

Abstract: Quality estimation models have been developed to assess the corrections made by grammatical error correction (GEC) models when the reference or gold-standard corrections are not available. An ideal quality estimator can be utilized to combine the outputs of multiple GEC systems by choosing the best subset of edits from the union of all edits proposed by the GEC base systems. However, we found that existing GEC quality estimation models are not good enough in differentiating good corrections from bad ones, resulting in a low F0.5 score when used for system combination. In this paper, we propose GRECO, a new state-of-the-art quality estimation model that gives a better estimate of the quality of a corrected sentence, as indicated by having a higher correlation to the F0.5 score of a corrected sentence. It results in a combined GEC system with a higher F0.5 score. We also propose three methods for utilizing GEC quality estimation models for system combination with varying generality: model-agnostic, model-agnostic with voting bias, and model-dependent method. The combined GEC system outperforms the state of the art on the CoNLL-2014 test set and the BEA-2019 test set, achieving the highest F0.5 scores published to date.

## introduction grammatical error correction (gec) is the task of automatically detecting and correcting errors in text, including but not limited to grammatical errors, misspellings, orthographic errors, and semantic errors @xcite @xcite . a gec model is evaluated by calculating the f -score @xcite ) from comparing the edits proposed by the gec model against gold (human-annotated) reference edits. gec edits are a set of insertion, deletion, or substitution operations that are applied to the original (source) sentence to make it free from errors. an edit is represented by three values: start index, end index, and correction string ( gec quality estimation model accepts a source sentence and its correction and produces a quality score. the quality score characterizes the accuracy and appropriateness of a correction with respect to the source sentence. the higher the score, the more accurate and appropriate the correction is. a quality estimation model is typically used as a filtering method to accept or reject a correction made by a gec model @xcite . it can also be used for choosing the best correction from the top-k outputs of a gec system @xcite . in this paper, we propose to extend that use case further. instead of choosing the best correction (hypothesis) from gec models, we can use a gec quality estimation model to produce a new and more accurate correction, based on the edits that appear in the hypotheses. we generate all possible hypotheses from the edit combinations and score them using a quality estimation model. the highest-scoring hypothesis is then deemed as the most appropriate correction of the source sentence. ## gec system combination via quality estimation to be able to use a gec quality estimation model for system combination, we assume an ideal quality estimation model that can discern good hypotheses from bad ones and produce appropriate quality scores. even though a perfect quality estimation model does not exist yet, quality estimation that behaves closely to our assumption will be good enough to be useful for combining gec systems. ## quality estimation method a correction produced by a gec model can be wrong in three aspects: keeping wrong words or phrases from the source sentence, changing the words or phrases into the wrong ones, or missing some words or phrases. in other words, a quality estimation model needs to know which words are correct and which are wrong, as well as determine whether the gaps between words are correct or wrong (in which case a word or phrase needs to be inserted). a gec quality estimation model should also produce the quality scores proportionately. a better correction of the same source sentence should get a higher score than a worse one. that is, a quality estimation model should be able to rank hypotheses by their quality scores correctly. in this section, we describe our approach to build a quality estimation model with the aforementioned qualities, which we call greco (gammaticalityscorer for re-ranking corrections). ## discussions this section discusses important characteristics of our model. from this section onward, we sometimes refer to our model greco voting as g v , and greco voting+esc as g v+e . ## conclusion in this paper, we present novel methods to utilize gec quality estimation models for system combination with varying generality: modelagnostic, model-agnostic with voting bias, and model-dependent method. we report that existing gec quality estimation models are not able to differentiate good corrections from bad ones, which is shown by their ineffectiveness on the re-ranking and system combination tasks. hence, there is a need for a new quality estimation model for gec. we present a new state-of-the-art quality estimation model, greco. our model outperforms existing models on quality estimation and re-ranking evaluation. our re-ranking of the top-5 hypotheses of riken&tohoku beats the performance of the top-1 hypothesis. our source code and model weights are publicly available, making our model directly usable as a post-processing tool for re-ranking gec systems' outputs. our model, combined with a voting bias and an edit-based system combination method, successfully improves the f 0.5 scores on the gec system combination task and produces the highest f 0.5 scores on the conll-2014 test set and bea-2019 test set to date, which are 71.12 and 80.84 respectively.