System Combination for Grammatical Error Correction Based on Integer Programming (2021)

Abstract: In this paper, we propose a system combination method for grammatical error correction (GEC), based on nonlinear integer programming (IP). Our method optimizes a novel F score objective based on error types, and combines multiple end-to-end GEC systems. The proposed IP approach optimizes the selection of a single best system for each grammatical error type present in the data. Experiments of the IP approach on combining state-of-the-art standalone GEC systems show that the combined system outperforms all standalone systems. It improves F0.5 score by 3.61% when combining the two best participating systems in the BEA 2019 shared task, and achieves F0.5 score of 73.08%. We also perform experiments to compare our IP approach with another state-of-the-art system combination method for GEC, demonstrating IP’s competitive combination capability.

## introduction grammatical error correction (gec) is the task of detecting and correcting grammatical errors of all types present in sentences in an essay, and generating a corrected essay @xcite . most of the latest gec systems rely on pretraining with synthetic data and fine-tuning with task-specific data, and employ deep neural networks with attention mechanisms @xcite . single gec systems can be highly effective in capturing a wide range of grammatical error types @xcite , but each individual system differs in its strengths and weaknesses in correcting certain error types, and the differences could result from the synthetic data used in pre-training a gec system @xcite . two main categories of synthetic data generation approaches have been introduced, including directly injecting noise into grammatically correct sentences according to error distributions @xcite , or by back-translation @xcite @xcite . while both categories could help a system achieve a high recall across many error types, it is hard to obtain a single uniform gec system that is good at correcting all error types. presented with this difficulty and the strengths of individual systems, combining single gec systems is thus a promising and efficient way to further improve precision and recall. creating an ensemble of multiple systems is a common approach when it comes to combining multiple models, and the work of @xcite has shown its effectiveness when combining single gec models with different random initializations and configurations. however, this mode of combination requires altering the component systems to achieve a tight integration. in contrast, we focus on combination methods that need to consider only the outputs of individual systems. state-of-the-art system combination approaches working in this manner include @xcite and @xcite . the approach in @xcite adopts the memt system combination technique of @xcite and learns a combined corrected sentence which is made up of different parts of multiple system outputs. the work by @xcite has proposed a system combination approach based on convex optimization. it treats single gec systems as black boxes, rounding system weights to 0 or 1, and iteratively combines two systems at a time. in this paper, we propose a novel system combination method based on nonlinear integer programming (ip). our method optimizes an f score objective based on error types, and combines multiple component gec systems simultaneously with binary selection variables. in section 2, we present related work on system combination for gec. then, our proposed ip approach is described in detail in section 3. we present experimental setup and results in section 4, provide analysis of results in section 5, and conclude in section 6. our source code is available at https://github.com/ nusnlp/gec_ip . ## related work system combination approaches for gec are based on pipelining, confusion networks, error types, or optimization. pipelining approaches, such as the camb system @xcite , adopt a pipeline of simpler to more complex gec systems for correction, but they suffer from error propagation. confusion networks, especially the memt approach for gec @xcite , learn the optimal word choice at a sentence location via a decoding scheme. the error type-based approaches aim at edit selection per error type and per system. the laix system @xcite employs a confidence table and a rule-based conflict solver to select the optimal edits from component systems. past work using integer linear programming (ilp) for gec includes @xcite . moving on to using optimization-based selection variables, the ibm approach @xcite combines edits for a pair of systems at a time, based on error types and subsets of corrections. continuous selection variables are learned by maximizing the subset-based f 0.5 objective. the ibm approach is most related to our proposed ip approach, and we highlight the key differences in section 3.3. ## system combination we combine systems based on the strengths of individual systems in terms of error types, and optimize directly the f score evaluation metric @xcite to obtain error type-based selection variables for each system. compared to @xcite , we make several major changes to achieve good precision and recall while making the combination more efficient. an overview of our proposed ip approach is illustrated in figure 1 . ## experiments 4.1 component systems to apply the ip combination method on gec, we choose three state-of-the-art gec systems as component systems. they are the systems uedin-ms @xcite and @xcite (the top two systems from the restricted track of the bea 2019 shared task), as well as tohoku @xcite . the three gec systems share interesting commonalities and exhibit salient differences. the major commonalities are the use of the transformer big architecture @xcite (the kakao system uses a variant, the copy-augmented transformer @xcite ), pre-training on 40m to 100m synthetic parallel data, ensemble of multiple models, and re-ranking. they differ in the synthetic data generation methods, the monolingual sources, the implementation of the architecture, and re-ranking features. these characteristics lead to individual strengths in correcting different types of errors, allowing room for improvements via system combination. ## analysis of results we analyze how much the combined system using the ip approach improves over individual component systems, on a per-sentence basis. since reference edits are unavailable for the blind test set, we base our analysis on the bea 2019 validation set and split it into two halves: the first half for training and the second half for testing. for each input sentence, we compare the f 0.5 scores of its output sentences generated by kakao, uedin-ms, and the system obtained by ip combination of both. we assign each output sentence s into one of two classes, based on whether (1) the f 0.5 scores of s are identical in kakao and uedin-ms; or (2) the f 0.5 scores of s are different. the findings are summarized as follows. of the 2,192 test sentences, there are 1,503 sentences where kakao and uedin-ms have the same f 0.5 score. for these sentences, the ip approach achieves the same or higher f 0.5 score on 1,501 sentences. the f 0.5 score of ip on these 1,501 sentences is 0.4% higher than each individual system. for the remaining 689 sentences that either kakao or uedin-ms performs better, 503 out of 689 sentences benefit from ip combination, with an increase of 11.58% in the overall f 0.5 score on the 503 sentences compared to the average f 0.5 score of kakao and uedin-ms. this analysis shows that the component systems benefit from the ip combination approach on a per-sentence basis. ## conclusion in this paper, we have proposed a system combination approach for gec based on nonlinear integer programming, which combines all systems at once. the use of binary selection variables is simpler and more direct, compared to using continuous variables then rounding them. the best f 0.5 score achieved is 73.08% on the bea 2019 test set.