Generating Diverse Corrections with Local Beam Search for Grammatical Error Correction (2020)

Abstract: In this study, we propose a beam search method to obtain diverse outputs in a local sequence transduction task where most of the tokens in the source and target sentences overlap, such as in grammatical error correction (GEC). In GEC, it is advisable to rewrite only the local sequences that must be rewritten while leaving the correct sequences unchanged. However, existing methods of acquiring various outputs focus on revising all tokens of a sentence. Therefore, existing methods may either generate ungrammatical sentences because they force the entire sentence to be changed or produce non-diversified sentences by weakening the constraints to avoid generating ungrammatical sentences. Considering these issues, we propose a method that does not rewrite all the tokens in a text, but only rewrites those parts that need to be diversely corrected. Our beam search method adjusts the search token in the beam according to the probability that the prediction is copied from the source sentence. The experimental results show that our proposed method generates more diverse corrections than existing methods without losing accuracy in the GEC task.

## introduction grammatical error correction (gec) is a task that corrects grammatical errors in an input text. depending on the input, there are multiple ways to correct such text. for example, 10 annotators can produce 10 different valid correction results for the same grammatically incorrect text @xcite . if a gec model presents multiple candidates for correction, it helps the user decide whether to utilize the correction results such that the user can select a favorite correct expression from among the candidates. however, currently existing gec models do not consider the generation of multiple correction candidates. generally, in gec, the method for obtaining multiple corrections involves the use of a plain beam search to generate the n-best candidates @xcite . however, it has been shown that a plain beam search does not provide a great enough variety of candidates and produces lists of nearly identical sequences @xcite . therefore, the n-best candidates generated by a beam search without diversity control are not expected to provide useful additional information. considering this problem, several beam search methods have been proposed to generate diverse candidates @xcite @xcite . these diverse beam search methods encourage diversity by globally rewriting all tokens in a sentence. we will refer to such methods as diverse global beam search methods. conversely, considering a local sequence transduction task in gec, wherein most of the tokens in the source and target sentences overlap, excessive correction of the input sentence is not preferred because unnecessary rewriting damages the grammatically correct parts of the input sentence. furthermore, encouraging more corrections than necessary decreases the performance of the gec itself @xcite . we hypothesize that both plain beam search and diverse global beam search methods may not be suitable for gec tasks, and a gec model must correct the grammatical errors of the input sentence in diverse ways while preserving the correct portions of the sentence. in this study, we propose a diverse local beam search method to obtain diverse outputs considering whether a token should be corrected during the beam search. note that our method can be used for any local sequence transduction task. figure 1 ## related work several studies have been proposed to obtain diverse outputs using beam search. @xcite modified standard beam search to penalize the scores of searches with the same parent node. their algorithm recommends only those hypotheses that come from different parent beams. @xcite proposed a method to divide the beam into several groups and perform beam search for each group. additionally, they added a constraint to make it harder to select tokens selected by other groups in the same time step. @xcite proposed an iterative beam search, which produces a more diverse set of candidate responses in neural dialogue modeling. however, these studies do not distinguish between the parts that do not need to be rewritten and those parts that require diverse corrections. ## diverse local beam search diverse local beam search encourages candidates of diverse corrections for parts of the input sentence that must be corrected and discourages candidates for already correct parts of the input sentence. consequently, it runs the computation for fewer parts to generate diverse candidates. for this purpose, a penalty score s b,t is assigned to each beam b at each time step t, indicating whether a correction should be made. although different methods can be used to calculate a penalty score, in this study, we use a copy probability from the copy-augmented model @xcite as a penalty score s b,t . we explain the copy-augmented model in greater detail in section 4.1. using the penalty score, we penalize the beam search score k as follows: ## conclusion existing methods of acquiring diverse outputs rewrite all tokens of the input sentence. in this work, we proposed a method to produce diverse candidates for only the parts that require corrections via gec. it was shown that the proposed method can diversify the outputs more properly in gec than the existing methods. in the future, we would like to apply this method to other model architectures and extend it to tasks other than gec.