Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation (2020)

Abstract: This paper presents performance measures for grammatical error correction which take into account the difficulty of error correction. To the best of our knowledge, no conventional measure has such functionality despite the fact that some errors are easy to correct and others are not. The main purpose of this work is to provide a way of determining the difficulty of error correction and to motivate researchers in the domain to attack such difficult errors. The performance measures are based on the simple idea that the more systems successfully correct an error, the easier it is considered to be. This paper presents a set of algorithms to implement this idea. It evaluates the performance measures quantitatively and qualitatively on a wide variety of corpora and systems, revealing that they agree with our intuition of correction difficulty. A scorer and difficulty weight data based on the algorithms have been made available on the web.

## introduction this paper explores difficulty-weighted performance measures for grammatical error correction. the main purpose of this work is to try to increase the diversity of grammatical error correction systems by considering error correction difficulty. in other words, we would like to encourage researchers in the domain in tackling errors that are difficult to correct automatically. despite recent progress in grammatical error correction performance, the conventional performance measures such as f 0.5 and gleu @xcite treat all errors equally. it should be emphasized that some errors are easier to correct than others. for example, errors in spelling 1 are relatively easy to correct. in contrast, errors in a/the/ϕ selection and in tense are expected to be much more difficult because it requires wider contextual information or even the intention of the writer to correct them. this nature of the conventional measures discourages researchers from tackling difficult errors. instead, it encourages them in focusing on frequent errors, which become dominant in the conventional measures. it will be good to have other measures that encourage researchers in attacking difficult errors regardless of frequency. from a technical point of view, it would be interesting to solve difficult problems. among them, there should be errors that are important in terms of language learning assistance. considering this background, this paper takes the first step toward developing performance measures that consider correction difficulty. generally, one can define the difficulty of a problem in many ways. our method adopts a simple idea inspired by academic tests (students take, for example). simply, the more students successfully solve a problem, the easier it is considered to be. this idea can be adopted in grammatical error correction. that is, the more systems successfully correct an error, the easier it is considered to be. in other words, the difficulty of error correction is related to the success rate defined by the number of systems that successfully correct the error in question divided by the total number of systems. our measures are basically weighted according to this success rate. this will naturally lead to the diversity of grammatical error correction systems because one has to correct errors that others cannot to achieve good performance in these measures. the contributions of this work are three-fold. first, we present a set of algorithms to calculate the difficulty-weighted measures. it may seem to be trivial to implement the above idea. contrary to expectation, however, there are some computation problems that need to be solved. second, we evaluate the measures quantitatively and qualitatively on a wide variety of corpora and systems, revealing their interesting behaviors. quantitatively, for example, we reveal that they are much more coherent than the conventional f 0.5 which is known to cause fluctuations in system ranking @xcite . qualitatively, we show that they agree with the intuitive difficulty of error correction as demonstrated in fig. 1 where errors are colored according to the success rate: pale (easiest) to deep (hardest) red (details are shown in sect. 5). in fig. 1 , errors in a/the/ϕ selection and tense are recognized as difficult ones while those in spelling and word form as easier ones. third and finally, we release a tool with difficulty weight data for major evaluation corpora so that anyone can readily evaluate their systems and look into easy and difficult errors foot_0 . ## related work in grammatical error correction, f 0.5 (based on recall and precision) and gleu are normally used as performance measures. in addition, evaluation tools including the maxmatch (m 2 ) scorer @xcite and errant @xcite are available to the public. without doubt, these measures and tools have greatly contributed to progress in grammatical error correction. @xcite propose a performance measure for grammatical error detection. in their measure, recall and precision are weighted according to annotation agreement rates obtained from crowdsourcing in order to take annotation reliability into account. in our measures, recall and precision are also weighted, but according to correction success rates. therefore, their measure and ours are similar in that both rely on a certain kind of rate. however, ours are different from theirs in that success rates are automatically obtained as system outputs and more importantly, they are used differently as weights; in ours, the lower the success rate is, the higher the weight is, to consider correction difficulty whereas in theirs, it is the other way around to measure annotation reliability. numerous corpora are available for grammatical error correction evaluation. these include the @xcite and @xcite datasets, cambridge esol first certificate in english (fce) @xcite , jhu fluency-extended gug corpus (jf-leg) @xcite , konan-jiem learner corpus (kj) @xcite , and international corpus network of asian learners of english, written essays (icnale) @xcite . these corpora differ in many aspects: proficiency levels and mother tongues of the writers, essay topics, and error rates to name a few. evaluation corpora for grammatical error correction have become available for other languages including german @xcite , russian @xcite , arabic @xcite , and chinese @xcite . our performance measures can be applied to any corpus in any language as long as it consists of original and correct sentence pairs. ## experiments this section evaluates the proposed performance measures in two ways: cross-corpora evaluation and system-oriented evaluation. ## conclusions this paper has taken the first step toward developing performance measures that consider correction difficulty to encourage researchers to tackle more difficult errors. it first introduced the basic idea that the more systems successfully correct an error, the easier it probably is. it then described a set of algorithms to implement the idea as difficulty-weighted performance measures. it showed empirically that they reflect the difficulty of error correction, at least to some extent, which gives incentive to tackling more difficult errors. it further discussed their characteristics and limitations. in future work, we will evaluate the correlation between the correction difficulties that the proposed method produces and that human experts estimate. also, we will investigate how we can solve the problem in the use of multiple references.