IMPARA : Impact-Based Metric for GEC Using Parallel Data (2022)

Abstract: Automatic evaluation of grammatical error correction (GEC) is essential in developing useful GEC systems. Existing methods for automatic evaluation require multiple reference sentences or manual scores. However, such resources are expensive, thereby hindering automatic evaluation for various domains and correction styles. This paper proposes an Impact-based Metric for GEC using PARAllel data, IMPARA, which utilizes correction impacts computed by parallel data comprising pairs of grammatical/ungrammatical sentences. As parallel data is cheaper than manually assessing evaluation scores, IMPARA can reduce the cost of data creation for automatic evaluation. Correlations between IMPARA and human scores indicate that IMPARA is comparable or better than existing evaluation methods. Furthermore, we find that IMPARA can perform evaluations that fit different domains and correction styles trained on various parallel data.

## introduction grammatical error correction (gec) is a task of correcting grammatically incorrect sentences @xcite @xcite @xcite . a gec system is designed to be applicable in various domains, such as web text @xcite and essays written by language learners @xcite and in different correction styles, such as minimal and fluency edits @xcite @xcite . idealy, gec models should be evaluated by manually assessing the quality of corrections made by these models for certain target domains and styles. however, it is expensive to perform manual evaluation every time a gec model outputs a correction; we thus need to establish an automatic evaluation method that correlates well with manual assessments. ## impara let x and y denote an original sentence and an output of gec system, respectively. given a gec output y , the qe yields a score qe(y ) ∈ [0, 1]. given x and y , the se computes the similarity se(x, y ) ∈ [0, 1] of the sentences. equation 1 defines the overall score of the correction from this study aims to build a relative evaluation measure that can compare two sentences in terms of the quality of grammatical correction foot_1 . we are not interested in building an absolute measure, unlike other metrics, e.g., recall and precision used in m 2 @xcite . adjusting score(x, y ) with manual assessment scores may be possible. however, we leave this as a future work because our focus is to learn an evaluation measure only from parallel data. ## experiments we conducted two types of experiments to examine the performance of impara over existing methods. first, we compared correlations between automatic evaluation metrics and human assessments in gec; this experiment measures the closeness of automatic and human evaluations. second, we evaluate automatic evaluation metrics trained and tested in different text domains and correction styles; this experiment investigates the importance to train evaluation metric in a target domain/style and the ability of impara to adapt to different domains and styles. ## related work reference-based metrics. reference-based metrics for gec use manually written reference sentences to evaluate a gec system. m 2 (dahlmeier and ng, 2012), i-measure @xcite , and errant @xcite are methods for evaluating gec systems based on f-score. these methods requires explicit edit annotations to recognize the difference between the output and reference sentences to calculate precision, recall, and f 0.5 . @xcite proposed gleu, a variant of bleu metric @xcite commonly used for machine translation. gleu does not require explicit edit annotations because it evaluates the quality of correction at the n-gram level. however, the reliability of reference-based metrics is low when reference sentences have low coverage @xcite . reference-less metrics. to address the issue of reference-based metrics, @xcite introduced the reference-less metric that does not use reference sentences for the gec evaluation. their metric comprises a grammatical error detection tool and a language model, and showed a comparable performance to reference-based metrics. correct using text-messaging language as an informal way of communicating on social media networks also has a bad effect on us in the long term . verb using text-messaging language as an informal way of communicating on social media 0.421 networks also brings in / has a bad effect on us in the long term . ## conclusion in this paper, we presented impara, a novel reference-less metric for gec. this method generates partially corrected sentences from parallel data comprising ungrammatical and grammatical sentence pairs. impara learns their preference order of pairs of partially corrected sentences, which are determined by the impact of edits. the experiment results demonstrated that impara achieved a comparable or better evaluation performance than existing reference-less methods even without tailored data of human assessments. in addition, impara exhibited a high capability of adapting its metric to the target domain and style given by parallel data. future work includes providing an interpretable scale to scores computed by impara. in addition, we plan to incorporate parallel data obtained by grammatical error generation. this may reduce the cost of building the parallel data and improve the quality of automatic evaluation metrics.