SOME : Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction (2020)

Abstract: We propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction (GEC). Previous studies have shown that reference-less metrics are promising; however, existing metrics are not optimized for manual evaluations of the system outputs because no dataset of the system output exists with manual evaluation. This study manually evaluates outputs of GEC systems to optimize the metrics. Experimental results show that the proposed metric improves correlation with the manual evaluation in both system- and sentence-level meta-evaluation. Our dataset and metric will be made publicly available.

## introduction grammatical error correction (gec) is the task of automatically correcting grammatically incorrect sentences, especially those written by language learners. to develop gec systems efficiently, we construct an evaluation metric that has a high correlation with manual evaluations. reference-based metrics such as max match (m foot_0 ) @xcite and gleu @xcite are commonly used for automatic evaluation in the gec task. however, these metrics penalize sentences whose words or phrases are not included in the reference, even if they are correct expressions because it is difficult to cover all possible references @xcite . in contrast, reference-less metrics @xcite do not suffer from this limitation. among them, @xcite achieved a higher correlation with manual evaluations than reference-based metrics by integrating sub-metrics from the three perspectives of (i) grammaticality, (ii) fluency, and (iii) meaning preservation. however, the correlation with the manual evaluation of system output can be further improved because they are not considered for optimizing each sub-metric. to achieve a better correlation with manual evaluation, we create a dataset to optimize each submetric to the manual evaluation of gec systems. our annotators evaluated the output of five typical gec systems in terms of each sub-metric of @xcite . we propose a reference-less metric consisting of sub-metrics that are optimized for manual evaluation (some). it combines three regression models trained on our dataset. experimental results show that the proposed metric improves correlation with the manual evaluation in both system-and sentence-level meta-evaluation. detailed analysis reveals that optimization for both the manual evaluation and the output of gec systems contribute to improvement. source text: system output: this will inversely improve the sale of the shop. this will definitely improve the sales of the shop. ## manual evaluation of gec system outputs data and gec systems we collected manual evaluations for the grammaticality, fluency, and meaning preservation of the system outputs of 1,381 sentences from conll 2013, foot_2 which are often used to evaluate gec systems. to collect the manual evaluations for various system outputs, each source sentence was corrected by the following five typical systems: statistical machine translation (smt) @xcite , recurrent neural network (rnn) @xcite , convolutional neural network (cnn) @xcite , self-attention network (san) @xcite , and san with copy mechanism (san+copy) @xcite . more details can be found in appendix a. annotation by excluding duplicate corrected sentences, manual evaluation for the grammaticality, fluency, and meaning preservation were assigned to a total of 4,223 sentences, as follows: grammaticality: annotators evaluated the grammatical correctness of the system output. we followed the five-point scale evaluation criteria (4: perfect, 3: comprehensible, 2: somewhat comprehensible, 1: incomprehensible, and 0: other) proposed by @xcite . fluency: annotators evaluated how natural the sentence sounds for native speakers. we followed the criteria (4: extremely natural, 3: somewhat natural, 2: somewhat unnatural, and 1: extremely unnatural) proposed by @xcite . meaning preservation: annotators evaluated the extent to which the meaning of source sentences is preserved in system output. we followed the criteria (4: identical, 3: minor differences, 2: moderate differences, 1: substantially different, and 0: other) proposed by @xcite . we used amazon mechanical turk 5 and recruited five native english annotators. the average of the ratings excluding "0: other" was used as the final sentence score. for more details, refer to appendix b. finally, we created a dataset with manual evaluations for a total of 4,221 sentences, excluding sentences in which three or more annotators answered "0: other." foot_4 figure 1 presents a histogram of manual evaluations and examples of annotation. ratings of 2 or lower generally exhibited a low frequency; the majority of the meaning preservation ratings were 3 or higher. ## automatic evaluation of gec using bert using our dataset introduced in the previous section, we trained regression models corresponding to each sub-metric of @xcite . for grammaticality and fluency, the manual evaluations were estimated only from the system outputs, whereas, for meaning preservation, the manual evaluations were estimated from pairs of source sentences and system outputs. we used bert @xcite for the regression models. bert is a sentence encoder pre-trained with large-scale corpora, such as wikipedia, based on both masked language modeling and next sentence prediction, which can achieve high performance in various natural language processing tasks by fine-tuning on a small dataset of the target task. we fine-tuned three bert models for each perspective of grammaticality, fluency, and meaning preservation, and constructed sub-metrics that were optimized for manual evaluations of each perspective. the final evaluation score is calculated by the weighted linear sum of each evaluation score: @xcite . score g , score f , and score m are the normalized scores for grammaticality, fluency, and meaning preservation, respectively. the non-negative weights satisfy ↵ + + = 1. ## experimental setting to verify the effectiveness of the proposed metric (some), we performed system-and sentence-level meta-evaluation and compared the results with those of existing metrics. furthermore, to verify the effectiveness of our dataset based on the gec systems, we compared our metric with a bert-based metric fine-tuned on datasets not based on the gec systems. ## results and discussion source sentence there are a lot of disadvantages that people may not realize of . reference there are a lot of disadvantages that people may not realize . source sentence therefore i believe the parents have their right to know the healthiness of their child . reference therefore , i believe the parents have the right to know about the healthiness of their child . corrected sentence 1 therefore , i believe parents have their right to know the healthiness of their child . manual evaluation m 2 gleu asano et al. (2017) some x 0.456 0.320 0.850 0.873 corrected sentence 2 therefore , i believe parents have their right to know the healthiness of their children . manual evaluation m 2 gleu asano et al. (2017) some ⇥ 0.333 0.245 0.883 0.881 ## examples we compared each metric for the evaluation data of @xcite . ## conclusions we created a dataset with the manual evaluations of grammaticality, fluency, and meaning preservation for gec system output and proposed a bert-based reference-less metric, some, in which each submetric was optimized for each manual evaluation. the experiments demonstrated that the proposed metric achieved the highest correlation with the manual evaluations in both the system-and sentencelevel meta-evaluations. furthermore, the effectiveness of optimizing the metrics for manual evaluations on the gec system output was confirmed by comparison with bert fine-tuned on existing datasets.