Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction (2021)

Abstract: Grammatical Error Correction (GEC) aims to correct writing errors and help language learners improve their writing skills. However, existing GEC models tend to produce spurious corrections or fail to detect lots of errors. The quality estimation model is necessary to ensure learners get accurate GEC results and avoid misleading from poorly corrected sentences. Well-trained GEC models can generate several high-quality hypotheses through decoding, such as beam search, which provide valuable GEC evidence and can be used to evaluate GEC quality. However, existing models neglect the possible GEC evidence from different hypotheses. This paper presents the Neural Verification Network (VERNet) for GEC quality estimation with multiple hypotheses. VERNet establishes interactions among hypotheses with a reasoning graph and conducts two kinds of attention mechanisms to propagate GEC evidence to verify the quality of generated hypotheses. Our experiments on four GEC datasets show that VERNet achieves state-of-the-art grammatical error detection performance, achieves the best quality estimation results, and significantly improves GEC performance by reranking hypotheses. All data and source codes are available at https://github.com/thunlp/VERNet .

## introduction grammatical error correction (gec) systems primarily aim to serve second-language learners for proofreading. these systems are expected to detect grammatical errors, provide precise corrections, and guide learners to improve their language ability. with the rapid increase of second-language learners, gec has drawn growing attention from numerous researchers of the nlp community. figure 1 : the grammaticality of generated hypotheses. the hypotheses are generated by @xcite with beam search decoding. the hypothesis is compared to the source sentence with a bert based language model and classified into win (the hypothesis is better), tie (the hypothesis and source are same) and loss (the source is better). the ratios of different classes are plotted with different beam search ranks. existing gec systems usually inherit the seq2seq architecture @xcite to correct grammatical errors or improve sentence fluency. these systems employ beam search decoding to generate correction hypotheses and rerank hypotheses with quality estimation models from kbest decoding @xcite or model ensemble @xcite to produce more appropriate and accurate grammatical error corrections. such models thrive from edit distance and language models @xcite @xcite @xcite . @xcite further consider the gec accuracy in quality estimation by directly predicting the official evaluation metric, f 0.5 score. the k-best hypotheses from beam search usually derive from model uncertainty @xcite . these uncertainties of multi-hypotheses come from model confidence and potential ambiguity of linguistic variation @xcite , which can be used to improve machine translation performance @xcite . @xcite further leverage multi-hypotheses to make convinced machine translation evaluation, which is more correlated with human judgments. their work further demonstrates that multi-hypotheses from well-trained neural models have the ability to provide more hints to estimate generation quality. for gec, the hypotheses from the beam search decoding of well-trained gec models can provide some valuable gec evidence. we illustrate the reasons as follows. the gec performance of the top-ranked hypothesis and the best one has a large gap in beam search. for two existing gec systems, @xcite and @xcite , the f 0.5 scores of these systems are 58.99 and 62.03 on the conll2014 dataset. however, the f 0.5 scores of the best gec results of these systems can achieve 73.56 and 76.82. existing quality estimation models (chollampatt and ng, 2018b) for gec regard hypotheses independently and neglect the potential gec evidence from different hypotheses. to fully use the valuable gec evidence from gec hypotheses, we propose the neural verification network (vernet) to estimate the gec quality with modeled interactions from multi-hypotheses. given a source sentence and k hypothesis sentences from the beam search decoding of the basic gec model, vernet establishes hypothesis interactions by regarding source, hypothesis pairs as nodes, and constructing a fullyconnected reasoning graph to propagate gec evidence among multi-hypotheses. then vernet proposes two kinds of attention mechanisms on the reasoning graph, node interaction attention and node selection attention, to summarize and aggregate necessary gec evidence from other hypotheses to estimate the quality of tokens. our experiments show that vernet can pick up necessary gec evidence from multi-hypotheses provided by gec models and help verify the quality of gec hypotheses. vernet helps gec models to generate more accurate gec results and benefits most grammatical error types. ## related work the gec task is designed for automatically proofreading. large-scale annotated corpora @xcite @xcite bring an opportunity for building fully datadriven gec systems. existing neural models regard gec as a natural language generation (nlg) task and usually use sequence-to-sequence architecture @xcite to generate correction hypotheses with beam search decoding @xcite . transformer-based architectures @xcite show their effectiveness in nlg tasks and are also employed to achieve convinced correction results @xcite . the copying mechanism is also introduced for gec models @xcite to better align tokens from source sentence to hypothesis sentence. to further accelerate the generation process, some work also comes up with non-autoregressive gec models and leverages a single encoder to parallelly detect and correct grammatical errors @xcite @xcite . recent research focuses on two directions to im-prove gec systems. the first one treats gec as a low-resource language generation problem and focuses on data augmentation for a grammar sensitive and language proficient gec system @xcite . various weak-supervision corpora have been leveraged, such as wikipedia edit history @xcite , github edit history @xcite and confusing word set @xcite . besides, lots of work generates grammatical errors through generation models or round-trip translation @xcite @xcite . @xcite further consider different data augmentation strategies to conduct better gec pretraining. reranking gec hypotheses from k-best decoding or gec model ensemble @xcite with quality estimation models provides another promising direction to achieve better gec performance. some methods evaluate if hypotheses satisfy linguistic and grammatical rules. for this purpose, they employ language models @xcite or grammatical error detection (ged) models to estimate hypothesis quality. ged models @xcite estimate the hypothesis quality on both sentence level @xcite and token level @xcite . @xcite further estimate gec quality by considering correction accuracy. they establish sourcehypothesis interactions with the encoder-decoder architecture and learn to directly predict the official evaluation score f 0.5 . the pre-trained language model bert @xcite has proven its effectiveness in producing contextual token representations, achieving better quality estimation @xcite and improving gec performance by fuse bert representations @xcite . however, existing quality estimation models regard each hypothesis independently and neglect the interactions among multihypotheses, which can also benefit the quality estimation @xcite . ## neural verification network this section describes neural verification network (vernet) to estimate the gec quality with multihypotheses, as shown in figure 3 . ## experimental methodology this section describes the datasets, evaluation metrics, baselines, and implementation details. datasets. we use fce @xcite , bea19 @xcite and nu-cle @xcite to construct training and development sets. four testing scenarios, fce, bea19 (restrict), @xcite and jfleg @xcite , are leveraged to evaluate model performance. detailed data statistics are presented in basic gec model. to generate correction hypotheses, we take one of the state-of-the-art autoregressive gec systems @xcite as our basic gec model and keep the same setting. the beam size of our baseline model is set to 5 @xcite , and all these beam search hypotheses are reserved in our experiments. we generate quality estimation labels for tokens in both source sentences and hypothesis sentences with errant @xcite , which indicate grammatical correctness and gec accuracy, respectively. as shown in evaluation metrics. we introduce the evaluation metrics in three tasks: token quality estimation, sentence quality estimation, and gec. to evaluate the model performance of tokenlevel quality estimation, we employ the same evaluation metrics from previous ged models @xcite @xcite , including precision, recall, and f 0.5 . f 0.5 is our primary evaluation metric. for the evaluation of sentence-level quality estimation, we employ the same evaluation metrics from the previous quality estimation model @xcite , including two evaluation scenarios: (1) gec evaluation metrics for the hypothesis that reranked top-1 and (2) pearson correlation coefficient (pcc) between reranking scores and golden scores (f 0.5 ) for all hypotheses. to evaluate gec performance, we adopt gleu @xcite to evaluate model performance on the jfleg dataset. the official tool errant of the bea19 shared task @xcite is used to calculate precision, recall, and f 0.5 scores for other datasets. for the conll-2014 dataset, the m 2 evaluation @xcite is also adopted as our main evaluation. baselines. bert-fuse (ged) @xcite ) is compared in our experiments, which trains bert with the ged task and fuses bert representations into the transformer. for quality estimation, we consider two groups of baseline models in our experiments, and more details of these models can be found in appendices a.1. ## evaluation results we conduct experiments to study the performance of vernet from three aspects: token-level quality estimation, sentence-level quality estimation, and the vernet's effectiveness in gec models. then we present the case study to qualitatively analyze the effectiveness of the proposed two types of attention in vernet. ## conclusion and future work this paper presents vernet for gec quality estimation with multi-hypotheses. vernet models the interactions of multiple hypotheses by building a reasoning graph, and then extracts clues with two kinds of attention: node selection attention and node interaction attention. they summarize and aggregate gec evidence from multi-hypotheses to verify the quality of tokens. experiments on four datasets show that vernet achieves the state-ofthe-art ged and quality estimation performance, and improves one published state-of-the-art gec system. in the future, we will explore the impact of different kinds of hypotheses used in vernet.