Grammatical Error Correction for Sentence-level Assessment in Language Learning (2023)

Abstract: The paper presents experiments on using a Grammatical Error Correction (GEC) model to assess the correctness of answers that language learners give to grammar exercises. We explored whether a GEC model can be applied in the language learning context for a language with complex morphology. We empirically check a hypothesis that a GEC model corrects only errors and leaves correct answers unchanged. We perform a test on assessing learner answers in a real but constrained language-learning setup: the learners answer only fill-in-the-blank and multiple-choice exercises. For this purpose, we use ReLCo, a publicly available manually annotated learner dataset in Russian (Katinskaia et al., 2022). In this experiment, we fine-tune a large-scale T5 language model for the GEC task and estimate its performance on the RULEC-GEC dataset (Rozovskaya and Roth, 2019) to compare with top-performing models. We also release an updated version of the RULEC-GEC test set, manually checked by native speakers. Our analysis shows that the GEC model performs reasonably well in detecting erroneous answers to grammar exercises and potentially can be used for best-performing error types in a real learning setup. However, it struggles to assess answers which were tagged by human annotators as alternative-correct using the aforementioned hypothesis. This is in large part due to a still low recall in correcting errors, and the fact that the GEC model may modify even correct words—it may generate plausible alternatives, which are hard to evaluate against the gold-standard reference.

## introduction grammatical error correction (gec) is the task of automatically detecting and correcting grammatical errors in text. given the recent advancements in transformer-based gec models, which have the ability to suggest fluent and grammatically accurate corrections for input sentences, our focus lies in examining their application in language learning settings. one potential application is to check essays written by learners and provide suggestions for corrections-this can be a useful tool for secondlanguage (l2) learners to improve their writing. we are interested in incorporating gec into an intelligent computer-aided language learning (call) system, but in a more constrained scenario: our objective is to evaluate whether a gec model can be used for automatic assessment of the learner's answers to fill-in-the-blank ("cloze") and multiplechoice (mc) grammar exercises. we assume that this task is comparatively easier than correcting free-text essays, since the number of possible errors in each input sentence is constrained by the number of exercises, these exercises do not change the word order, and our focus is only on grammar. we empirically test the hypothesis: the gec model can be employed to assess the grammatical correctness of learner answers to grammar exercises, because in an input sentence containing learner answers, the gec model will fix only tokens with errors-for each erroneous answer it will suggest a correction, and will leave all correct answers unchanged. in our setting, exercises are generated by revita, a language learning system, which is used by several hundred l2 learners. these exercises are automatically generated based on a text selected for practice @xcite . the system has one particular expected answer for each exercisethe one found in the original text. when doing an exercise, the learner may insert the expected answer, an error, or an alternative-correct answer, which is not expected, but fits the context. the problem can be stated as follows: an unexpected but suitable answer should be recognized as alternative correct, since providing incorrect feedback for valid answers can discourage learners @xcite . for example, in certain sentences, using the present or past tense can be equally acceptable. however, few corpora provide this type of annotation, therefore gec models are predominantly trained and evaluated using only one reference per instance @xcite . we use a freely available dataset relco, collected from revita over several years @xcite . this dataset contains short paragraphs with answers from learners of russian. the paragraphs include multiple answers provided to the same grammar exercises, which were manually checked and tagged as acceptable or erroneous. to the best of our knowledge, this is the only freely available dataset of this type. as a gec model, we fine-tune a pre-trained monolingual t5 language model @xcite . the contributions of this paper are: (1) we show that a gec model can achieve reasonable performance in assessing erroneous answers for fill-in-theblank and mc grammar exercises, if we use several top correction hypotheses. we empirically confirm the intuition that a transformer-based gec model cannot be used for assessing alternative-correct answers since top correction hypotheses can include corrections even for valid words. the lower-ranked hypotheses change the input sentence more freely: include more lexical changes, and more word removals or insertions. (2) we release a new version of the manually corrected rulec-gec test set, which, we believe, can improve the evaluation of gec models in the future. (3) we present the first experiment with @xcite , the semi-automatically collected learner data, to train a gec model. using relco shows an improvement in gec performance. (4) we extensively evaluate the performance of our model on the rulec-gec (henceforth-rulec) test set automatically and manually, including an evaluation of several top hypotheses, and show an improvement of f 0.5 score over the existing state-of-the-art results for russian. prior work showed that evaluating gec output only by automatically comparing it with a single gold-standard reference per sentence results in under-estimating the performance (rozovskaya and roth, 2021). the paper is organized as follows. section 2 covers prior work on the gec task. section 3 describes the problem and our approach. section 4 presents the data for training the gec model, the training procedure, and the evaluation. section 5 presents the experiments on assessing learner answers using the trained gec model. it includes a discussion of results and error analyses. section 6 presents the conclusions and future work. ## related work most current approaches treat gec as a natural language generation task. it can be formulated as a monolingual translation from incorrect to correct language using various architectures @xcite @xcite @xcite @xcite . due to the paucity of annotated training data for gec, it has become standard practice to generate synthetic data, using various ways of creating erroneous sentences-by back-translation or random token-level transformations @xcite , using the history of wikipedia edits @xcite , confusion sets suggested by spell-checkers @xcite , real error patterns @xcite @xcite , or applying noise to a latent representation of an errorfree sentence @xcite . a comparative study of methods of generating synthetic data is presented in @xcite . another approach is text editing-generating a sequence of edits to apply to the incorrect input sentence @xcite @xcite . in gec-tor @xcite , the authors develop a set of custom token-level transformations to recover the target text from the source. editing is faster than generating the whole corrected sentence, but requires constructing many language-specific transformations. more on gec and existing approaches to the problems and evaluation is reviewed in @xcite . a number of papers focus on the actual use of gec models by language learners. homma and komachi (2020) approach the problem of gec usability as a part of a writing-support system for japanese, with a focus on inference speed and working with incomplete sentences. @xcite present a writing-improvement model, which is adapted to the writer's first language (l1). the model's output was evaluated on grammaticality, acceptability, and lexical and syntactic diversity. an example-based gec with a focus on interpretability is introduced in @xcite : the model presents to the learners correction results and examples as a base for correction. @xcite explore the learners' proficiency-wise evaluation for quality figure 1 : proposal for how to use a gec model to check the learner's answers in automatically generated exercises: if an answer was corrected in the majority of the top-7 hypotheses, assume it is an error. otherwise, assume it is correct if it was not altered in the top-3 hypotheses. red denotes incorrect forms, blue-correct. ## problem setup our task is to evaluate and provide feedback on the grammatical correctness of answers given by the learner to all grammar exercises (cloze and mc) generated by a call system in a sentence. for example, in the sentence in figure 1 , "вероятно, такие приборы уже изобретены." ("probably, such gadgets have already been invented."), the learner receives one mc exercise (приборы vs. преборы, "gadgets") and a cloze exercise with lemma "изобрести" ("to invent"). the mc has only one correct answer, if the exercise is well-designed. in the cloze, the student's answer can be: (1) a definite error; (2) definitely correct, if it matches the expected past passive form "изобретены"; or (3) impersonal past tense "изобрели", which is an acceptable, slightly different way of saying the same thing ("probably someone has already invented such gadgets"). these alternative corrections can be potentially incorrect in a wider context, but we focus on the context of one sentence to simplify the task. the proposed approach is to use a gec model whose input is a sentence with all of the learner's answers inserted jointly. this is important, because words chosen by the call system for exercises can grammatically depend on each other, and various combinations of answers could be correct, e.g., "gadgets have" vs. "a gadget has." our conjecture is: if an answer was corrected by a gec model, it is likely an error; if it was not corrected, it is probably correct. to increase our trust in the model's predictions and address the potential issue of undercorrected errors, we employ a beam search to generate multiple top-ranked hypotheses instead of relying solely on the top-1 correction, see details in section 5. previous research by @xcite has demonstrated experimentally that lower-ranked hypotheses produced by gec systems could also be taken into account because they can be qualitatively even better than the top-1 hypotheses, which often suffer from the tendency of gec systems to under-correct errors due to training with one gold reference per input sentences. ## gec for evaluating learner answers the following task is to evaluate whether a gec model can be directly used for assessing learners' answers in a call system. evaluation. we generated 7 hypotheses for each sentence in the relco test set. the source sentences did not need pre-processing. we applied a post-processing step to filter out 874 hypotheses containing word repetitions, following the method used for the rulec test set: a filtered-out hypothesis is replaced with its source sentence. next, we describe the procedure for checking learner answers based on the suggested corrections. we define the word inserted by the learner as an answer to an exercise as the target word. firstly, we align the suggested gec hypotheses with the corresponding source sentences. then, for each target word, we follow the steps: we chose to evaluate the top-3 hypotheses because previous testing on rulec showed that they had the highest quality among all generated hypotheses. the results of evaluating grammatical correctness of answers using this algorithm are presented in ## conclusions and future work we present experiments on using transformerbased gec models to evaluate the correctness of answers provided by language learners to grammar exercises. to the best of our knowledge, it is the first attempt to directly employ a gec model for this task. we find that the top-performing gec model demonstrates the potential to detect and correct errors in user answers provided to fill-in-theblank and multiple-choice grammar exercises, if we use multiple top hypotheses generated with beam search. however, this approach is less effective for assessing alternative-correct answers. given the current low recall of the gec model, there is a high chance of labeling erroneous answers as acceptable. furthermore, the number of possible alternative corrections proposed by more advanced gec models can be high, meaning that when the gec model corrects an answer, it does not necessarily indicate the presence of an error. the problem of evaluating alternative correct answers is equivalent to the problem of multiple possible corrections for a given error span in gec. this issue is particularly challenging because gec models are primarily trained and evaluated using a single reference for each sentence, as discussed in @xcite . in our future work, we aim to focus on developing methods for evaluating the suggested corrections by combining reference-based and reference-free scoring approaches. while gec is typically approached as a task involving isolated sentences, there have been studies addressing document-level gec as well @xcite . in our experiments, we also focused on assessing grammatical correctness at the sentence level. however, in future work, we plan to investigate the assessment of learner answers within a paragraph. we intend to conduct further research on leveraging large language models to evaluate the acceptability of answers and explore the combination of various re-ranking methods.