Assessing Grammatical Correctness in Language Learning (2021)

Abstract: We present experiments on assessing the grammatical correctness of learners’ answers in a language-learning System (references to the System, and the links to the released data and code are withheld for anonymity). In particular, we explore the problem of detecting alternative-correct answers: when more than one inflected form of a lemma fits syntactically and semantically in a given context. We approach the problem with the methods for grammatical error detection (GED), since we hypothesize that models for detecting grammatical mistakes can assess the correctness of potential alternative answers in a learning setting. Due to the paucity of training data, we explore the ability of pre-trained BERT to detect grammatical errors and then fine-tune it using synthetic training data. In this work, we focus on errors in inflection. Our experiments show a. that pre-trained BERT performs worse at detecting grammatical irregularities for Russian than for English; b. that fine-tuned BERT yields promising results on assessing the correctness of grammatical exercises; and c. establish a new benchmark for Russian. To further investigate its performance, we compare fine-tuned BERT with one of the state-of-the-art models for GED (Bell et al., 2019) on our dataset and RULEC-GEC (Rozovskaya and Roth, 2019). We release the manually annotated learner dataset, used for testing, for general use.

## introduction many intelligent tutoring systems (its) and computer-aided language learning systems (call) generate exercises and try to assess the learner's answers automatically. providing feedback to the learner is difficult, due to the critical requirement of very high precision-providing incorrect feedback is much more harmful than no feedback at 1 revita.cs.helsinki.fi all. for this reason, most existing systems have prefabricated sets of exercises, with possible expected answers and prepared feedback. revita is an online l2 learning system for learners beyond the beginner level, which can be used in the classroom and for self-study @xcite @xcite . it covers several languages, most of which are highly inflectional, with rich morphology. in contrast to the pre-fabricated approach, revita allows the learner to upload arbitrary texts to be used as learning content, and automatically creates exercises based on the chosen content. at practice time, revita presents the text one paragraph at a time with some words hidden and used as fill-inthe-blank (cloze) exercises. for each hidden word, revita provides a hint-the base form (lemma) of the word. the learner should insert the inflected form of the lemma, given the context. continuous assessment of the user's answers is also performed automatically @xcite . revita checks the learner's answer by comparing it with the expected answer-the one found in the original text. the problem arises when, for some exercise, besides the expected answer, another answer is also valid in the context. as a result, revita may provide undesirable feedback by flagging answers that are not expected, but nonetheless correct, as "errors"-this can strongly mislead, confuse and discourage the learner. for example, both highlighted answers in the example below can be considered correct, but revita expects the learner to use only the past tense form "сдавал" ("took"): "мне приснилось, как я сдавал экзамены." ("i saw a dream how i took exams.") "мне приснилось, как я сдаю экзамены." ("i saw a dream how i take exams.") hence, detecting alternative-correct answers is essential in the learning context. we manually checked a large set of answers, which were marked learner level advanced others gram. error 649 (62.2%) 4777 (72.8%) alternative corr. 395 (37.8%) 1024 (15.6%) by revita as "erroneous" and discovered that the percentage of alternative-correct answers for advanced learners is more than double the percentage for other learners (see to address this problem, we build a model, which takes a paragraph with learner answers and decides whether they are grammatically correct. if the model is not certain about a user's answer, we can fall back on the "default" method-comparing to the expected answer. for evaluation, we created a dataset of paragraphs containing answers given by real learners, manually annotated for acceptability in their context. 2 additionally, we do not focus on semantics, due to the current setup of the exercisesif a learner inserts an answer with a lemma that is different from the given hint, that is always considered erroneous. learners may give such answers, but they are easily identified and are not considered and are not annotated in this set of experiments. the paper is organized as follows. in section 2, we review prior work on ged, with a focus on limited training data. in section 3, we describe the learner corpora collected by revita and used for evaluation and propose a novel method for generating data with simulated grammatical errors. in section 4, we first experiment with a pre-trained bert as a masked language model (mlm). we fine-tune the pre-trained bert on synthetic data and measure its ability to assess grammatical correctness on the learner data. in section 5, we discuss the results of the experiments. in section 6, we summarize our contribution and discuss future work. ## related work early experiments with ged utilized rules @xcite and supervised learning from error-annotated corpora @xcite . much work focused on detection of particular types of errors, e.g., verb forms @xcite . later work is mostly independent of the type of errors, and explores various neural architectures. @xcite first approach error detection by bi-lstm models, which achieved strong results on in-domain data. @xcite added character-level embeddings to capture morphological similarities between words. @xcite experiment with using a secondary language modeling (lm) objective. @xcite perform experiments with adding multiple auxiliary tasks for error detection. the best result was achieved by combining the main error detection task with predicting error types, pos tags, and types of grammatical relations. in subsequent experiments, the architecture was modified for jointly learning to label tokens and sentences @xcite . @xcite extended the above model by incorporating contextualized word embeddings produced by bert, @xcite @xcite . bert embeddings produced the best performance across all test sets. previous work on ged mostly uses bi-lstm as a classification model, combined with various approaches for augmenting the training data @xcite , or creating new, grammatically-specific word embeddings @xcite . more recent work utilizes transformer models @xcite @xcite . several works on gec focus on lower-resource languages, including russian, using the rulec-gec dataset for training or fine-tuning @xcite @xcite . @xcite outperformed results of @xcite by over 100% on f 0.5 , but still showed poor performance compared with other languages in the experiment. gec for russian is demonstrated to be the most challenging task, which is explained in part by the small size of rulec-gec. the problem of scarce training data for ged can be approached by using pre-trained language models. @xcite explored the ability of a lstm model trained without grammatical supervision to detect grammatical errors by performing an unsupervised cloze test. the authors use a dataset of sentence pairs: an error-free original and an erroneous one. the erroneous sentence can be built manually or automatically, and differs from the original by only one word-the target position. they feed complete sentences into the model, collect all predictions for the target position, and compare the scores assigned to the original correct word and the incorrect one, e.g., write vs. writes. errors should have a lower probability than correct forms. the lm performs much worse than supervised models, especially in case of long syntactic dependencies @xcite @xcite . this work was done on italian, hebrew, and russian. @xcite adapted the described evaluation methods and applied them to pre-trained bert models by masking out the target words. bert showed high scores on all test cases with subjectverb agreement and reflexive anaphora, except for sentences with relative clauses. the experiments were extended by @xcite by evaluating the openai generative pre-trained transformer (gpt) of @xcite . bert outperformed the openai gpt on the datasets from @xcite and @xcite , but not on the dataset from @xcite . the problem of data scarcity can be addressed by generating artificial training data. among the existing approaches are oversampling a small learner corpus @xcite , utilizing additional resources, such as wikipedia edits @xcite , or introducing natural and synthetic noise into error-free data @xcite . natural noise means harvesting naturally occurring errors from the available corpora and creating a look-up table of possible replacements. using natural noise also tries to imitate the distribution of errors in the available learner corpora. synthetic noise can be generated by probabilistically injecting characterlevel or word-level noise into the source sentence, as shown in @xcite @xcite . synthetic error generation based on confusion sets extracted from a spellchecker was used by one of the top-scoring systems at the restricted and the low resource tracks at the bea-2019 shared task @xcite . both tracks suppose limited use of available learner corpora. this method was compared in @xcite with another top scoring approach @xcite which relies on tokenbased and pos-based confusion sets extracted from a small annotated sample of the w&i +locness dataset @xcite . extensive evaluation showed that the methods are better suited for correcting different types of errors. in general, the token-and pos-based pattern method demonstrated stronger results. if enough training data is available, errors can be generated by back-translation from correct data to data with errors (reverse error correction), which can be modified by additional random noise @xcite @xcite . ## data first, we describe our real learner data. this data was used as the test set for all experiments presented below. then, we present the method for generating ungrammatical data for training. ## models we explore two ways to tackle the problem of scarce data: 1. use a lm in an unsupervised fashion to detect grammatical irregularities; 2. train ged models with supervision on synthetic data. ## results and discussion bert as mlm: we calculated balanced accuracy scores bacc for both classes-grammatical errors and alternative-correct answers (see the results for the one-target approach are not strictly comparable with the results in @xcite because our data includes grammatical errors in many syntactic relations, not only in subjectverb agreement or reflexive anaphora. however, we can conclude that the pre-trained bert models capture syntactic-sensitive dependencies markedly worse for russian than for english, especially if multiple target words are masked. exploring different layers showed that the lower layers of the pre-trained model are weaker at detecting the errors. figure 1 presents histograms and kernel density estimation over log-probabilities that bert as mlm assigns to a. errors, b. correct words sampled randomly from the learner corpus (which were not exercised) and c. alternative-correct answers. these three groups of words clearly have different distributions, but they are not easily separable to assess the learner answers in a reliable fashion. fine-tuned bert: on the target positions, the fine-tuned models perform better with 8 layers, regardless of the loss function, which is consistent with experiments for english @xcite ). 9 performance on the target positions is worse than for all tokens because all 9 results with 6 layers are the worst for all models and are not reported in ## conclusions and future work we present a study on assessing grammatical correctness in the context of language learning. our focus is on assessing alternative-correct answers to cloze exercises-such answers are given more frequently by the more advanced learners. this work was done with the russian version of revita, using a learner corpus collected automatically and annotated manually. we release the corpus to the research community with this paper. the motivation behind approaching the problem of alternative-correct answers as ged is based on the hypothesis that models for error detection can assess the correctness of potentially valid answers. because learner data is limited, we experimented with pre-trained bert as a mlm, and with several bert models fine-tuned on synthetic data, which we generated for the task. the evaluation shows that the pre-trained bert is not able to assess grammatical correctness of learner answers; the performance for russian is considerably lower than for similar experiments with english. comparison with a baseline model and evaluation on another leaner corpus demonstrates that fine-tuning on synthetic data is a promising approach and generalizes well. we plan to improve the generation of synthetic data based on error analysis, to cover a wider range of error types, and continue work on estimation of the confidence of the model predictions, since it is critical to provide reliable feedback to the learners. we also plan to specify the positions of answers as part of the model's input, which is natural for the exercise-oriented set-up in revita.