Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors (2022)

Abstract: In this paper, we explore the capacity of a language model-based method for grammatical error detection in detail. We first show that 5 to 10% of training data are enough for a BERT-based error detection method to achieve performance equivalent to what a non-language model-based method can achieve with the full training data; recall improves much faster with respect to training data size in the BERT-based method than in the non-language model method. This suggests that (i) the BERT-based method should have a good knowledge of the grammar required to recognize certain types of error and that (ii) it can transform the knowledge into error detection rules by fine-tuning with few training samples, which explains its high generalization ability in grammatical error detection. We further show with pseudo error data that it actually exhibits such nice properties in learning rules for recognizing various types of error. Finally, based on these findings, we discuss a cost-effective method for detecting grammatical errors with feedback comments explaining relevant grammatical rules to learners.

## introduction recent studies have shown that masked language models pre-trained on a large corpus (hereafter, simply language models) achieve tremendous improvements over a wide variety of natural language processing tasks with fine-tuning. these results suggest that they are also effective in recognizing erroneous words and phrases, the task known as grammatical error detection. there has been, however, much less work on this aspect of grammatical error detection than in other tasks. one can argue that since language models are trained on language data produced by native speakers of a language (specifically, english in this paper), they might not work well on the target language data produced by non-native speakers of that language. in other words, english language models do not know at all about grammatical errors made by non-native speakers. even apart from grammatical errors, the target language is different from the canonical english, meaning that it contains unnatural phrases and characteristic language usages that native speakers do not normally use, as @xcite demonstrate. these differences might affect performance of language model-based methods in grammatical error detection. actually, researchers have reported on performance of language models on grammatical error detection and correction. @xcite and @xcite have shown that bert @xcite -based methods improve grammatical error detection performance in chinese and english, respectively. @xcite and @xcite have shown a similar tendency in grammatical error correction. while these studies empirically prove the effectiveness of language models in grammatical error detection and correction, the questions of why and where language models benefit error detection/correction methods are left unanswered. in this paper, we explore this aspect of language models in grammatical error detection to better answer the research questions. we first show that a bert-based method incredibly quickly learns to recognize various types of error as summarized in figure 1 ; it achieves only with 5 to 10% of training data an f 1.0 that a non-language model-based method can achieve with the full training data (the details will be described in sect. 4). this implies that the bert-based method (i) should have a good knowledge of the grammar required to recognize certain types of error and (ii) can transform it into error detection rules by fine-tuning with very few training samples. following this, we further show with real and pseudo error data why and where it gains in error detection, revealing the insights of language model-based methods. for instance, we show that the bert-based method trained on few instances of a transitive verb with a preposition (e.g., *discuss about) can detect the same type of error in other verbs (e.g., *approach to and *attend in). finally, based on these findings, we discuss a cost-effective method for detecting grammatical errors with feedback comments explaining relevant grammatical rules to learners. there has been a long history of studies that investigate the linguistic knowledge of language models including the work by @xcite ; ettinger (2020); @xcite to name a few. a popular approach is to test whether a language model assigns higher likelihood to the appropriate word than an inappropriate one, given context. the linguistic knowledge to be explored ranges from syntactic/semantic knowledge to common sense. these studies mostly use (i) synthetic test data: sentences that are generated synthetically by using a certain kind of template or (ii) perturbed test data: sentences that are generated by perturbing a natural corpus. our work is different from these previous studies in two points: (i) to our best knowledge, we examine linguistic phenomena that have never been explored before in the conventional studies (e.g., subjects marked with a preposition and errors involving the usages of transitive and intransitive verbs); (ii) we use a real learner corpus with real errors as our test data. @xcite examine if an encoderdecoder neural network for grammatical error correction (not bert-based) can learn the knowledge of grammar through the task of grammatical error correction. they target five error types: subject-verb agreement, verb form, word order, adjective/adverb comparison, noun number. they use both synthetic and real learner data. they report a negative answer to the research question except for word order errors. they also report that their model learns the knowledge to detect the target errors in their synthetic data. however, there is still room for debate in this argument because error positions tend to be rather obvious in their synthetic data (e.g., adjective forms erroneously used as adverbs almost always appear at the end of a sentence). our study expands and deepens their findings for a wider variety of error types that are much more difficult to detect (in that it requires a much wider range of linguistic knowledge including pos, lexical, and syntactic knowledge). ## performance on real data figure 1 (see the second page) to figure 3 show the relationship between the number of training sentences and f 1.0 in fce 6 , nucle, and bea with its corresponding precision-recall curves, respectively. all f 1.0 graphs show the high generalization ability of the bert-based method. they also show that the bert-based method exhibits a performance saturation at a very early stage (1,000-3,000 training sentences). it is worthwhile to point out the fact that the f 1.0 curves for all proficiency levels (a, b, and c) 7 in bea (figure 3 ) exhibit the same trend as the other corpora although the absolute performances differ depending on the levels. these results empirically confirm that the high generation ability of the bert-based method holds in various writer populations. unlike the bert-based method, the bilstmbased method improves steadily as the number of training sentences increases although even with the full training data, it only achieves an f 1.0 that the bert-based method can achieve with only 500 or less training sentences. also, the bert-based method without bert training does not perform well at all. this is probably because it requires much more degrees of freedom in terms of the network parameters to learn rules for detecting a wide variety of grammatical errors, which have a certain degree of complexity. to investigate the results from a different point of view, let us now consider precision-recall curves 6 the best performance of the bert-based method in fce is a recall and precision of 0.455 and 0.628, respectively. these numbers are considerably lower than those of the state-of-the-art @xcite report; they show that the electra-based method achieves the best recall and precision of 0.505 and 0.821, respectively (c.f., recall: 0.480, precision: 0.757 for their bert-based method). note that they use the large models whereas we use the bert-base model, which should be part of the reason for the performance differences. while our results do not achieve the best results, they explain well why large-scale masked language models perform well in grammatical error detection. 7 the proficiency levels a, b, and c in bea, which corresponds to those in cefr, becomes higher in this order. (the right graphs; the numbers at the edges of each curve correspond to the numbers of minimum and maximum training sentences and the plots are lined in minimum to maximum order). all precision-recall curves show that only the bertbased method quickly improves in recall as the number of training sentences increases while the bert-based and bilstm-based methods both improve in precision. in other words, only the bertbased method learns to recognize various error types with little exposure to error examples. figure 4 shows recalls per detailed error types in fce 8 where error types are automatically obtained by using errant. figure 4 shows that the bertbased method quickly achieves a good recall in spell (spelling errors) while the bilstm-based 8 punct, other and those whose frequency is less than 150 are excluded in figure 4 . method shows a much milder rise. this is not surprising considering that bert is trained on large native corpora. namely, it virtually has a large vocabulary list and knows about english spellings well. more interestingly, it exhibits a performance saturation at a very early stage (500-3,000 training sentences) in all errors, resulting in log-like-shape curves while the bilstm-based method improves rather linearly (except for spell). even more interestingly, it shows a sharp rise in recall in det (determiner errors) and noun:num (errors in noun number). the notion of noun countability with pos plays an important role in detecting these two types of error as in *i am student/countable. and an evidence/uncountable. this suggests that bert contains some kind of knowledge corresponding to noun countability and pos (singular/plural nouns). more generally, from these, one 0 2000 4000 6000 8000 10000 12000 # of #e te ce# 0.0 0.2 0.4 0.6 0.8 1.0 f 1.0 part-time job bilstm bert-ba#ed (w/o bert trai ) bert-ba#ed (w/ bert trai ) 100 100 12163 100 12163 12163 figure 5: detection performance in relation with training size (icnale, in-domain test data). figure 5 and figure 6 show performance in part-time job in icnale; in figure 5 , all models are trained on essays on part-time job and tested on (test) essays on the same topic (indomain setting) while in figure 6 , they are trained on smoking and tested on part-time job (out-of-domain setting). in both settings, 300 to 500 training sentences are again enough for the bert-based method to rival the bilstm-based method with the full training data, which exhibits again a linear improvement in f 1.0 . also, only the bert-based method quickly improves in recall. a closer look at the detection results reveals that many of the errors require linguistic knowledge including pos, syntactic relations, and lexical properties such as transitive/intransitive verbs. for example, "preposition + subject" errors, which were introduced in subsect. 3.1, require the notions of pos such as verbs and syntactic relations such as subjects. similarly, the distinction between transitive and intransitive verbs play a crucial role in "transitive verb + preposition" and "intransitive verb + object". the fact that the bert-based method can detect these types of errors with a few training instances suggest that it has an access to grammarlike knowledge and that it can turn the knowledge into error detection rules by fine-tuning. we will explore this in more detail in the following section. these results also shed light on an important aspect of the bert-based method in practice. namely, a cost-effective way of developing an error detection system based on bert would be to create a small amount of training sentences (e.g., 1,000) for each essay topic; according to the above figures, the gain would be much smaller after 1,000 training sentences. of course, the results are only for a simple bert-based method. no one knows how differently the performance curves grow with different architectures and/or with a much larger set of training instances. it will be interesting to investigate these points for future work. ## performance on synthetic data in the previous section, we have seen that the bertbased method has a much higher generalization ability in grammatical error detection. to analysis this phenomenon in detail, we now turn to detection performance of the bert-based method on the pseudo error data. as described in subsect. 3.1, we train it on the ten sets of training data (i.e., 2, 4, • • •, 1024 training sentences for each error type) and test the trained models on the fixed test set. doing so, we estimate the relationship between the number of training sentences and detection performance for each error type. figure 7 shows the relationship between the size of training data and f 1.0 for each error type where the size is measured by the number of sentences. figure 7 reveals that the bert-based method already recognizes some of the target errors at early stages (even with two or four training sentences). performance goes much higher even with eight training sentences in most of the error types with an exception of the error type "intransitive verb + object". for instance, the bert-based method recognizes more than half of the "preposition + subject" errors with a precision of 0.800 only with eight training instances. this implies again that bert has certain knowledge about the notions of pos such as verbs and syntactic relations such as subjects; otherwise, it would be difficult to achieve the same argument applies to "transitive verb + preposition" and "intransitive verb + object". it should be emphasized that the bert-based method has to detect errors in verbs that never 9 appear in the training data; recall that there is no overlap of the target transitive/intransitive verbs in the training and test data. in other words, the bert-based method can recognize unseen erroneous combinations, for example, *visited in atlanta (transitive verb + preposition type) and *specialized environ-mental litigation (intransitive verb + direct object error type) after just seeing *mention in, discussed about and *were related drugs and belongs lon's grandmother. these training and test sentences have almost nothing in common except that they are the combinations of transitive/intransitive verbs and prepositions/objects. besides, the fact that correct combinations of other verbs and prepositions/objects often appear in the test data makes the task even more difficult without the knowledge of pos and syntactic relations. these findings support the hypotheses that bert has linguistic knowledge and that it can turn the knowledge into error detection rules by fine-tuning. ## exploration for cost-effective error detection with feedback comments the findings we have obtained so far bring out the possibility that one can implement with few training instances a system that accurately detects grammatical errors and recognizes their detailed error types. for example, manually or automatically, by creating few instances of the erroneous combination of transitive verbs and prepositions as we saw in the previous sections (e.g., *discuss about), one can develop a system detecting the same type of error in other transitive verbs and prepositions (e.g., *attend in it). with the detailed error types, the system can also output feedback comments to the user such as transitive verbs do not take a preposition. instead, they take a direct object instead of just indicating them as preposition errors. as a pilot study, we trained the bert-based method on the pseudo error data and tested it on the real (learner) data to examine the above possibility. to achieve it, we manually annotated the real data with the target five error types based on the feedback comments available in icnale. figure 8 shows the results. it reveals that the bert-based method trained on the pseudo data does not perform on the real data as well as on the pseudo data. performance growths stop at an early stage (around eight training sentences). a possible reason for this is that in the real data, multiple errors often appear in a sentence. also, multiple errors in a sentence can range over multiple types of error. besides, the error rate is much lower in the real data than in the pseudo error data where one error occurs per sentence except 200 error-free sentences (although multiple types of error appear in the whole data set). these conditions make the having said that, the results shown in figure 7 still encourage us to develop language model-based systems with a small amount of in-domain training data in order to detect grammatical errors with detailed error types. one possible way to achieve it is to sample sentences from unannotated essays written on the target topic, and then to annotate them with the specific error types that the developer wants to give feedbacks to learners. this will naturally mitigate the problems caused by the multiple-type multiple error situation and the error rate difference. one can also manually create sample error sentences (and their correct versions). ## conclusions in this paper, we have explored the capacity of a large-scale masked language model to recognize grammatical errors. our findings are summarized in the following three points: (1) experiments with the real learner data show that a bert-based error detection method has a much higher generalization ability in grammatical error detection than a nonlanguage model-based method, and the first performance saturation comes at the point of around 1,000-3,000 training instances; (2) it starts to recognize the target errors involving a wide variety of grammatical knowledge with very few instances of them; (3) the high generalization ability brings out its potential for developing systems that detect and explain grammatical errors with very few training instances.