Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9 B parameters: A Comparative Study of 17 Models (2025)

Abstract: Recent language models can successfully solve various language-related tasks, and many understand inputs stated in different languages. In this paper, we explore the performance of 17 popular models used to correct grammatical issues in texts stated in English, German, Italian, and Swedish when using a single model to correct texts in all those languages. We analyze the outputs generated by these models, focusing on decreasing the number of grammatical errors while keeping the changes small. The conclusions drawn help us understand what problems occur among those models and which models can be recommended for multilingual grammatical error correction tasks. We list six models that improve grammatical correctness in all four languages and show that Gemma 9B is currently the best performing one for the languages considered.

## introduction grammatical error correction (gec) is one of the most practical tasks in the natural language processing (nlp) field. being able to use computers to detect and fix grammatical errors and spelling mistakes is especially beneficial for language learners and professional writers. recent years have shown great promise in using large language models (llms) for various nlp-related tasks, including machine translation, text generation, or text classification. these models, trained on vast amounts of data, learn to predict the most probable continuation of a given sequence. assuming large corpora are used to train these models, the models should encounter instantiations of various tasks, including questions to be answered followed by their answers, texts expressed in one language followed by their translations, long fragments of texts followed by their summarizations, or grammatically incorrect sentences fixed by some experts as part of the text's continuation. all of these lead to the increasing ability of llms to address human requests. recently published llms are bigger and trained on more data, which enables them to solve more sophisticated tasks with better results. even though llms are currently suboptimal in some tasks, it seems that soon they may become the dominant solution for most nlp-related problems @xcite . however, due to llms' large sizes and high computational costs, researchers focus on training smaller models of quality similar to bigger ones @xcite . in this paper, we aim to explore the abilities of llms to solve the gec task when dealing with several languages at once, including three highly popular ones: english (en), german (de), and italian (it), as well as a less popular -swedish (sv). having one common model for multiple languages may be beneficial in various ways. using specialized models for each language requires a lot of storage space, which increases the costs of products and hinders the synergy between languages, which is frequently observed in real life. the energy consumption related to using multiple models increases the costs of serving models and impacts our natural environment. to mitigate these issues, smaller models that can be run locally and are more environmentally sustainable may be considered @xcite . these smaller models are easier to control, as they can be loaded, analyzed, and fine-tuned on personal computers, so here, we focus on using single llms of moderate size (up to 9b parameters) as they can be loaded on consumer-grade gpus. in this paper, we aim to address the following research questions: rq1: which model is the best for multilingual grammar correction considering english, german, italian, and swedish? rq2: do models preserve the original input when no errors are present? rq3: which type of prompt is more effective: short and general, or longer and more specific? ## related works the research on the gec problems has experienced substantial progress in the last years. recent surveys highlight a shift from traditional rulebased methods, statistical classifiers, and statistical machine translation techniques to neural machine translation, emphasizing the superior performance of neural approaches @xcite . however, there are challenges associated with the predominant use of supervised learning in gec, primarily due to the necessity for annotated datasets. researchers in the field have reported issues such as annotation inconsistency, human error, and a scarcity of data for languages other than english @xcite . in this context, leveraging llm-based gec with zero or a few examples emerges as a promising solution. aside from unsupervised approaches, other significant challenges in gec include multilingualism, low-resource gec, and the evaluation @xcite . here, we focus on three main areas relevant to our research: the application of llms in gec, multilingual gec, and the evaluation of gec systems. ## dataset we use the multiged dataset @xcite for our experiments. originally, multiged was proposed in 2023 to evaluate the ability of ai systems to identify grammatically incorrect tokens in five languages: english, german, italian, czech, and swedish. the dataset is a compilation of various datasets that include fce for english @xcite , or falko-merlin for german @xcite . as the original dataset represents a binary classification task, where each token in a sentence is classified as correct or not, we preprocessed the dataset using moses detokenizer @xcite , to reconstruct entire sentences annotated with the information whether the whole sentence is grammatically correct or not. if any token in the sentence was annotated as incorrect, the full sentence was tagged as incorrect. there is no golden-standard corrected candidate for a given sentence provided, as the multiged dataset is a token classification task. however, following research of @xcite , our goal is to analyze the gec problem from a broader perspective trying to mitigate annotator bias that may be observed in golden standard corrections. to achieve this, we use languagetool as one of the key tools for scoring the models. as it does not support czech, we focus on english, german, italian and swedish only. to evaluate a wide range of llms, we selected the multiged's dev set to limit the costs of experiments. this step reduced ## results we calculated values for all the metrics introduced in section 4.3 using all the prompts described in section 4.2 for all 17 llms considered. then, we selected the best performing prompt, and used it to verify which llms support all the languages, and which llms work best for multilingual scenarios as well as for each language separately. prompt selection for each metric, each language, and each prompt p1, p2, p3, we calculated the average metric value over all models considered. while the detailed scores can be seen in the biggest gain of using the last prompt is seen in the f 1 metric, which checks how well llms language support analysis several llms struggle with handling languages other than english, often producing outputs (or considerable fragments of outputs) in english. models like bloom and smollm exhibit the highest drifts, converting a significant portion of german, italian, and swedish texts into english. other models, such as xglm, karen-creative, karen-strict, towerllm, mistral, and phi-3, also show considerable drift, with each producing outputs in a different language in at least a quarter of the cases. consequently, these models are not considered effective for multilingual grammatical error correction. detailed perlanguage results can be found in appendix, in tables 9, 10, 11, 12. the aggregated scores can be found in out of 17 llms considered, 9 of them (aya, eurollm 1.7b, eurollm 9b, gemma 2b and 9b, llama 3.1, openchat 3.5, qwen 2.5, and yi) produce outputs in all languages considered in a vast majority of cases. mistral and phi-3 fail to produce texts in one language -italian, while the remaining models have problems with at least two languages. since not all llms support all languages, we repeated the prompt quality analysis considering only aya, eurollms, gemmas, llama 3.1, openchat 3.5, qwen 2.5, and yi. as a result, we confirmed the previous conclusion -the p3 prompt is the best one in this scenario in 32/36 cases. the details on this analysis can be found in appendix, ranking models as our goal is to identify models that correct texts with the highest quality (maximizing correctness metric), preserving the meaning of the original text (maximizing metrics based on bert score, sentencebert and bleurt), applying possibly small changes (maximizing metrics based on levenshtein, length difference, and gleu), and estimating the ability not to change a given text when it is correct (maximizing f 1 metric), we analyzed those metrics per language and then aggregated them to obtain a general overview of these models. we left language drift aside, as it was primarily used to filter out llms not supporting all languages considered. to reach this goal, we followed a three-step analysis consisting of metric calculation, per-metric ranking creation, and global ranking creation: rankings overview the rankings presented in ## discussion the rankings presented in model size vs. quality four models considered are much smaller than the rest -phi-3 (3.7b), gemma 2b (2b), smollm (1.7b), and eurollm (1.7b). while phi-3's performance is mediocre, smollm behaves badly on the gec task. one may think that it is due to an insufficient number of parameters. however, gemma 2b (ranked foruth overall) and eurollm 1.7b (ranked 10th), which are of similar size to smollm (ranked 17) present outstanding performance considering their sizes. even though gemma 2b and eurollm 1.7b are very small, they understand all four languages. this proves that appropriate training (gemma 2b is a distilled version of larger gemma) and appropriate data (eurollms are european language-focused) are more crucial than the model size itself. recurring problems several recurring issues have been identified in the results produced by llms. the most popular examples are listed in result stability due to the non-deterministic nature of llm outputs, we also assessed the stability of the generation process to ensure that our results were consistent and that the rankings obtained in repeated runs of the experiments remained stable. for that, we selected the best-performing model (gemma 9b) and a randomly chosen one (tower-llm) and reran the experiments five times, maintaining all original parameters. although minor differences were observed, their magnitudes were negligible. across both models and all prompts, the variation in language tool scores did not significantly exceed 0.001. the largest difference for the gemma 9b model was 0.00037, obtained on the swedish dataset with prompt p1. while for the towerllm model, it was 0.00129, obtained on the italian dataset with prompt p3. similar magnitudes of deviations were observed for other metrics. based on these findings, we conclude that the results are both reliable and stable across runs. ## karen analysis: corrections vs preservation in the field of gec, there is an inherent dilemma: should a model prioritize making corrections or preserving the original text with minimal changes? when aiming to preserve the original text, it's crucial to consider both semantic faithfulness and syntax preservation. this challenge is particularly pronounced for llms, which tend to over-correct, or even paraphrase provided texts. the authors of karen models addressed this by offering two versions: "strict" and "creative." when focusing exclusively on the english language, the karenstrict model excels by making minimal changes and maintaining the highest similarity to the original text. it is ranked above the creative version across all metrics related to both semantic and syntactic preservation. despite being ranked 8th in the lt score for english, the model still achieved a high score of 0.938 (with the best model scoring 0.963). conversely, the karen-creative model produces higher quality output (with an lt score of 0.946), but the differences between the original and corrected texts are more significant. it still maintains high semantic and syntactic preservation, especially when compared to other llms. this performance can be attributed to the karen models being fine-tuned specifically for the gec task. ## conclusions in this paper, we analyzed whether popular llms not larger than 9b parameters are able to handle grammatical error correction in a multilingual manner and proposed a framework for referenceless gec llm comparison. we found that gemma (2b and 9b), eurollm (1.7b and 9b), openchat 3.5, and llama 3.1 are able to handle all analyzed languages (en, de, it, sv) with good quality. since gemma 9b is the top-ranked model in the multilingual scenario and the language-averaged lt score and f 1 -correct input preservation are the highest for it, it is the recommended model to use (answer to rq1). also, its smaller sibling -gemma 2bperforms very well. we found out that some models (gemmas, eurollm 9b, qwen, karens, aya, and openchat) preserve the original text well if no error is introduced. comparing fine-tuned models (karen strict and karen creative) with their base mistral model, we see that the fine-tuned models excel in terms of overall correction quality, which shows the importance of fine-tuning to prevent hallucinations in llms. thus, rq2 can be answered positively. finally, we found that the longest, most concrete prompt is the best performing one overall, which answers rq3. we made the corrections generated using all prompts and models publicly available online foot_6 .