No Error Left Behind: Multilingual Grammatical Error Correction with Pre-trained Translation Models (2024)

Abstract: Grammatical Error Correction (GEC) enhances language proficiency and promotes effective communication, but research has primarily centered around English. We propose a simple approach to multilingual and low-resource GEC by exploring the potential of multilingual machine translation (MT) models for error correction. We show that MT models are not only capable of error correction out-of-the-box, but that they can also be fine-tuned to even better correction quality. Results show the effectiveness of this approach, with our multilingual model outperforming similar-sized mT5-based models and even competing favourably with larger models.

## introduction grammatical error correction (gec) systems are a vital link between expert language use and clear communication, enhancing writing skills and language learning. however, gec research has primarily focused on the english language with much less coverage for other languages, resulting in english-oriented methodologies and data scarcity for other languages. this highlights the need to diversify gec research, ensuring that the benefits of these systems extend to all languages for a more inclusive global linguistic landscape. in the evolving multilingual and non-english grammar error correction (gec) landscape, two recent notable directions have risen: the utilization of synthetic data @xcite and the integration of pretrained models, particularly the multilingual textto-text transformer model (mt5) @xcite . the use of mt5 extends to correcting grammar in various specific languages, including ukrainian, icelandic, and lithuanian @xcite @xcite , and serves as an inspiration for other multilingual research @xcite . however, achieving substantial performance enhancements beyond training basic transformer models necessitates further adjustments, such as the incorporation of high-quality synthetic data, additional information, or the utilization of significantly larger models. we demonstrate that building upon similarly sized multilingual machine translation (mt) models is more effective than fine-tuning mt5 (kementchedjhieva and søgaard, 2023). previous studies have shown the value of information obtained through machine translation as training data or additional hypotheses @xcite @xcite . we revisit the concept of utilizing zero-shot translation for error correction @xcite , developing the idea further. we demonstrate that massively multilingual mt models can function as multilingual gec models and can be improved further via fine-tuning to error correction data. this approach underscores the potential of multilingual mt models as an even simpler yet effective gec system, allowing for the integration of standard practices in gec research. in doing so, we highlight that multilingual mt models acquire valuable information for grammatical error correction and it is possible to leverage this knowledge during training. in our work, we experiment with four languages: english, german, and czech for the purpose of comparison with other multilingual studies, plus estonian, an underexplored language in terms of error correction with a similarly limited publicly available dataset. as a result, our model achieves higher scores than work based on similar-sized mt5 models and performs competitively with even significantly larger models. since large language models have recently showed good performance in several nlp tasks via prompting, we also assess gpt-4's performance on the gec task for the four included languages for comparison. while more sophisticated prompts may lead to improved results, results shown by gpt-4 are worse than state-of-the-art gec results, and our best results also surpass its performance. thus, our main contributions are: ## related work the connection between grammatical error correction (gec) and machine translation (mt) has been significant since junczys-dowmunt et al. in the state-of-the-art english gec, a different paradigm emerged, with the use of sequence tagging rather than sequence generation. this approach, initially introduced by @xcite , employs various transformer encoders for tagging errors within sentences and then replaces these parts with corrections. while this approach has proven effective for english, attempts to apply it to other languages have yielded less impressive results compared to sequence generation methods @xcite . lately several massively multilingual machine translation models have been released, including m2m-100 @xcite , nllb @xcite and madlad-400 @xcite . in our experiments we make heavy use of the nllb models. finally, most recently, large language models have shown capability to correct errors via prompting @xcite @xcite . reported results mostly fall behind gec-specific approaches. ## methodology our methodology is centred around exploiting the zero-shot translation capabilities of multilingual translation models applied to the gec task. we also explore fine-tuning the translation models on parallel data, synthetic error data and humanannotated error correction data yielding improved performance. finally, we explore the combination of parallel and error correction data, showing that the benefits of both tasks (translation and error correction) can be combined. ## experimental setup this section presents an overview of our experimental setup, covering data sources, models, and evaluation metrics, providing insights into the technical details of our work. ## results we first describe the results of our experiments related to mixing data during pre-training, then show how different data and pre-training affect the model's behaviour and, lastly, we benchmark our models with comparable and state-of-the-art research solutions and gpt-4 performance. ## discussion our tuned multilingual mt models consistently have higher f 0.5 -scores than mt5-based approaches. in addition to mt5-based works, our approach outperforms or achieves comparable f 0.5scores with gpt-4 in a zero-shot setting for all the languages we tested. it surpasses gpt-4 with a larger margin for english, german, and czech and gets comparable performance for estonian. however, gpt-4, being a large general-purpose model, is not practical for real-time gec due to its current quality, availability, and speed. therefore, we have not explored few-shot prompts or fine-tuning options for gpt models at this time. our evaluation relies on a reference-based metric, which tends to reward minimal alterations to the text and may not always align with human judgements @xcite @xcite . this approach could bias evaluations in favour of more conservative systems that make fewer edits and be unfair to the mt model's zero-shot translation and gpt models that tend to alter text more. consequently, the 75:25 mixing ratio we selected might not be universally applicable across all languages, as evidenced by its performance with estonian, among other scenarios. our approach is adaptable, allowing for the creation of systems capable of extensive rephrasing to correct a wider range of errors, as well as those that are more conservative in their edits by changing the data ratio. another point to note is that multilingual training presents both advantages and complexities. it demonstrates effectiveness for languages that are well-represented in the translation model, while languages with limited representation may not experience such clear benefits. this disparity may be attributed to their weaker zero-shot performance, indicating that they have more to learn from synthetic data. to address this, a potential solution could involve more extensive pre-training or initial training with select translation data. this approach may negatively impact other languages, as indicated by decreasing english and german scores for zero-shot translation with balanced translation training. regarding future work, our work focused on one mt system as a starting point for building a gec system, but there is much to explore. future research can explore different models and sizes, improve data balance during pre-training, use better synthetic data, and refine fine-tuning strategies. a recent study, madlad-400 @xcite , has already covered twice as many languages, indicating a promising direction for further investigation and language coverage. ## conclusion we propose a simple approach for a multilingual gec system, simplifying the creation of non-english gec solutions. through the use of multilingual machine translation models supplemented with synthetic and error correction data, we have presented an effective approach to enhancing gec performance. our results reveal the superiority of this method, with our multilingual model consistently outperforming similar-sized models and even competing with larger counterparts.