LLM -based Code-Switched Text Generation for Grammatical Error Correction (2024)

Abstract: With the rise of globalisation, code-switching (CSW) has become a ubiquitous part of multilingual conversation, posing new challenges for natural language processing (NLP), especially in Grammatical Error Correction (GEC). This work explores the complexities of applying GEC systems to CSW texts. Our objectives include evaluating the performance of state-of-the-art GEC systems on an authentic CSW dataset from English as a Second Language (ESL) learners, exploring synthetic data generation as a solution to data scarcity, and developing a model capable of correcting grammatical errors in monolingual and CSW texts. We generated synthetic CSW GEC data, resulting in one of the first substantial datasets for this task, and showed that a model trained on this data is capable of significant improvements over existing systems. This work targets ESL learners, aiming to provide educational technologies that aid in the development of their English grammatical correctness without constraining their natural multilingualism.

## introduction code-switching (csw), the practice of fluidly alternating between two or more languages in conversation, has become commonplace in recent years. this linguistic phenomenon, emerging as a natural consequence of multilingualism, is now widely accepted in social and professional settings @xcite . many works have highlighted the utility and cultural importance of csw in general conversation @xcite . further research indicates that these advantages extend to language learning, with csw offering many pedagogical benefits. these include increasing students' access to content and improving their confidence. @xcite discuss the mechanisms for this, where students use a familiar language to grasp foreign, complex concepts. csw can also serve as a scaffolding tool, helping to bridge gaps in a student's comprehension of a language and enabling them to build upon existing knowledge. these benefits reduce the barriers between a student and their target language and help promote a learning environment conducive with active exploration and deeper understanding. therefore, it is essential that english as a second language (esl) learners are not penalised for expressing their cultural identity through csw. grammatical error correction (gec) is the task of automatically detecting and correcting errors in text. research on gec for csw text remained largely unexplored. @xcite were the first to demonstrate that exposing a sequence-tagging gec model to csw text during the training process improves performance compared to a monolingual system. however, further work is essential to ensure language technology is inclusive and reflective of real-world linguistic practices. figure 1 shows two examples of csw from our target population with their grammatical corrections. 1 despite significant advancements in gec in recent years, a gap persists in addressing csw texts, with monolingual gec datasets labelling csw as a type of error @xcite . there are several reasons for this, the most prominent being the scarcity of high-quality training data, a problem that plagues monolingual gec systems. the unique linguistic features of csw, including its variable syntax, semantics and pragmatics, add additional complexity to this task. monolingual seq2seq gec models, e.g. t5 @xcite , struggle with csw text as they fail to represent the non-english inputs, resulting in their inability to output the csw text. on the other hand, multilingual seq2seq models and edit-based gec models like gector @xcite can handle csw text but struggle with the ambiguity present at language switching points. this ambiguity challenges the models' ability to accurately correct the text. this paper aims to bridge this gap. firstly, to address the data scarcity issue, we propose a method for generating high-quality synthetic csw gec data, using which we produce, to our knowledge, one of the first substantial datasets labelled for this task 2 . secondly, we train a token classificationstyle gec system, tailored to correct errors in texts produced by esl learners. this demographic is significant for our study as they not only present consistent csw patterns but also stand to benefit greatly from a gec system capable of handling csw text. ## csw gec systems for our gec system targeting csw texts, we chose a gector model @xcite , with a roberta-base foundation, due to its proven efficacy with limited training data and stronger performance on csw texts compared to seq2seq models. we added a new csw class to the error detection head, adding the ability to detect csw tokens. following @xcite , we used a 3-stage training schedule. in the first, we used the same distilled 1bw corpus, and added all our synthetic csw gec data. for the second, we used several gec datasets: nucle @xcite , fce @xcite , w&i locness @xcite , lang-8 and our 2 synthetic csw datasets. as our genuine csw dataset is a subset of the lang-8 corpus, we checked and removed any duplicates. following previous works, we finished training using the w&i locness dataset due to its superior quality. in this final stage, we added a sampled subset of our synthetic csw sentences and 90% of our authentic csw data, ensuring exposure to synthetic and genuine csw text. at each stage, we reserved 5% of the data for validation. 9 finally, we tuned inference parameters using a grid-search to optimise the f 0.5 on the final validation set. by beginning with pretraining on large amounts of lower-quality data in the early stages, this multi-stage learning process allows the model to first build a robust gec foundation before refining it with high quality data in the latter stages. this approach allows the model to learn incrementally, reducing the risk of the model being overwhelmed by the complexity of the task from the outset. ## conclusion the primary aim of this paper was to build a gec system capable of effectively correcting english errors in csw text, whilst maintaining competitive performance on monolingual data. to address the scarcity of csw data, we explored methods of generating synthetic csw text. we used several csw metrics to establish that the llm promptingbased approach was the most capable of generating text resembling the content in our genuine dataset. from there, we used two error injection methods to create the first substantial datasets labelled for csw gec. this significantly expanded the training data available. importantly, it also opened up 11 error type analysis of our model is given in appendix e.