Beyond E nglish: Evaluating LLM s for A rabic Grammatical Error Correction (2023)

Abstract: Large language models (LLMs) finetuned to follow human instruction have recently exhibited significant capabilities in various English NLP tasks. However, their performance in grammatical error correction (GEC), especially on languages other than English, remains significantly unexplored. In this work, we evaluate the abilities of instruction finetuned LLMs in Arabic GEC, a complex task due to Arabic’s rich morphology. Our findings suggest that various prompting methods, coupled with (in-context) few-shot learning, demonstrate considerable effectiveness, with GPT-4 achieving up to 65.49 F1 score under expert prompting (approximately 5 points higher than our established baseline). Despite these positive results, we find that instruction finetuned models, regardless of their size, are still outperformed by fully finetuned ones, even if they are significantly smaller in size. This disparity highlights substantial room for improvements for LLMs. Inspired by methods used in low-resource machine translation, we also develop a method exploiting synthetic data that significantly outperforms previous models on two standard Arabic benchmarks. Our best model achieves a new SOTA on Arabic GEC, with 73.29 and 73.26 F1 on the 2014 and 2015 QALB datasets, respectively, compared to peer-reviewed published baselines.

## introduction as interest in second language learning continues to grow, ensuring the accuracy and effectiveness of written language becomes increasingly significant for pedagogical tools and language evaluation @xcite . a key component in this respect is grammatical error correction (gec), a sub-area of natural language generation (nlg), which analyzes written text to automatically detect and correct diverse grammatical errors. figure 1 shows an instance of gec from @xcite . despite the growing attention to gec, it is predominantly studied within the english language. extending gec systems to other languages presents significant challenge, due to lack of high-quality parallel data and/or inherent challenges in these languages. recognizing this, our work focuses on arabic. in addition to being less-explored for gec @xcite @xcite @xcite , arabic has complex grammar and rich morphology that present significant challenges and further motivate our work. focusing primarily on english, the field of gec has witnessed significant advancements, specifically with the emergence of sequence-tosequence (seq2seq) @xcite and sequence-to-edit approaches (seq2edit) @xcite achieving sota results in the @xcite and the bea-2019 shared task @xcite , respectively. in spite of the efficacy of these approaches, they rely heavily on large amounts of labeled data. this poses issues in low-resource scenarios @xcite . yet, scaled up language models, aka large language models (llms) have recently demonstrated remarkable potential in various nlp tasks. the core strength of llms lies in their capacity to gen-eralize across a wide range of languages and tasks, and in-context learning (icl), enabling them to handle various nlp tasks with just a few examples (i.e., few-shot learning). a key strategy for llms is instruction fine-tuning, where they are refined on a collection of tasks formulated as instructions @xcite . this process amplifies the models' ability to respond accurately to directives, reducing the need for few-shot examples @xcite @xcite . given the ability of llms to adeptly address the low-resource challenge, we investigate them in the context of gec. focusing primarily on chatgpt, we examine the effectiveness of various prompting strategies such as few-shot chain of thought (cot) prompting @xcite and expert prompting @xcite . our research extends the realm of gec research by concentrating on the unique challenges posed by arabic. drawing upon the work of @xcite , we frame these challenges within the context of a low-resource mt task. we then carefully conduct a thorough comparison of the different methodologies employed in addressing gec in arabic. our key contributions in this paper are as follows: the rest of this paper is organized as follows: in section 2, we review related work with a particular emphasis on arabic. in section 3, we outline our experimental setups. we present our experiments on llms and prompting strategies in section 4. in section 5, we introduce our seq2seq approach along with data augmentation techniques; section 6 discusses our seq2edit approach. in section 7, we conduct a comprehensive analysis of our best model. we discuss our results in section 8, and conclude in section 9. ## llms and prompting techniques this section outlines our experiments designed to instruction fine-tune llms and explore different prompting methods for chatgpt in the context of agec. we begin by experimenting with various prompting strategies using chatgpt, comparing its performance against smaller llms and our listed baselines. we evaluate the performance of chatgpt-3.5 turbo (chatgpt) and gpt-4, under two prompting strategies: few-shot cot @xcite and expert prompting @xcite . we now describe our prompting strategies. ## data augmentation motivated by the significant improvements observed in low-resource gec tasks in languages such as german, russian, and czech through synthetic data @xcite , and recognizing the recent efforts to develop synthetic data for agec @xcite , we experiment with three distinctive data augmentation methods. ## conclusion this paper provided a detailed exploration of the potential of llms, with a particular emphasis on chatgpt for agec. our study highlights chat-gpt's promising capabilities, in low-resource scenarios, as evidenced by its competitive performance on few-shot setttings. however, arat5 v2 and arabart still exhibit superior results across various settings and error types. our findings also emphasize the role of high-quality synthetic data, reinforcing that both quantity and quality matter in achieving optimal performance. moreover, our work unveils trade-offs between precision and recall in relation to dataset size and throughout all the other experimental settings. these insight, again, could inform future strategies for improving gec systems. although our exploration of chatgpt's performance on agec tasks showcases encouraging results, it also uncovers areas ripe for further study. notably, there remains significant room for improvement in gec systems, particularly within the context of low-resource languages. future research may include refining prompting strategies, enhancing synthetic data generation techniques, and addressing the complexities and rich morphological features inherent in the arabic language.