Leveraging LLM s for B angla Grammar Error Correction: Error Categorization, Synthetic Data, and Model Evaluation (2025)

Abstract: Large Language Models (LLMs) perform exceedingly well in Natural Language Understanding (NLU) tasks for many languages including English. However, despite being the fifth most-spoken language globally, Grammatical Error Correction (GEC) in Bangla remains underdeveloped. In this work, we investigate how LLMs can be leveraged for improving Bangla GEC. For that, we first do an extensive categorization of 12 error classes in Bangla, and take a survey of native Bangla speakers to collect real-world errors. We next devise a rule-based noise injection method to create grammatically incorrect sentences corresponding to correct ones. The Vaiyākaraṇa dataset, thus created, consists of 5,67,422 sentences of which 2,27,119 are erroneous. This dataset is then used to instruction-tune LLMs for the task of GEC in Bangla. Evaluations show that instruction-tuning with Vaiyākaraṇa improves GEC performance of LLMs by 3-7 percentage points as compared to the zero-shot setting, and makes them achieve human-like performance in grammatical error identification. Humans, though, remain superior in error correction. The data and code are available from https://github.com/Bangla-iitk/Vaiyakarana.

## introduction grammatical error correction (gec) aims to detect and correct grammatical errors in a text automatically. for example, given the following sentence in english, "a ten year oldest boy go to school." a gec system detects errors in the use of the superlative degree and verb number and corrects it to "a ten-year-old boy goes to school.". the conll 2013 and 2014 shared tasks @xcite significantly advanced gec research, but focus primarily on english. bangla (bengali, বাঙ্গলা/বাংলা bāṅgalā 1 ) is the 1 we use iso15919 transliteration scheme for bangla: large language models (llms), particularly those trained with instruction fine-tuning, have demonstrated strong capabilities in generating synthetic data for various nlp tasks, particularly in english @xcite . in this paper, we explore the ability of llms to perform gec for bangla. for that, we first do an extensive categorization of possible grammatihttps://en.wikipedia.org/wiki/iso_15919 cal errors found in bangla into 12 error categories. we then collected real-world errors by doing an essay writing survey of native speakers. we use the distribution of real errors found from this survey (of 2,576 sentences) to generate synthetic data that contains grammatical errors. the generation is done using a rule-based noise injection scheme on grammatically correct sentences to guarantee that the generated sentences are actually grammatically incorrect. we curate a dataset, vaiyākaran . a 2 , consisting of 2,27,119 erroneous and 5,67,422 total bangla sentences to instruction-tune llms. we use the vaiyākaraṇa dataset to evaluate the ability of llms for gec. we compare the performance of llms thus instruction-tuned with vaiyākaraṇa against both zero-shot performance as well as human evaluators. we also assess their performance for another task, namely, paraphrasing. experiments show that using vaiyākaraṇa improves llm performance by approximately 3-7 percentage points compared to the zero-shot setting for both gec and paraphrasing tasks. however, humans still significantly outperform llms in correcting grammatically erroneous sentences. our key contributions in this paper are: 1. we do an extensive categorization of possible grammatical errors in bangla into 12 distinct categories (sec. 3), making this the first such extensive attempt. standardization of grammatical error categories is essential for addressing challenges in low-resource language @xcite . this categorization of errors is applicable for many major indian languages (appx. c). 2. we collect and analyze 2,576 human-annotated sentences to identify common grammatical errors. we also present statistics on error distributions based on real-world usage (sec. 4). 3. we propose a structured error generation approach (sec. 6) for systematically injecting grammatical errors into bangla sentences. this approach allows for scalable generation of error-annotated corpora. our approach is extendable to other indian languages (appx. c). 4. we curate vaiyākaraṇa, a dataset containing 2,27,119 erroneous bangla sentences, which can be directly used instruction-tuning llms for bangla gec. 5. we evaluate the effectiveness of instruction2 the word means "grammarian" in bangla. ## related work grammatical error correction (gec) for indian languages, including bangla, is still in its early stages. while existing methods have explored rule-based, statistical, and neural approaches, they lack comprehensive error categorization and robust datasets. early works on gec focused on rule-based and statistical models. @xcite categorized inflectional errors for hindi gec, and rachel et al. (2023) proposed vyakaranly, a toolkit for hindi grammar correction. alam et al. (2007) introduced a rule-based statistical approach, but it failed to generalize beyond simple sentences. islam et al. (2018) attempted to generate erroneous bangla sentences via random word swaps, insertions, and deletions. however, this method of generating wrong sentences does not always give the desired result as shown in table a1 of appx. a. rahman et al. (2023) developed a cnn-based spelling correction model, while oshin et al. (2023) curated a 10k-sentence dataset for bangla text error classification, with only 3,140 erroneous sentences. @xcite proposed panini, a vaswanistyle monolingual transformer for bangla gec, and synthetically generated a 7.7m+ sentence corpus over 10 error categories. however, their error classification lacked key categories such as tense errors, gurucan . d . āl ī dōs . a, and semantic errors, which are significant as reported in error class sub-class example of wrong sentence (in violet) followed by correct sentence (in blue) spelling non-dictionary আিম কারখানায় কাড কির। (āmi kārakhānāyȧ kāḍa kari., @xmath0.) আিম কারখানায় কাজ কির। (āmi kārakhānāyȧ kāja kari., i work in factory.) dictionary আিম কাল বাির যাব। (āmi kāla bāri yāba., i will go water tomorrow.) আিম কাল বািড় যাব। (āmi kāla bāṛi yāba., i will go home tomorrow.) আিম কাল শািড় যাব। (āmi kāla śāṛi yāba., i will go saree tomorrow.) আিম কাল বািড় যাব। (āmi kāla bāṛi yāba., i will go home tomorrow.) word tense আিম গতকাল পড়ােশানা করব। (āmi gatakāla paṛāśōnā karaba., i will study yesterday.) আিম গতকাল পড়ােশানা কেরিছলাম। (āmi gatakāla paṛāśōnā karēchilāma., i studied yesterday.) যখন শীত আসেব তখন ফু ল ফু েটিছল। (yakhana śīta āsabē takhana phula phuṭēchila., when winter comes, flowers bloomed.) যখন শীত আসেব তখন ফু ল ফু টেব। (yakhana śīta āsabē takhana phula phuṭabē., when winter comes, flowers will bloom.) person আিম কারখানায় কাজ কের। (āmi kārakhānāyȧ kāja karē., i works in factory.) আিম কারখানায় কাজ কির। (āmi kārakhānāyȧ kāja kari., i work in factory.) number আিম এখােন চারজন থািক। (āmi ēkhānē cārajana thāki., i four stay here.) আমরা এখােন চারজন থািক। (āmarā ēkhānē cārajana thāki., we four stay here.) gender উত্তম একজন অসাধারণ অিভেনতৰ্ী। (uttama ēkajana asādhāraṇa abhinētrī., uttam is an outstanding actress.) উত্তম একজন অসাধারণ অিভেনতা। (uttama ēkajana asādhāraṇa abhinētā., uttam is an outstanding actor.) case আিম রান্নাঘরেক ভাত খাই। (āmi rānnāgharakē bhāta khāi., i eat rice to kitchen.) আিম রান্নাঘের ভাত খাই। (āmi rānnāgharē bhāta khāi., i eat rice in kitchen.) parts-of-speech িহমালেয়র সু ন্দর অিবস্মরণীয়। (himālayēra sundara abismaraṇīyȧ., the beautiful of himalaya is unforgettable.) িহমালেয়র েসৗন্দযর্ অিবস্মরণীয়। (himālayēra saundarya abismaraṇīyȧ., the beauty of himalaya is unforgettable.) missing আিম কাল বািড় •। (āmi kāla bāṛi •., i • home tomorrow.) আিম কাল বািড় যাব। (āmi kāla bāṛi yāba., i will go home tomorrow.) উত্তম একজন অসাধারণ •। (uttama ēkajana asādhāraṇa •., uttam is an outstanding •.) উত্তম একজন অসাধারণ অিভেনতা। (uttama ēkajana asādhāraṇa abhinētā., uttam is an outstanding actor.) gurucaṇḍālī dōṣa নন্দবাবু ইহা লক্ষয্ কেরেছন। (nandabābu ihā lakṣya karēchēna., nanda has noticed this.) নন্দবাবু ইহা লক্ষয্ কিরয়ােছন। (nandabābu ihā lakṣya kariyāchēna., nanda has noticed this.) নন্দবাবু ইহা লক্ষয্ কেরেছন। (nandabābu ihā lakṣya karēchēna., nanda has noticed this.) নন্দবাবু এটা লক্ষয্ কেরেছন। (nandabābu ēṭā lakṣya karēchēna., nanda has noticed this.) punctuation আিম গতকাল পড়ােশানা কেরিছলাম? (āmi gatakāla paṛāśōnā karēchilāma?, i studied yesterday?) আিম গতকাল পড়ােশানা কেরিছলাম। (āmi gatakāla paṛāśōnā karēchilāma., i studied yesterday.) semantic মানস আকাশ েখেত ভােলাবােস। (mānasa ākāśa khētē bhālōbāsē., manas loves to eat the sky.) মানস আকাশ েদখেত ভােলাবােস। (mānasa ākāśa dēkhatē bhālōbāsē., manas loves to see the sky.) মানস আকাশ েখেত ভােলাবােস। (mānasa ākāśa khētē bhālōbāsē., manas loves to eat the sky.) মানস মাছ েখেত ভােলাবােস। (mānasa mācha khētē bhālōbāsē., manas loves to eat fish.) back-translation has been used for data augmentation in gec @xcite @xcite . however, round-trip translation for bangla using english as a bridge fails to consistently produce grammatically incorrect sentences. moreover, errors generated via back-translation are challenging to categorize and localize, making them unsuitable for bangla gec (sec. 5). since bangla lacks large-scale gec datasets, we adopt a structured noise injection approach based on real-world error patterns, ensuring the controlled generation of incorrect sentences across specific error types. this approach improves sentence quality while maintaining category-wise error distributions observed in real time. unlike previous methods, our dataset explicitly covers common bangla errors and is adaptable to other indian languages (appx. c). ## grammar error categories standardization of error categories is necessary for gec to alleviate problems associated with low-resource languages @xcite . in this section, we categorize grammatical errors in bangla formally. we follow a standard bangla grammar book @xcite as reference for grammar error types. the book explicitly does not have any error categories; however, following the grammatical rules described in the book, we have formalised these error categories. to our humble understanding, these categories are exhaustive and cover all possible error types in bangla. they can be also used for other major indian languages (appx. c). first, we classified the grammatical errors in bangla into 5 broader categories. these broad categories are then further sub-divided into 12 finer distinctions. ## manual generation to understand the nature of real-life grammatical errors made by native speakers, we organized a survey in which participants were asked to write an essay on a specific topic within a specific time. in this way, we collected handwritten sentences from native bangla speakers and analyzed the occurrence of various grammatical errors in those sentences. each participant was allowed 30 minutes to write an essay comprising at least 15 sentences and 150 words, choosing from a set of topics provided. the survey was conducted in a proctored environment to simulate an exam-like situation, allowing us to collect real-time data (including errors) on bangla writing. we collected 123 essays, resulting in 2,576 sentences and 28,713 words, produced by 51 participants (30 participants wrote 2 essays each, while 21 participants wrote 3 essays). the longest sentence contained 69 words, while the shortest had just 1 word. detailed information about the topics and participants can be found in appx. e. a team of 3 bangla language experts then evaluated the written sentences and the errors were categorised based on majority voting. of the 2,576 sentences written, 1,045 (41%) were grammatically incorrect. 4 of the erroneous sentences, 804 (77%) contained single errors, while 241 (23%) had multiple errors. among the sentences with multiple errors, 185 contained 2 errors (18%), and 42 sentences contained 3 errors (4%). the remaining 9 sentences contained up to 6 errors. a total of 678 (4%) words were erroneous in these 1,045 sentences. spelling mistakes emerged as the most prevalent type of error, representing over 62% of occurrences. further analysis indicates that more than 45% of spelling errors stem from the confusion between the characters 'ন'(n)/'ণ'(ṇ); 'র'(r)/'ড়'(ṛ)/'ঢ়'(ṛh); and 'স'(s)/'শ'(ś)/'ষ'(ṣ). the most significant dictionary-based spelling error involves the mixing of "িক" (ki, whether) and "কী" @xcite . al- ## error injection methods in this section, we focus on generating synthetic data mimicking the real-world error distribution. @xcite proposed various synthetic data generation methods, including backtranslation @xcite , round-trip translation @xcite and error injection methods @xcite using grammatical methods. we next show the advantages and disadvantages of each type of methodology for synthetic data generation for bangla gec. ## instruction-tuning dataset: vaiyākaran . a to leverage llms for bangla gec, we curate a corpus of grammatically incorrect sentences by following the rule injection methodology as described in the previous section. we use the vācaspati @xcite corpus as the base set of grammatically correct sentences. we chose this corpus since it consists of only literature data and, hence, sentences sampled from this corpus are grammatically correct, as reported by the authors. further, the corpus captures stylistic, linguistic, spatial and temporal variations of bangla. the temporal diversity, in particular, is especially useful for gurucan . d . āl ī dōs . a errors since that particular writing style started becoming rare from 1960s. (newspapers, blogs, and social media data are not suitable for this.) we collected 5,67,172 sentences from vācaspati, which serve as our gold-standard sentences. additionally, we incorporated 250 sentences from a well-known grammar book @xcite to enhance the grammatical and linguistic richness of the dataset. these sentences are beneficial for generating errors related to number, gender, and semantics according to the rules specified in the book, which are less frequently found in literary data. consequently, we curated a dataset containing a total of 5,67,422 sentences. we followed the data cleaning and pre-processing steps outlined in appx. h to ensure that this dataset is suitable for error generation. we inject errors into randomly selected sentences following grammatical rules, with the type of error and the words to inject errors chosen randomly. we choose roughly 40% sentences for error injection. this process ensures a diverse range of errors. since most real-world erroneous sentences typically involve single-word mistakes, we generate sentences that reflect this pattern. to cover various error categories, we carefully monitor the number of erroneous words to and other similar sounding words, we place additional emphasis on sentences containing these characters or words when introducing spelling errors. for all other types of mistakes, errors are injected randomly with equal probabilities. the detailed steps taken to introduce noise for various types of errors are described in appx. g. sec. 4 showed that ∼77% of sentences contain single errors, while the remaining contain multiple errors. amongst them, the most common is multiple spelling errors. if there are multiple errors of the same category in a sentence, we have categorized it in that error category only, and not included them into the multiple error category. table a6 shows the distribution of sentences with multiple error types in vaiyākaraṇa. we have considered only those category of multiple errors that were present in the human survey in sec. 4, and not all possible combinations. although we focused on generating a gec dataset vaiyākaraṇa for bangla, the aforementioned procedure of injecting noise to generate grammatically wrong sentences can also be applied to other major indian languages with little or no modification (appx. c). we next show some anecdotal examples in additionally, we validated these sentences by a group of 12 individuals. each participant was provided with a unique set of 50 sentences and was asked to mark whether they believed the errors were naturally occurring, as well as the minimum education level required to detect the errors. of the 12 participants, 10 felt that the generated sentences were natural and that a minimum education level of 10th grade would be sufficient to identify the errors. the remaining 2 participants noted that the generated sentences sometimes seemed artificial. participants also rated the level of naturalness on a likert scale from 1 (least natural) to 5 (most natural), and the average score for the generated sentences was 3.62. ## evaluation in this section, we evaluate the effectiveness of our rule-based noise-injection method and the instruction-tuned dataset vaiyākaraṇa by evaluating large language models (llms) on grammatical error detection and correction, as well as on a separate task of paraphrasing. ## limitations curating a large quality benchmark for gec requires a good quality lemmatizer and pos tagger. bangla suffers from a lack of quality lemmatizers and pos taggers. hence, we had to manually add words from a grammar book @xcite . also, hand-written bangla data is not readily available. we conducted a survey for several weeks and could still collect only 2,576 handwritten sentences. finally, while the 12 human evaluators are all native speakers of bangla, evaluating against bangla grammarians could have given us more insights into the process. we are planning to do that in the future. additionally, we have evaluated only gpt-4o and not other commercially available models like claude and gemini due to resource constraints. in contrast, we have experimented with different architectures and loss functions to show the generalizability of the results. ## ethics statement the vaiyākaraṇa benchmark is curated by merging sentences from vacaspati corpus @xcite and @xcite . the authors of vacaspati provided us with the corpus, and @xcite is publicly available. hence, there is no copyright infringement in curating vaiyākaraṇa. we have made efforts to ensure that vaiyākaraṇa is also devoid of any objectionable statements. we have also conducted a manual essay writing survey for gathering real word errors. the participants have kindly allowed us to use their essays for research purpose.