Encode Errors: Representational Retrieval of In-Context Demonstrations for Multilingual Grammatical Error Correction (2025)

Abstract: Grammatical Error Correction (GEC) involves detecting and correcting the wrong usage of grammar. While large language models (LLMs) with in-context learning (ICL) capabilities have shown significant progress on various natural language processing (NLP) tasks, their few-shot performance on GEC remains suboptimal. This is mainly due to the challenge of retrieving suitable in-context demonstrations that capture error patterns instead of semantic similarity. In this paper, we demonstrate that LLMs can inherently capture information related to grammatical errors through their internal states. From these states, we extract the Grammatical Error Representation (GER), an informative and semantically neutral encoding of grammatical errors. Our novel GER-based retrieval method significantly boosts performance in ICL settings on multilingual GEC datasets, improving the precision of correction. For high-resource languages, our results on 8B-sized open-source models match those of closed-source models such as Deepseek2.5 and GPT-4o-mini. For low-resource languages, our F_{0.5} scores surpass the baseline by up to a factor of 1.2. This method provides a more precise and resource-efficient solution for multilingual GEC, offering a promising direction for interpretable GEC research.

## introduction grammatical error correction (gec) is an important research field in natural language processing (nlp), as it requires language models to understand the syntax, semantics, and pragmatics underlying the subtle structures of natural sentences @xcite . initially considered a specific case of machine translation @xcite , gec has input: she has swim for 14 hours. prediction: she has swum for fourteen hours. ## methods in this section, we describe a novel method for extracting vectors that characterize grammatical error information and using them to create semantically neutral grammatical error representations (ger). ger from the training dataset is stored in a database, where each error is associated with its original and corrected texts. during inference, the model retrieves similar correction examples based on ger to guide corrections, with the flexibility to dynamically adjust the number of examples depending on the complexity of the input sentence. the final gec prediction is generated by combining the retrieved examples with a correction template. ## experiments 4.1 datasets, models, and metrics we evaluate the proposed method on five gec datasets across four languages to testify to ger's ability to encode and retrieve errors. following the multilingual setup in @xcite , we process the training dataset and use llamaindex @xcite to construct the database and retriever. for high-resource english (en), we use the w&i+locness @xcite as the training dataset, and the conll-14 @xcite and bea-19 @xcite datasets for testing. for medium-resource german (de), we use the falko-merlin @xcite dataset for both training and testing. to showcase the generalizability of our method, we also include low-resource romanian (ro) and estonian (et). for romanian, we choose the ronacc @xcite training and test datasets; for estonian, we use the tartu l2 learner corpus @xcite as the database and the l1 (tartu-l1) as the test data. 5 since ger requires the model's internal states, all experiments are conducted using recent opensource multilingual llms, including meta's llama3.1-8b-instruct @xcite and qwen2.5-7b-instruct @xcite by tongyi. adhering to the dataset-specific evaluation pipeline for each language, we use the errant toolkit @xcite to align edits between initial and final predictions. for evaluation, we apply m2scorer @xcite for conll-14, falko-merlin, and tartu-l1, while errant for bea-19 and ronacc. our method is compared with the following baselines: all experiments are conducted in an 8-shot setting. for all baseline methods, we retrieve 4 erroneous and 4 correct examples, following @xcite . since our method dynamically determines the number of examples needed for each sentence, we retrieve 4 examples for each error and ensure that the average demonstration number is 8. ## conclusion in this paper, we delve into the internals of llms and develop a novel method for extracting precise and interpretable grammatical error representations (ger) with less semantic noise. the effectiveness of ger in encoding fine-grained error patterns enables the retrieval of high-quality error demonstrations, improving the few-shot performance of llms on gec across diverse language settings. our preliminary exploration and successful utilization of llms' internal states highlight the potential of utilizing the model's inherent knowledge to strengthen gec performance, alignment, and interpretability, all without the need for additional components or training resources.