Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correction (2024)

Abstract: In the era of large language models (LLMs), in-context learning (ICL) stands out as an effective prompting strategy that explores LLMs’ potency across various tasks. However, applying LLMs to grammatical error correction (GEC) is still a challenging task. In this paper, we propose a novel ungrammatical-syntax-based in-context example selection strategy for GEC. Specifically, we measure similarity of sentences based on their syntactic structures with diverse algorithms, and identify optimal ICL examples sharing the most similar ill-formed syntax to the test input. Additionally, we carry out a two-stage process to further improve the quality of selection results. On benchmark English GEC datasets, empirical results show that our proposed ungrammatical-syntax-based strategies outperform commonly-used word-matching or semantics-based methods with multiple LLMs. This indicates that for a syntax-oriented task like GEC, paying more attention to syntactic information can effectively boost LLMs’ performance. Our code is available at https://github.com/JamyDon/SynICL4GEC.

## introduction recently, large language models (llms) have shown awesome power in many areas and ended the contest on many tasks @xcite @xcite . unfortunately for llms, grammatical error correction (gec), which aims at automatically correcting grammatical errors in erroneous text @xcite , is still a challenging task where they cannot beat conventional models yet. @xcite and @xcite explore the performance of llms on gec, demonstrating mainstream llms lag over 10 points behind the state-of-the-art result. therefore, it is significant to explore new strategies to further improve the power of llms on gec. as a helpful prompting strategy, in-context learning (icl) has achieved impressive results on many tasks @xcite . in icl, several in-context examples are presented to llms as demonstrations before the input test sample in order to make llms aware of the requirement and output format of the specific task, thereby enhancing llms' performance during the subsequent generation process. since the quality of incontext examples plays a crucial role under the few-shot setting, some strategies of example selection have been proposed @xcite @xcite . to the best of our knowledge, no existing work on icl example selection has taken syntactic information into consideration. however, gec aims to correct grammatical errors and is a typical syntaxoriented task. in gec, common errors can be classified into four types: misuse, missing, redundancy and word order @xcite , and the last three of which are closely related to syntactic structure. that is, the missing, redundancy or disorder of text constituents may lead to ill-formed syntax @xcite , suggesting the important role syntax plays in gec. hence, selecting in-context examples based on syntactic structure is likely to benefit llms more than conventional word-matching-based or semanticsbased approaches. comparing with semantic similarity, syntactic similarity of text is less-studied. previous works have leveraged the similarity of dependency trees to help multi-document summarization @xcite and semantic textual similarity @xcite . to compute syntactic similarity, several effective algorithms computing similarity between syntactic trees have been proposed. tree kernel is a typical one, which counts the shared sub-structures of two trees to measure their similarity @xcite @xcite . polynomial distance is another handy one, source (erroneous sentence) target (corrected sentence) ## conclusion in this work, we make use of two conventional treebased syntactic similarity algorithms and the selectthen-rank two-stage framework to select in-context examples for the gec task. empirical results show that our syntax-based in-context example selection method is effective for gec. we call on the nlp community to pay more attention to the help of syntactic information for many other syntax-related tasks besides gec. exist a better size than the values we use in our experiments. fifth, except for the stanford parser (which splits sentences itself), our experiments do not split instances with multiple sentences into single-sentence instances. some instances in gec datasets contains more than one sentence. directly parsing these instances without splitting them into single sentences may hurt the parsing performance and lead to unreliable results. last, we do not treat in-context examples as a whole, which might lead to a lower level of diversity of examples and sub-optimal performance, as addressed in @xcite and @xcite .