Search if you don’t know! Knowledge-Augmented K orean Grammatical Error Correction with Large Language Models (2024)

Abstract: Grammatical error correction (GEC) system is a practical task used in the real world, showing high achievements alongside the development of large language models (LLMs). However, these achievements have been primarily obtained in English, and there is a relative lack of performance for non-English data, such as Korean. We hypothesize that this insufficiency occurs because relying solely on the parametric knowledge of LLMs makes it difficult to thoroughly understand the given context in the Korean GEC. Therefore, we propose a Knowledge-Augmented GEC (KAGEC) framework that incorporates evidential information from external sources into the prompt for the GEC task. KAGEC first extracts salient phrases from the given source and retrieves non-parametric knowledge based on these phrases, aiming to enhance the context-aware generation capabilities of LLMs. Furthermore, we conduct validations for fine-grained error types to identify those requiring a retrieval-augmented manner when LLMs perform Korean GEC. According to experimental results, most LLMs, including ChatGPT, demonstrate significant performance improvements when applying KAGEC.

## introduction grammatical error correction (gec) task aims to detect and correct textual errors in a given source sentence. in the real world, gec is a practical and essential task used in applications such as writing assistance and language teaching @xcite @xcite . the development of large language models (llms) has led to the integration of various tasks with llms and has shown remarkable performance @xcite @xcite . the recent gec studies have also achieved decent outcomes by focusing on leveraging llms' inherent knowledge through prompting methods @xcite . however, these achievements have been mainly in english, which is heavily used in llms' training and have not shown the same level of superiority for non-english data including as korean, compared to pre-trained language models (plms) @xcite . figure 1 (a) shows the actual generation results of the conventional gec approach with llm 1 . the source sentence "실수 하지 않는 사과는 아무 일도 못 해요. (an apple that doesn't make mistakes can't do anything.)" exhibits an error where the subject is described as performing an action it practically cannot do. despite the mistake of 'people' being replaced by 'apple' in the source sentence, the llm fails to detect this error and outputs it as is in the target sentence, leading to an under-correction problem. we hypothesize that relying solely on the parametric knowledge of llms makes it challenging to understand the given context in the korean gec task thoroughly. we find the clue to mitigate this issue by injecting linguistic information from external sources to enhance understanding of the source. therefore, we propose a knowledge-augmented gec (kagec) that refines information helpful for performing the gec task from external sources and provides it in the prompt. we consider two aspects to provide relevant external knowledge for a given input: i) salient information @xcite and ii) relevant knowledge @xcite . by identifying key keywords in the input and retrieving information on these from external sources to provide alongside llms, we can enhance llms performance. figure 1 (b) represents the correction process of kagec. it involves a process of extracting salient key phrases from the source sentence, through which the errored word '사과는 (apple)' is identified. afterward, knowledge corresponding to the extracted keywords is retrieved, and an augmented input is built by including them. through this method, llms enhance their linguistic understanding of the source sentence, recognizing the incorrect substitution of 'people' with 'apple' and precisely correcting it to "사람은 아무 일도 못 해요. (people who don't make mistakes can't do anything.)." this indicates that kagec effectively supports llms' inherent capabilities by augmenting retrieval from non-parametric knowledge. furthermore, we conduct a detailed validation for diverse error types in korean gec with llms to identify those specifically requiring knowledge augmentation. we provide additional experiments and extensive analysis, including investigation of qualitative results related to informativeness. our contributions are threefold: i) we categorize and validate fine-grained error types for the korean gec task, ii) we propose a knowledge-augmented gec (kagec) framework that enhances the understanding of context (source) by furnishing non-parametric information as knowledge, and iii) we demonstrate the effectiveness of kagec through extensive additional experiments, including in-depth analysis and diversification of external knowledge types. ## preliminary: validation of error types in this section, we assess the gec capabilities of each model across different error types and categorize these types based on specific criteria. to conduct this validation, we employ chatgpt, known for its exceptional performance across a range of downstream tasks. ## knowledge-augmented grammatical error correction (kagec) in this section, we propose an approach that enhances the effectiveness of the gec task through a knowledge-augmented generation method (kagec) by adopting the retrieval of non-parametric knowledge. in many knowledge-intensive tasks, such as open-ended question answering, building a valid pool of non-parametric knowledge and effective retrieval plays a crucial role @xcite . due to the challenges of capturing relevance in a zero-shot setting, recent studies have demonstrated the efficiency and effectiveness of involving llm in the search process for relevance modeling @xcite . inspired by this, we first construct a set of knowledge, k, that can finally enhance the generation ability of llms. from this constructed knowledge set, we retrieve a subset of knowledge, k s , that is relevant to the given source sentence, s, and reconstruct the input with the retrieved knowledge. afterward, the input enriched with knowledge, s ′ , supports the generative capabilities of llms, resulting in the generation of the target sentence, t. ## experiments detailed setups are described in appendix c. ## analysis 6.1 effectiveness of retrieved knowledge we conduct additional experiments to ascertain whether the provision of deliberately retrieved knowledge actually guarantees effectiveness in the gec task. kagec enables llms to extract key phrases from a source sentence and retrieve corresponding non-parametric knowledge. source sentence 이번 여름에는 시원하게 보내고 싶어서 옷을 새로 싶어. (i want to stay cool this summer, so i want to new clothes.) target sentence 이번 여름에는 시원하게 보내고 싶어서 옷을 새로 사고 싶어. (i want to stay cool this summer, so i want to buy new clothes. (a) predicted results according to prompting method baseline 이번 여름에는 시원하게 보내고 싶어서 옷을 새로 사고 싶어요. (i want to stay cool this summer, so i would like to buy new clothes.) cot 이번 여름에는 시원하게 보내고 싶어서 옷을 새로 살고 싶어요. -"옷을 새로 싶 어" → "옷을 새로 살고 싶어요" : "싶어" 는 동사 "새로 살다"와 함께 사용되어야 합니다. 왜 옷을 새로 살고 싶을까요? -시원한 옷을 입어서 더 쾌적하게 여름을 보내고 싶어서예요. (i want to stay cool this summer, so i want to live anew clothes. -"want new clothes" → "want to live anew clothes.": "want" should be used in conjunction with the verb "to live anew". why want to live anew clothes?: it is because i want to spend the summer more comfortably by wearing cool clothes.) decomposed -step 1 새로 싶어 → 새로 샀어 (want to @xmath0. (i want to stay cool this summer, so i bought new clothes. kagec -top-n: ['이번', '여름', '시원하게', '보내다', '옷'] (['this', 'summer', 'cool', 'stay', 'clothes.']) {{retrieved definitions}} 이번 여름에는 시원하게 보내고 싶 어서 옷을 새로 사고 싶어. (i want to stay cool this summer, so i want to buy new clothes.) (b) predicted results by external information types random -top-n: ['원함수', '바래움', '복화합물', '베어링', '명감'] (['this', 'summer', 'cool', 'stay', 'clothes.']) {{random definitions}} 이번 여름에는 시원하게 보내고 싶 어서 옷을 새로 살고 싶어. (i want to stay cool this summer, so i want to live anew clothes.) type desc. 이번 여름에는 시원하게 보내고 싶어서 옷을 새로 사고 싶어요. (i want to stay cool this summer, so i would like to buy new clothes.) since it is unknown which error type the given source sentence falls into during inference, all detailed error types are provided. for example, for the auxiliary predicate error type, a description like "using auxiliary verbs that do not conform to grammar. this occurs..." is contained. providing random knowledge results in performance degradation for all models for the auxiliary predicate type and shows only slight performance increases or decreases for the other error types across models. this indicates the high informativeness of providing knowledge that is deliberately relevant to the given source sentence. moreover, providing descriptions of error type classifications instead of knowledge about dictionary definitions generally results in poor performance across most models and error types. for instance, the gemini model exhibits a performance decline of over 20%p for the 'element' type compared to the baseline method, which can be interpreted as the provided descriptions acting as noise. furthermore, figure 2 demonstrates the results for additional knowledge provision methods in chatgpt. 'keyword only' refers to cases where only key phrases are provided in the input prompt as additional clues, without corresponding knowledge in the dictionary. in the 'pos tag keyword' method, keywords are extracted through the pos tagger (park and cho, 2014) instead of llms. in comparison with various knowledge provision methods, kagec continues to exhibit the most superior performance. therefore, the effectiveness of kagec in enhancing understanding of the context through the provision of relevant semantic information can be observed. ## conclusion llms have demonstrated exceptional generative capabilities in gec systems. however, their performance in non-english languages, such as korean, remains relatively lacking. we hypothesize that relying solely on the parametric knowledge of llms makes it challenging to understand the given context in the korean gec task thoroughly. to address this, we propose the knowledge-augmented gec (kagec) method, which consists as follows: it extracts the top-n key phrases, retrieves definitions for these terms, and reconstructs the input for generation. afterward, this augmented input is fed into llms to enrich the generated answer (target sentence). moreover, through detailed val-idation of llms, we investigate the error types requiring knowledge augmentation for the korean gec task. experimental results indicate that the proposed kagec method consistently leads to performance improvements across various llms and the target error types. through kagec, we can effectively support the inherent capabilities of llms by augmenting non-parametric knowledge.