Reasoning or Memorization? Investigating LLM s’ Capability in Restoring C hinese I nternet Homophones (2025)

Abstract: Chinese homophones, prevalent in Internet culture, bring rich linguistic twists that are challenging for language models. While native speakers disambiguate them through phonological reasoning and contextual understanding, it remains untested how well LLMs perform on this task and whether LLMs also achieve this via similar reasoning processes or merely through memorization of homophone-original word pairs during training.In this paper, we present HomoP-CN, the first Chinese Internet homophones dataset with systematic perturbations for evaluating LLMs’ homophone restoration capabilities. Using this benchmark, we investigated the influence of semantic, phonological, and graphemic features on LLMs’ restoration accuracy, measured the reliance levels of each model on memorization during restoration through consistency ratios under controlled perturbations, and assessed the effectiveness of various prompting strategies, including contextual cues, pinyin augmentation, few-shot learning, and thought-chain approaches.

## introduction homophonic wordplay in chinese internet culture creatively utilizes phonological similarities between characters to construct new words and layered semantic meanings @xcite . for example, the homophone "蕉绿" (jiao1 lü4, "banana-green") replaces the original word "焦 虑" (jiao1 lü4, "anxiety"), reconfiguring a negative emotion into a playful and lighthearted expression. unlike english puns, which rely on intralingual homophony (e.g., "a good pun is its own reword/reward") @xcite , chinese homophonic wordplay creatively substitutes characters with similar pronunciations within the logographic writing system. recent advances in natural language processing (nlp), particularly through large language models (llms), have demonstrated substantial progress in disambiguating english homophones @xcite @xcite . however, due to the high homophone density in chinese pinyin(e.g., shi4 mapping to dozens of characters such as 是/事/市) and tonal complexity (identical syllables with different tones convey distinct meanings, e.g., ma1妈/ma2马/ma3码/ma4骂), llms encounter greater challenges in comprehending chinese homophones than english. previous research has explored chinese homophones in nlp tasks such as spelling correction @xcite @xcite , offensive language detection @xcite , and humor generation @xcite . nev-ertheless, there is no systematic study on llms' ability to understand and restore chinese homophones, which is crucial for practical applications such as improving llms' ability to understand social media text and identifying offensive content. for instance, chinese netizens may replace "太 贱 (too mean)" with the same pronunciation "肽 键 (peptide bond)" to use a non-offensive biological term conveying discriminatory and offensive content @xcite . it has been suggested that native chinese speakers leverage their perceptual systems to retrieve original words from homophonic variants through phonological similarity-based reasoning and contextual information understanding @xcite @xcite . building upon this human cognitive paradigm, we propose the following research question: how do llms perform in homophone restoration? is this capability of llms driven by human-like reasoning through phonological similarity, or simply stem from memorization of homophone-original word pairs in pretraining? additionally, can strategies like contextual information or providing pinyin foot_0 to enrich prompts enhance llm performance in restoration? in this work, we comprehensively explored llms' effectiveness and enhancement in chinese internet homophone restoration by utilizing our homop-cn dataset. first, we analyzed the restoration capacity of llms by considering the differences between the original words and the homophones from semantic, phonological, and graphemic perspectives. second, drawing inspiration from @xcite , we designed a set of adversarial variations as perturbations to quantify the extent of memorization, as shown in figure 1 . finally, we delved into the role of different prompting strategies, including context cues, pinyin-augmentation, few-shot, chain-of-thought (cot) @xcite , memory-of-thought (mot) @xcite in this task. our results demonstrate that llms exhibit substantial variation in restoring chinese internet homophones, with model scale emerging as a critical factor: larger models achieve reasoning-based restoration while smaller ones depend predominantly on memorization. this performance gap is further modulated by semantic, phonological, and graphemic disparities between original words and their homophone counterparts, which systematically affect both restoration accuracy and memorization dependence. while contextual cues, few-shot learning, and thought-chain strategies (cot/mot) prove effective for performance enhancement, pinyin augmentation shows limited utility. these findings provide valuable insights into llms' robustness in handling intralingual and usergenerated content in internet contexts. ## conclusion in this study, we present the first chinese internet homophones dataset with language perturbations to evaluate llms' restoration capabilities and their reliance on memorization. our results show that llms exhibit significant differences in restoring homophones: larger models rely more on reasoning, while smaller ones depend on memorization. performance variations are further influenced by semantic, phonological, and graphemic differences between original words and homophones, systematically affecting accuracy and memorization dependence. although strategies like contextual cues, few-shot learning, and mot improve performance, pinyin-based augmentation unexpect-edly failed to enhance restoration. these findings shed light on llms' robustness with intralingual and user-generated online content.