P - C o T : A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLM s (2025)

Abstract: This study explores the potential of phonological reasoning within text-based large language models (LLMs). Utilizing the PhonologyBench benchmark, we assess tasks like rhyme word generation, g2p conversion, and syllable counting. Our evaluations across 12 LLMs reveal that while few-shot learning offers inconsistent gains, the introduction of a novel Pedagogically-motivated Participatory Chain-of-Thought (P-CoT) prompt, which is anchored in educational theories like scaffolding and discovery learning, consistently enhances performance. This method leverages structured guidance to activate latent phonological abilities, achieving up to 52% improvement and even surpassing human baselines in certain tasks. Future work could aim to optimize P-CoT prompts for specific models or explore their application across different linguistic domains.

## introduction the performance of transformer decoder-based large language models (llms), such as chatgpt @xcite , has advanced rapidly in recent years @xcite @xcite @xcite @xcite . these models excel at a wide range of tasks through in-context learning @xcite and have been extensively evaluated for their capabilities in syntax, semantics, morphology, and discourse @xcite @xcite . in contrast, phonology has received comparatively little attention in this area, primarily because it has traditionally been studied using multi-modal @xcite or speech-based models @xcite @xcite . this is despite its significant role in generating natural prosody @xcite and accurately modeling dialectal variations @xcite . recent work such as @xcite shows that while text-based llms encode latent phonological patterns from language data, they often struggle with explicit tasks, resulting in a notable gap compared to human performance. therefore, enhancing phonological reasoning in text-based llms emerges as a timely and important research direction. our study investigates the phonological competence of modern text-based language models using phonologybench foot_0 @xcite , a benchmark that evaluates three key tasks: rhyme word generation, grapheme-to-phoneme (g2p) conversion, and syllable counting. we first evaluated 12 state-of-the-art models and examined the impact of few-shot learning @xcite on their performance. our findings reveal that while fewshot prompting sometimes results in improvements over baseline, these enhancements are inconsistent and model-dependent, with performance occasionally falling below baseline levels. this underscores the need for more robust techniques to enhance phonological reasoning in text-based llms. to address these challenges, we propose a novel approach that integrates role-playing pedagogy within a chain-of-thought (cot) prompting framework @xcite . we hypothesize that although llms inherently encode meta-knowledge in phonology, they lack effective strategies to leverage this information. by integrating two distinct pedagogical techniques-scaffolding @xcite and discovery learning @xcite )-and building on prior studies that have effectively combined these methods @xcite @xcite , our pedagogically-motivated participatory chain-of-thought (p-cot) prompt consistently improves performance across all 12 models on the phonological tasks. these results persist even though we use ## assessment of phonological understanding in llms in this section, we aim to explore the extent to which llms inherently possess phonological knowledge by evaluating their performance in a zero-shot context. additionally, we plan to examine the effectiveness of few-shot learning in further enhancing the models' phonological reasoning. in our exploration of the phonological capabilities of text-based llms, we focus on evaluating three key feature count g2p conversion (high) 1042 g2p conversion (low) 2084 rhyme word generation (common) 199 rhyme word generation (rare) 110 syllable counting 993 ## conclusion our study provides a foundational analysis of phonological reasoning within text-based llms, a domain that has been largely overlooked in prior research. contrary to earlier assumptions, our findings reveal a substantial potential for phonological competence within these models. by introducing the pedagogically-motivated participatory chain of thought (p-cot) prompting method, grounded in guided discovery learning and scaffolding, we demonstrate significant performance enhancements across rhyme word generation, g2p conversion, and syllable counting tasks, highly surpassing traditional few-shot methods. our work paves the way for future investigations to refine llm capabilities in phonology, and encourages the exploration of similar pedagogical approaches in other linguistic domains. our framework offers a persuasive model for enhancing llm performance, demonstrating the ability to tap into hidden capabilities using wellstructured, education-inspired prompts.