P honology B ench: Evaluating Phonological Skills of Large Language Models (2024)

Abstract: Phonology, the study of speech’s structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.

## introduction large language models (llms) are a class of deep generative models @xcite that are trained on an internet-scale text corpora (e.g., two trillion tokens). to assess their usefulness in the real-world scenarios, prior work has focused on holistic evaluation of llms @xcite . for instance, they are evaluated on tasks that require syntactic and semantic understanding of the language such as summarization, rationale generation, story generation and question answering @xcite @xcite @xcite @xcite . however, there are various text-based tasks of practical importance that require joint understanding of the written and spoken language such as poetry generation @xcite @xcite . in particular, these tasks require the model to have phonological skills i.e., understanding the patterns of speech units and rules governing pronunciation in language. while llms have been adopted to perform text-based tasks that require phonological skills @xcite , it is unclear to what extent they acquire phonological skills by training on large-scale text data, without access to speech data. despite being trained solely on textual data, llms have been applied to many tasks that benefit from a deep understanding of phonology. these applications include poetry generation, song writing, machine translation, and language learning @xcite @xcite . for example, the generation of poetry and lyrics leverages the models' knowledge of rhyme and meter to produce rhythmically engaging content. similarly, phonetic transcriptions from related dravidian languages can enhance the accuracy of multilingual machine translations @xcite . we argue that due to their extensive training and further alignment, llms may leverage the learnt associations between written and spoken forms to accomplish these tasks. however, due to the lack of deep phonological understanding, they fail to perform phonological reasoning tasks in open-world scenarios @xcite . for example, although llms are popular for composing poetry, machine-generated poetry lacks diverse rhyming schemes and is highly criticised by experts @xcite . while earlier studies @xcite @xcite , have evaluated llm performance on poetry generation, machine translation and language learning. these studies were designed to analyse llm performance on the downstream task rather than probe the model's inherent understanding of phonology. to this end, we devise three diagnostic tasks that benchmark various llms on fine-grained tasks that explicitly apply phonological concepts in english. our benchmark, phonologybench, comprises of three tasks -grapheme-to-phoneme (g2p) conversion, syllable counting and rhyme word generation, is shown in figure 1 . g2p is a widely adopted task for speech recognition applications that leverages pronunciation information to produce phonemes. additionally, syllable counting and rhyme word generation are educational exercises used to enhance early language skills in children @xcite . thus, phonologybench provides a systematic framework to analyse the phonological skills of llms that inadvertently affects their performance in real-world applications (writing assistants, translators) figure 1 . we conduct a comprehensive study across 6 widely used llms, including three closed-source models -gpt-4 (openai, 2023), claude-3-sonnet @xcite , gpt-3.5-turbo (ope-nai, 2022), and three open-source models -, mistral-7b @xcite and mixtral-8x7b @xcite . furthermore, we study the impact of word frequency, tokenization and sentence complexity across all three tasks. we find that overall llms exhibit surprisingly good performance on the three evaluated tasks while being trained solely on textual data. however, we observe a gap of 45% between human performance and llm per-formance on syllable counting whereas the gap is nearly 17% for rhyme word generation. we also note that, based on our evaluation, no single model consistently outperforms the others across all evaluated tasks. for example, gpt-4 is the best model in rhyme generation while claude-3-sonnet outperforms gpt-4 by a large margin in syllable counting. this highlights that models are good at different phonological tasks and researchers should choose llms based on their performance on the core task related to the downstream applications. our main contributions are: grapheme-to-phoneme. many studies have been conducted on g2p conversion. early works have proposed joint n-gram models @xcite and joint-sequence models @xcite for g2p conversion. recent developments in g2p studies have shown lstm @xcite and transformer @xcite to be powerful g2p models. these models have access to various pronunciation dictionaries and are trained to explicitly accomplish the grapheme-phoneme conversion for speech applications like automatic speech recognition(asr) and text-to-speech (tts) @xcite . finally, @xcite combines the cmudict 2 corpus with a neural-network model to convert graphemes to phonemes. however, previous works have not analysed the ability of llms as g2p models. syllable counting. syllable counting is used for testing the phonological awareness of children at an early age @xcite . it is also useful in second language acquisition and commonly used as an educational tool. additionally, counting of syllables is crucial for composing songs, poems and haikus @xcite @xcite . several works have studied the syllable structure of haikus and poetry in english, haikus are particularly popular for they 5-7-5 syllable structure @xcite . recently, @xcite , @xcite and @xcite have utilized syllable counts as decoding contraints or metre descriptors to generate formatted sonnets, lyrics and poems. @xcite evaluates the ability of llms to generate syllable-controlled texts (e.g. -complete a sentence in 5 syllables). in this work, we focus 2 http://www.speech.cs.cmu.edu/cgi-bin/cmudict on evaluating the ability of llms to recognise and count the number of syllables in a sentence. ## phonologybench here, we present our benchmark, phonologybench, that evaluates the english phonological skills of llms across three tasks: grapheme-to-phoneme conversion ( §3.1), syllable counting ( §3.2), and rhyme word generation ( §3.3). an overview of these tasks, along with their significance in a variety of downstream applications, is provided in figure 1. starting with grapheme-to-phoneme conversion, this task is pivotal for enhancing speechrelated applications, including automatic speech recognition and text-to-speech systems, by ensuring accurate phonetic interpretations of text. furthermore, the incorporation of phonemic and phonetic transcriptions into machine translation models introduces additional layers of understanding through phonetic embeddings, thereby improving translations @xcite . in addition, syllable counting and rhyme word generation are traditionally popular as educational tools. beyond facilitating language learning, they are essential in the creative processes involved in poetry and song composition, illustrating their versatility. overall, we find that the tasks curated in phonologybench shed light on a wide range of real-world applications for llms, and benchmarking llm performance on these tasks can advace research in llm development for improved linguistic and creative generations. we provide the dataset statistics in ## experimental results here, we aim to benchmark the phonological skills of llms across the three tasks -g2p ( § 4.1), syllable counting ( § 4.2) and rhyme word generation ( § 4.3). we further study the impact of word frequency in the pretraining dataset, tokenization strategy, and sentence complexity on llm performance across all the tasks. figure 7 shows human as well as model responses on the three tasks along with the gold references. we prompt all models in a zero-shot setting. ## conclusion in this work, we present, phonologybench that consists of 4k data samples to evaluate the phonological skills of llms across g2p conversion, syllable counting and rhyme word generation in english. we show that llms fail to outperform human performance in syllable counting and rhyme word generation. our findings thus highlight that there is scope of improving llm performance on these core phonological task and encourages future research in training phonologically aware models. a straightforward approach to improving model performance across these phonological tasks is to add more phonological data during pre-training. prior work by @xcite have shown the efficacy of joint textual and phonetic embedding in neural machine translation, thus, future work can focus on augmenting llms with phonetic representations. overall, we show that no single model consistently performs well on all the tasks. thus, we encourage researchers to develop downstream applications that rely on these phonological tasks by carefully selecting llms that perform well on the particular task.