Exploring Layer-wise Representations of E nglish and C hinese Homonymy in Pre-trained Language Models (2025)

Abstract: Homonymy can easily raise lexical ambiguity due to the misunderstanding of its multiple senses. Correct recognition of homonym sense greatly relies on its surrounding context. This ambiguous nature makes homonyms an appropriate testbed for examining the contextualization capability of pre-trained (PLM) and large language models (LLMs). Considering the impact of part of speech (POS) on homonym disambiguation and the prevalence of English-focused studies in word embedding research, this study extends to Chinese and provides a comprehensive layer-wise analysis of homonym representations in both languages, spanning same and different POS categories, across four families of PLMs/LLMs (BERT, GPT-2, Llama 3, Qwen 2.5). Through the creation of a synthetic dataset and computation of disambiguation score (D-Score), we found that: (1) no universal layer depth excels in differentiating homonym representations; (2) bidirectional models produce better contextualized homonym representations compared to much larger autoregressive models; (3) most importantly, POS affects homonym representations in models in ways that differ from human research findings. The individual differences between LLMs uncovered in our study challenge the simplistic understanding of their inner workings. This reveals a compelling research frontier: conducting controlled experiments with purposefully manipulated inputs to enhance the interpretability of LLMs. We have made our dataset and codes available publicly at https://github.com/neurothew/exploring-homonym-rep-in-llm.

## introduction the efficient and economic use of lexical inventory results in multiple word senses converging into a single lexical item, leading to lexical ambiguity @xcite . among these * these authors contributed equally. lexical items, homonyms represent a common type. they denote two (or more) semantically and etymologically unrelated meanings. for instance, "bank" can refer to a financial institution or the side of a river. resolution of lexical ambiguity, while rarely conscious in everyday language use, can pose specific challenges to human readers or listeners. psycholinguistic and neuroimaging research suggest that homonyms, with their unrelated meanings, often make comprehension more difficult as indicated by enhanced neural activation and longer response time @xcite @xcite . lexical ambiguity also poses challenges to distributional semantic models, though the nature of these difficulties differs from those encountered in human (lake and murphy, 2023). early static word representation models such as lsa @xcite and word2vec @xcite cannot be modulated by surrounding contexts after the training process. different senses of a word must share the same representation, hindering its ability to differentiate word senses. contextualized word representations and language models such as elmo @xcite and @xcite were proposed to address the problem. followed by this development, modern plms and llms are all contextualized language models. contextualized word representations from these models are influenced by its surrounding contexts, allowing the representations to vary instead of remain static. the context can be preceding tokens in autoregressive models like gpt @xcite , or both preceding and following tokens in bidirectional, autoencoding models like bert @xcite . homonyms, which greatly rely on their surrounding context to be accurately understood, serve as appropriate candidates to experiment on the contexutalization capability of plms and llms. besides, the difference between homonyms whose senses belong to the same or different syntactic categories or parts of speech (pos) also warrants attention @xcite . for example, 'fly' can serve as both a noun (a small insect) and a verb (to move through air). electrophysiological responses and blood-oxygen-level-dependent (bold) signals, as observed during lexical ambiguity processing, are significantly influenced by pos variability, according to previous research @xcite . homonyms with different pos senses elicit greater neural activation compared to those with the same pos meanings, suggesting a greater cognitive resource demand for syntactic category processing @xcite . given these findings in human language processing, it becomes interesting to explore whether these two types of homonyms would be represented differently in language models, as they are in the human brain. our main contributions are: ## related works our present work is largely related to previous studies on the investigation of the contextualization capabilities of plms. contextualized embeddings can be easily translated into their corresponding definitions in an english dictionary, capturing sense-specific information @xcite . they can also be used to predict human behaviours and explain variances of human judgements on meanings @xcite . a study that focuses on polysemes and homonyms revealed that while word embeddings can differentiate ambiguous words in terms of cosine similarity, the distinctive power was much less compared to human ratings @xcite . it reveals the limitations of word embeddings. the distance between ambiguous word embeddings was found to show different trajectories across model layers, which in turn depended on the architectural factors, including but not limited to model size @xcite . @xcite analyzed the contextual word representations in elmo, bert and gpt-2. it was found that the representations were more contextspecific in the higher layers, consistent with previous studies @xcite @xcite . additionally, @xcite observed that the word embeddings in language models suffered from anisotropy, referring to the non-uniform distribution of the embeddings in the embedding space. they proposed subtracting the cosine similarity from the baseline computed from the text materials to create an adjusted measure that enhances interpretability. while there is existing research on contextuality and lexical ambiguity, our work makes significant contributions in several ways. for instance, while @xcite examined the contextualization of words along the functionality continuum ranging from homonyms, modals, to articles, their study was not positioned to explore the nuances, such as the contrast between contextualized homonym representations computed from homonyms of the same or different meanings. in contrast, we specifically targeted homonyms with two different meanings by developing our own dataset. we compared the performance of 21 models across four model families, varying in size (from 110m to 8b) and architecture, which is more extensive than the few models used in @xcite and @xcite . we proposed an angle-based disambiguation score to account for the nonlinearity of cosine similarity (section 3.2.1). furthermore, we investigated the impact of pos on contextualized embeddings by controlling for the selection of homonyms. while pos tagging has attracted substantial research attention in nlp @xcite , the inner workings of pos processing in llms remain unclear. this issue is particularly intriguing, as pos is known to affect human comprehension of homonyms @xcite . last but not least, our work extends the analysis of contextualized embeddings to chinese, in contrast to previous studies that have predominantly focused on english @xcite and other western languages such as spanish @xcite . due to its logographic writing system, a single sinogram in chinese can often represent multiple meanings depending on context @xcite @xcite . in many cases, the meaning of a bisyllabic compound word, which constitutes a major portion of the vocabulary, can be inferred from its monosyllabic sinogram components. this ambiguity is especially pronounced in mandarin, whose writing system underwent a revolutionary simplification that merged different traditional sinograms into a single simplified form (e.g., both "台风" meaning "stage manners" and "颱風" meaning "typhoon" are merged as "台风"). this merging complicates the homonym system in chinese compared to english. the inherent ambiguity of chinese sinograms thus makes them an ideal testbed for exploring the capabilities and limitations of contextualized representations. ## analyses the results on layer-wise representation of english and chinese homonyms are shown in figure 2 , showing the averaged d-score over all homonyms. ## conclusion we presented a comprehensive analysis on english and chinese homonym representations, spanning same and different pos categories, across 21 plms and llms in four model families. patterns of layerwise trajectories of d-score were found to differ across models and languages, suggesting that these different models might excel differently in encoding distinct levels of linguistic information (e.g., meanings, pos, etc..) towards encoding distinct homonym representations. model architecture and pretraining data portion appear to be important factors as bidirectional and multilingual models excel in homonym disambiguation. we also highlighted the functional role of pos in word-sense disambiguation as models disambiguate same-pos homonyms better than different-pos homonyms. intriguingly, this is opposite to what have been observed in human studies. to conclude, the individual differences between llms complicate our understanding of their inner workings, there is a need to conduct rigorous, controlled experiments using purposefully manipulated input, in order to enhance interpretability in future llm research.