On Functional Competence of LLM s for Linguistic Disambiguation (2024)

Abstract: We study some Large Language Models to explore their deficiencies in resolving sense ambiguities. In this connection, we evaluate their performance on well-known word sense disambiguation datasets. Word Sense Disambiguation (WSD) has been a long-standing NLP problem, which has given rise to many evaluation datasets and models over the decades. Recently the emergence of Large Language Models (LLM) raises much hope in improving accuracy. In this work, we evaluate word sense disambiguation capabilities of four LLMs: OpenAI’s ChatGPT-3.5, Mistral’s 7b parameter model, Meta’s Llama 70b, and Google’s Gemini Pro. We evaluate many well-established datasets containing a variety of texts and senses on these. After observing the performances of some datasets, we selectively study some failure cases and identify the reasons for failures. We explore human judgments that would correct these failures. Our findings suggest that many failure cases are related to a lack of world knowledge and the reasoning to amalgamate this knowledge rather than the lack of linguistic knowledge. We categorize the judgments so that the next generation of LLMs can improve by incorporating deeper world knowledge and reasoning. We conclude that word sense disambiguation could serve as a guide for probing the reasoning power of LLMs to measure their functional competency. We also list the accuracy of these datasets. We find that on many occasions, accuracy drops to below 70%, which is much less than that of well-performing existing models.

## introduction large language models have been shown to achieve human-like linguistic competence. in various linguistic tasks, their abilities have been documented @xcite , @xcite . however, conflating linguistic competence with common-sense reasoning abilities has also been de-cried among researchers. in one experiment @xcite , researchers report that language models still do not show evidence of cognitive abilities on par with humans. some studies @xcite ) make the competencies of language models distinct: formal and functional linguistic competence. whereas formal linguistics competence manifests in forming coherent, fluent, and syntactically correct texts, functional competence is evidenced in identifying motives and formulating a strategy with world knowledge to decipher the true intention of the writer. though language models excel in formal competence, they are not known to perform at the human level on functional competence. why is functional competence important in nlp tasks? one answer could be functional competence could enhance machine translation performance. in transferring meaning from one language to another, the senses must be interpreted. many words have more than one sense. divining the sense of a word requires formal as well as functional competence. for example, consider the following sentence: at first blush it seemed that what was striking about him rested on the fact that his dress was exotic, his person foreign. we will consider two definitions of the word person: the word person could be interpreted as a "human being" considering the surrounding collocating words. an alternative interpretation could be "the physical body of a being seen as distinct from the mind, character", which is the correct one. while the former interpretation is derived by applying formal competence, which involves analyzing the syntactic relations among a text's constituents, the latter definition can only be determined after considering the historical use of person. arriving at the latter meaning requires greater cognitive deliberation and a broader understanding of world knowledge. the inability to settle on the proper meaning would result in suboptimal translations. that word sense disambiguation (wsd) helps in machine translation has been documented in much research @xcite , @xcite , @xcite , @xcite , @xcite . most well-performing wsd methods rely on supervised machine learning. using artificial neural networks have been shown to improve wsd performance @xcite @xcite @xcite @xcite @xcite . existing datasets for evaluating wsd performance have been a by-product of decades-long research, which have been time-tested, some containing infrequent use of senses. we intend to use these datasets for our experiments. in this study, large language models (llm) are prompted with the examples of the datasets described in subsection 7.1 1 . the responses are matched and tallied to summarize overall performance (figure 1 ). in summary, our contribution is as follows: we share some insights into why, in some wsd cases, llms fail by highlighting certain functional deficiencies, and we present findings that wsd datasets could be repurposed to gauge the reasoning power of llms. the remaining sections are organized as follows: sections 2, 3, and 4 discuss the similarities and differences between llms and humans. sections 5, 6, 7, and 8 provide detailed descriptions of our experiments. 1 the experiment could be reproduced with the code available at functional competence of llms 2 linguistic regularities and formal ## divergence between llms and humans whereas llm's human-like processing of language has been documented, some research papers highlight certain deficiencies compared to humans in reasoning tasks. take for example a theory of mind task and its alteration (ullman, 2023): original task: here is a bag filled with popcorn. there is no chocolate in the bag. yet, the label on the bag says "chocolate" and not "popcorn." sam finds the bag. she had never seen the bag before. she cannot see what is inside the bag. she reads the label. altered task: here is a bag filled with popcorn. there is no chocolate in the bag. the bag is made of transparent plastic, so you can see what is inside. yet, the label on the bag says 'chocolate' and not 'popcorn.' sam finds the bag. she had never seen the bag before. sam reads the label. gpt3.5 was prompted with predicting the following: she believes that the bag is full of __, the machine got the answer right in the original task (chocolate), but not in the altered version. given llm's excellent linguistic ability and yetunproven performance on reasoning at the human level, researchers are apt to classify the llm capabilities into two: formal and functional competencies. this motivation comes from observing brain activities. the language network in the human brain is quite distinct from the day-to-day reasoning center as revealed in fmri scans @xcite . in other words, linguistic abilities should be separately considered from the world knowledge. ## word sense disambiguation and functional competence in evaluating the wsd performance of the llms we find that some difficult disambiguation tasks that machines fail to perform, rely on having world knowledge in addition to linguistic knowledge. we categorize these with examples. to the best of our knowledge, these categories have not been previously documented. some are related to historical, old english, cultural, geographical, trade relational, religious, satiric/figurative use of languages, and spatial knowledge. as an example consider the following sentence: the discovery of the mines of america ... does not seem to have had any very sensible effect upon the prices of things in england. there are eight different senses for the target word sensible, of which we are listing just two: sense#1 is a false choice. to detect the correct choice sense#2, one must reason with knowledge involving history, trade relations, and possibly geography. here is our analysis of why sense#2 is the correct choice: historically america and england have been closely related in terms of commerce. close relation implies some effect of events in one country on another. it is common knowledge that any effect should be perceivable/appreciable. the writer is informing of no effect, which is counter-intuitive; but that is what writers do -provide surprising information. to disambiguate, knowledge of trade relations, and possibly geography is needed. and, of course, good reasoning. we provide a taxonomy of failure cases in tables 1. more can be found in the appendix. 5 background on word sense ## methodology we test the word sense disambiguation capability of some llms. our choice of methodology for wsd research is influenced by established knowledge about the pitfalls of existing corpus and sense definitions. ## experimentation the datasets we use in our experiments contain a variety of sense keys: some use their self-conceived sense keys extracted from popular text sources such as wikipedia and wiktionary. some use wordnet sense keys. still, others could be found using some other lexicon's keys, for example, babelnet @xcite . not all datasets present wsd as a classification task. for example, word in context (wic) dataset (pilehvar and camacho-collados, 2018) presents each evaluation sample as a simple true or false by giving two sentences, probing the llm to verify whether the two sentences carry the same meaning of the target word (figure 5a and figure 5b ). data sets are in different formats: some in xml format while others in simple texts. some datasets contain the sense keys, whereas others refer to senses from external sense inventories. after extracting the sentences and collecting definitions of senses suitable for prompting, we prepare prompts similar to figure 1 . @xcite this dataset is the largest. it also contains multilingual content. however, the dataset lacks proper human evaluation -random samples reveal that it has 67.7% inter-annotator agreement. we do not include this dataset in our experiments. ## discussion given that the llms have not been fine-tuned, it is understandable from the test results that accuracy is comparable to the state-of-the-art models on wsd. sometimes a language model fails to accurately identify a sense due to its lack of spatial knowledge; other times it fails because it seems not to be able to put the text in historical context; still other times the lack of application of humans' social relation is to be the reason for failure. many disambiguation cases require knowledge from different avenues: political, spatial, cultural, historical, and the like. many researchers would sometimes club these missing pieces as commonsense knowledge. while investigating the failure cases, we prompted the llms to test their world knowledge. we discovered that by using different prompts, it can be confirmed that the llms appear to possess much of this knowledge. however, the failure arises when these models do not leverage knowledge across multiple dimensions to integrate it effectively. much research in the av-enue of reasoning is needed to further advancement of artificial general intelligence, which concurs with some research findings @xcite . we stop short of calling our results a benchmark since not all llms we considered are open-source and the technology is continuously evolving as a result of which it will be difficult to compare across generations of llms. ## conclusion in this research, we demonstrate that wsd involves not just the knowledge of language but world knowledge and the capability of piecing together facts from multiple sources -in other words, functional competence. our findings also suggest that wsd could be used to verify the reasoning power of llms. wsd datasets are aplenty, and some have been human-validated. we conclude that it is worth paying heed to improving the wsd capabilities of llms and using these datasets in a novel way to probe. we also release a taxonomy of failure cases requiring world knowledge for wsd, which could further research in this direction. large-scale task planning. advances in neural information processing systems, 36. zhi zhong and hwee tou ng. 2010. it makes sense: a wide-coverage word sense disambiguation system for free text. in proceedings of the acl 2010 system demonstrations, pages 78-83, uppsala, sweden. association for computational linguistics.