Can LLM s assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation (2024)

Abstract: Ambiguous words are often found within modern digital communications. Lexical ambiguity challenges traditional Word Sense Disambiguation (WSD) methods, due to limited data. Consequently, the efficiency of translation, information retrieval, and question-answering systems is hindered by these limitations. This study investigates the use of Large Language Models (LLMs) to improve WSD using a novel approach combining a systematic prompt augmentation mechanism with a knowledge base (KB) consisting of different sense interpretations. The proposed method incorporates a human-in-loop approach for prompt augmentation where prompt is supported by Part-of-Speech (POS) tagging, synonyms of ambiguous words, aspect-based sense filtering and few-shot prompting to guide the LLM. By utilizing a few-shot Chain of Thought (COT) prompting-based approach, this work demonstrates a substantial improvement in performance. The evaluation was conducted using FEWS test data and sense tags. This research advances accurate word interpretation in social media and digital communication.

## introduction in natural language processing (nlp), identifying the exact meaning of words within sentences is key. this is because misunderstandings of word sense can lead to false information which results in misinformation. in the context of cyber threat intelligence, such misinformation and ambiguity can conceal the true nature of threats, leading to inadequate responses and potentially leaving systems vulnerable @xcite . words that have multiple meanings (polysemy) are a major challenge that nlp can overcome using computational methods. even though there's been a lot of research on figuring out the right meaning of words (wsd) in different languages, using various methods, it has not been completely successful @xcite . for instance, previous studies performed on wsd have not been able to solve some tricky cases due to its poor contextual understanding by the models @xcite . however, research shows that a word's meaning is closely linked to the words around it proving that isolated word analysis is insufficient to perform correct sense identification @xcite . therefore, proper word sense with positional value, pos tag and aspect of the sentence is being considered for accurate models. llms and generative ai, which are based on transformers, show promising results in the contextual understanding of words @xcite . these models have shown a strong ability to handle complex language tasks because of extensive training on vast amounts of data. finetuning such base models for downstream tasks such as question answering and domain specific knowledge generation has shown promising results @xcite . in our study, we were mainly focused on evaluating how llms can be used for specific downstream tasks like wsd by investigating their capability of identifying the right meaning of words. more specifically, we want to understand if llms can be used to match words with multiple meanings to their correct sense in a sentence. even if research focuses on supervised wsd methods (often using paradigmatic relationships like synonyms, hyponyms, and hypernyms), this study explores an alternative path to ensemble different computational techniques like kb to improve the prediction accuracy for wsd. prior work on wsd has attempted to extract the correct gloss/sense tag by reframing the problem into different aspects. however, the major ## related works ambiguity in natural language poses a significant challenge for various natural language processing tasks, with wsd being a fundamental problem. wsd has been one of the continuing research areas in different languages as the proper word sense directly impacts many nlp tasks like machine translation, question answering, text summarization, text classification, and word sense induction. several advanced new neural architectures have been suggested by many researchers for the wsd task by integrating kb models @xcite . different nlp techniques are grouped to perform the effective wsd task and an overview of these works can be found below. ## methodology according to previous studies, it is evident that the usage of llms based approaches for wsd tasks can be effective. in our work, we evaluate the understanding of lexical ambiguity by different llms using different computational approaches like parameter tuning and prompt augmentation. the optimal prompt has been constructed using an iterative approach. the results were analyzed on corner cases and the augmented prompts have been tested and evaluated. the prompt augmentation process utilized different techniques like few-shot cot prompting and its variations. ## conclusion and future directions this research illustrates the effectiveness of integrating prompt augmentation techniques using large language models with a knowledgedriven strategy to address word sense ambiguity. future work should focus on evaluating these techniques on comprehensive datasets such as semcor, senseeval, and semeval to provide a robust validation of their efficacy. additionally, exploring the potential for enhancing performance through the incorporation of additional parameters warrants further investigation. by accurately disambiguating the true meaning of words within their context, we can significantly enhance cyber threat intelligence efforts, thereby curbing the spread of misinformation in natural text. this study introduces a novel method for word sense disambiguation that incorporates prompt augmentation within a human-in-the-loop framework, yielding promising results and suggesting practical utility for various nlpbased tasks. subsequent research will aim to expand this approach across a diverse array of fine-tuned commercial and open-source models, to validate its generalizability and explore its applicability across various real-world scenarios. this comprehensive approach not only advances the state-of-the-art in wsd but also opens new avenues for practical applications in natural language processing. do the following tasks. 1. examine the sentence below. "{sentence}". 2. identify the meaning of the word enclosed within @xmath0. you need to consider the total sentence before you get the exact meaning of the word. use the following format for the output. <json object that contain ambiguity word and the finalized @xmath1.4] you are going to identify the corresponding sense tag of an ambiguous word in english sentences. use multiple reasoning strategies to increase confidence in your answer. 1. the word "{wordwsd}" has different meanings. below are possible meanings. comprehend the sense tags and meanings: {filtered_definitions} 2. you can learn more on the usage of each word and the meaning through the examples below. each sentence is followed by its corresponding sense id. "{examples}" 3. now carefully examine the sentence below. the ambiguous word is enclosed @xmath2."{sentence}" 4. analyze the sentence using the following three approaches. for each approach, identify the meaning of the ambiguous word and the corresponding sense ids. if there are multiple sense ids, separate them with commas. strategy 1: focus on keywords in the sentence surrounding the ambiguous word. see which sense definition aligns best with these keywords. strategy 2: consider the part of speech (noun, verb, adjective, etc.) of the ambiguous word in the sentence and how it functions within the sentence structure. choose the sense definition that fits this grammatical role. strategy 3: think about the overall topic and intent of the sentence. decide on the sense of the word that makes the most logical sense within the wider context. 5.compare the sense id(s) identified by each strategy. if all three strategies agree on the same sense id, that is your most confident answer. if two strategies agree on a same sense id, that becomes your answer. if there is a disagreement, list the sense id(s) from each strategy for further review. 6. return a json object containing the following: below are possible meanings. comprehend the sense tags and meanings. synonyms are provided if available. {filtered_definitions} 2. you can learn more on the usage of each word and the its sense through the examples below. each sentence is followed by its corresponding sense id. "{examples}" 3. now carefully examine the sentence below. the ambiguous word is enclosed @xmath3."{sentence}" 4. analyze the sentence using the following techniques and identify the meaning of the ambiguous word. focus on keywords in the sentence surrounding the ambiguous word. think about the overall topic and intent of the sentence. decide on the sense of the word that makes the most logical sense within the context. 5. based on the identified meaning, try to find the most appropriate senseids from the below sense tag list. you are given definition of each sense tag too."{filtered_definitions}". 6. if you have more than one senseids identified after above steps, you can return the senseids in order of confident level, follow the given format to return the value. 7. return a json object containing the following: