In the LLM era, Word Sense Induction remains unsolved (2025)

Abstract: In the absence of sense-annotated data, word sense induction (WSI) is a compelling alternative to word sense disambiguation, particularly in low-resource or domain-specific settings. In this paper, we emphasize methodological problems in current WSI evaluation. We propose an evaluation on a SemCor-derived dataset, respecting the original corpus polysemy and frequency distributions. We assess pre-trained embeddings and clustering algorithms across parts of speech, and propose and evaluate an LLM-based WSI method for English. We evaluate data augmentation sources (LLM-generated, corpus and lexicon), and semi-supervised scenarios using Wiktionary for data augmentation, must-link constraints, number of clusters per lemma.We find that no unsupervised method (whether ours or previous) surpasses the strong “one cluster per lemma” heuristic (1cpl). We also show that (i) results and best systems may vary across POS, (ii) LLMs have troubles performing this task, (iii) data augmentation is beneficial and (iv) capitalizing on Wiktionary does help. It surpasses previous SOTA system on our test set by 3.3%. WSI is not solved, and calls for a better articulation of lexicons and LLMs’ lexical semantics capabilities.

## introduction disambiguating the senses of potentially ambiguous words in a text (i.e. word sense disambiguation, wsd) is a historic nlp task, essential for obtaining a formal representation of a text's meaning. however, this task has the drawbacks of (i) relying on predefined sense inventories of arbitrary granularity and ill-suited for specialized domains, and (ii) requiring labor intensive sense-annotated data, unavailable for most languages of the world. this requirement still holds in the large language models (llm) era: @xcite and @xcite show that open llms outperform bertbased supervised systems only when fine-tuned on sense-labeled data. the word sense induction task (wsi) does away with the need for a predefined sense inventory and sense-labeled data (except for evaluation), albeit at the expense of quality. in addition, one of the standard techniques in wsi is to cluster vector representations of a lemma's occurrences. when applied to all content words in a corpus, it provides corpusdependent pseudo-sense labeling @xcite . although in the llms era the utility of induced senses is less clear for downstream applications, it remains central in computationally less intensive environments, corpus studies such as lexical change detection @xcite , or specific contexts such as scientific literature mining and sense-aware retrieval @xcite . our work focuses on a "full-corpus wsi scenario", taking a corpus as input, inducing senses for all content-word lemmas occurring more than once (typically by clustering occurrences), yielding as a by-product pseudo-sense annotations. we note that popular wsi datasets created for semeval shared tasks @xcite (i) exhibit artificial polysemy levels (because unambiguous lemmas are trivial to process, these datasets over-represent polysemous ones, thus not addressing the task of discovering which lemma is polysemous), (ii) and some exhibit artificial selection of lemmas and number of occurrences per lemma. state-of-the-art wsi systems for english are thus biased towards these unrealistic distributions, and it remains to prove that they work as well when faced with more natural data. this paper makes the following contributions: ## related work we focus here on previous work on wsi compatible with the full-corpus scenario (hence we ignore wsi approaches based on a lexical network such as @xcite ). such corpus-based approaches can be categorized into six groups: importantly, these llm approaches to wsd and wsi are computationally intensive, prompting the llm once for each instance to disambiguate. in the following, (i) to the best of our knowledge, we report the first results of directly asking the llm to cluster sets of instances of a given lemma, a much more lightweight technique; (ii) we compare wsi performance across datasets, target lemma pos, and evaluation metrics for models falling into category (iii) (lsdp) and (iv) (polylm). the approaches (i) and (ii) are the focus of §5. ## issues in wsi evaluation the evaluation of wsi models relies on datasets containing manual sense assignments for instances of a set of lemmas, and on metrics assessing how well the automatic assignment matches the manual one. datasets have issues related to lemma and instance selection and dev/test data splits. they are often associated to heterogeneous and incomplete metrics, resulting in a complex landscape in which it is extremely difficult to compare, reproduce and/or replicate results. ## variation of systems' performance across datasets, metrics and pos in this section, we investigate the impact of datasets' characteristics (polysemy level, target lemmas pos) and of evaluation metrics on performance in wsi. we first describe our wsi evaluation dataset extracted from semcor @xcite , respecting the original distributions of corpus occurrences and senses. we then compare wsi performance across (i) recent state-of-the art systems (lsdp and polylm) 5 , and (ii) direct llm prompting for wsi. ## investigating full-corpus wsi we reported earlier that the top-performing systems do not surpass the 1cpl baseline when switching to a more naturally distributed dataset. this suggests that over-representation of polysemy in earlier datasets may have influenced the systems' design. in this section, we investigate the performance achievable on semcor-wsi using the basic technique of clustering the contextualized embeddings of a given lemma instances and we hypothesize that performance may vary depending on polysemy injecting lexical information either using unlabeled . we perform a grid search using (i) two clustering algorithms which automatically determine the number of clusters (x-means and ag silh ), (ii) two bert plms, plus plms fine-tuned to better perform on wic task. this allows us to assess which model and algorithm combination performs best on a more naturally distributed dataset. ## investigating data augmentation in this section, we investigate the simple technique of adding unlabeled examples to the set of instances to cluster. augmenting the set of instances makes it denser, potentially creating new similarity links, in particular for lemmas with originally few instances. moreover in such cases, considering more instances helps to avoid undefined silhouette cases, defaulting to one cluster (cf. appendix l). we investigate 1) unsupervised augmentation, adding either attested examples from external corpora or synthetic examples generated by llms; 2) semi-supervised augmentation, where we leverage wiktionary examples for either direct dataset augmentation (based on the wiktionary senses), and/or fine-tuning the embedding model (bert-l-wikt model @xcite already used in previous sections). ## conclusions in this paper, we advocated for evaluating wsi on data respecting more natural distributions of occurrences and number of senses per lemma. we proposed experiments on english, evaluated on an extract of semcor, both reusing state-of-theart previous methods, and investigating an llm prompting technique, data augmentation, and a semi-supervised setting where the english wiktionary is used in three ways (as a source for finetuning a bert model on the word-in-context task @xcite , for data augmentation and for must-link clustering constraints). the llm prompting technique we proposed lays far behind, calling for better leveraging the lexical semantics knowledge of llms. our striking conclusion is that, when considering a dataset following a more natural sense distribution and polysemy level, none of the fully unsupervised systems we tested surpass the simple baseline of clustering all instances of the same lemma into a single cluster (considering a dataset with an average polysemy close to 2, cf. simple data augmentation allows to surpass the much more complex previous sota model (polylm). bert embeddings fine-tuned using contrastive learning on wiktionary examples @xcite are always better compared to the original bert embeddings. more generally, we showed several ways to leverage wiktionary allowing to surpass 1cpl on the dev and test sets (must-link constraints, data augmentation, definition of the number of clusters, fine-tuning of plms). note this can be applied for the languages having a large wiktionary or another electronic lexicon.