T axo LL a MA : W ord N et-based Model for Solving Multiple Lexical Semantic Tasks (2024)

Abstract: In this paper, we explore the capabilities of LLMs in capturing lexical-semantic knowledge from WordNet on the example of the LLaMA-2-7b model and test it on multiple lexical semantic tasks. As the outcome of our experiments, we present TaxoLLaMA, the “all-in-one” model for taxonomy-related tasks, lightweight due to 4-bit quantization and LoRA. TaxoLLaMA achieves 11 SOTA results, and 4 top-2 results out of 16 tasks on the Taxonomy Enrichment, Hypernym Discovery, Taxonomy Construction, and Lexical Entailment tasks. Moreover, it demonstrates a very strong zero-shot performance on Lexical Entailment and Taxonomy Construction with no fine-tuning. We also explore its hidden multilingual and domain adaptation capabilities with a little tuning or few-shot learning. All datasets, code, and pre-trained models are available online (code: https://github.com/VityaVitalich/TaxoLLaMA)

## introduction recent studies in natural language processing widely utilize large language models (llms) for their capability to store extensive knowledge @xcite @xcite and to adapt quickly to different tasks via incontext learning without backpropagation @xcite . however, the application of llms to the classical lexical semantic tasks still remains understudied: for instance, no recent experiments with llms have been performed for the hypernym discovery task @xcite for different domains and languages. in taxonomy enrichment, llms are mostly used to extract vector representations which are further processed with a complex pipeline @xcite . our work aims to investigate the capabilities of llms in addressing four tasks requiring taxonomic knowledge: hypernym discovery, taxonomy en-1 https://github.com/vityavitalich/ taxollama hyponym: taxollama richment, lexical entailment, and taxonomy construction. we hypothesize that the model finetuned with hypernym (is-a relationships) would be useful for solving taxonomy-related tasks. to verify this hypothesis, we develop a method inspired by @xcite to compile a taxonomyfocused instruction tuning dataset, sourced from english wordnet @xcite , to bring the implicit word knowledge of an llm to the forefront when addressing lexical semantic tasks. having trained our model in this specialized setting, we are releasing the taxollama -the finetuned version of the llama-2-7b model @xcite -that is capable of solving tasks requiring taxonomic knowledge. figure 1 presents the main idea of the model finetuning process. taxollama operates effectively in a zero-shot setting, surpassing sota results in lexical entailment and taxonomy construction. with additional tuning, it also achieves sota performance in the hypernym discovery task across several languages and in half of the taxonomy enrichment tasks. furthermore, we have optimized taxollama to be lightweight through 4-bit quantization @xcite and the application of lora @xcite , making it feasible to run on gpu devices with only 4.8gb of gpu for forward pass and 5.5gb for fine-tuning, ensuring its accessibility for widespread use, e.g. using colab foot_0 . the contributions of the paper are as follows: in this section, we briefly describe previous approaches to the lexical semantics tasks that we are experimenting with in the paper. ## methodology this section outlines the process of data collection and the subsequent training of the model. ## experiments in this section, we assess the proposed methodology and the finetuned models, taxollama and taxollama-bench, on four lexical semantic tasks: hypernym discovery, taxonomy enrichment, lexical entailment, and taxonomy construction. we evaluate models in a zero-shot setting and after fine-tuning on the provided train sets for each task. ## results this section describes the main results of generative and ranking setup experiments for all tasks. ## conclusion in this paper, we introduce taxollama-an llm finetuned on wordnet-3.0, capable of solving various lexical semantic tasks via hypernym prediction. it achieved sota results in 11 out of 16 tasks and securing the second position in 4 tasks. manual and chatgpt-based error analysis shows that the most errors (75%) are overly broad predicted concepts, due to overfitting to the idiosyncratic wordnet structure and inability to adapt to the target datasets. experiments showed that, definitions greatly contribute to the final scores for taxonomy enrichment, similarly to @xcite , as they help to better disambiguate input words. regarding error analysis, the most difficult datasets were mags @xcite , as they greatly differ from the data used for training of our model.