W ordnet for Definition Augmentation with Encoder-Decoder Architecture (2023)

Abstract: Data augmentation is a difficult task in Natural Language Processing. Simple methods that can be relatively easily applied in other domains like insertion, deletion or substitution, mostly result in changing the sentence meaning significantly and obtaining an incorrect example. Wordnets are potentially a perfect source of rich and high quality data that when integrated with the powerful capacity of generative models can help to solve this complex task. In this work, we use plWordNet, which is a wordnet of the Polish language, to explore the capability of encoder-decoder architectures in data augmentation of sense glosses. We discuss the limitations of generative methods and perform qualitative review of generated data samples.

## introduction transformer models have appeared to be very successful in solving a large variety of natural language processing tasks and applications. the research on neural language modeling has been intensified in recent years and has yielded many new developments, such as pre-trained autoregressive language models for text generation. text generation models such as bart @xcite , gpt @xcite or t5 @xcite have increased the performance even further, due to their few-shot abilities @xcite . the knowledge resources such as wordnets @xcite are often incomplete and still require constant development, especially for lowresourced languages. in słowosieć @xcite (also called plwordnet) -a wordnet of the polish language, one of the largest wordnets in the world -over 40% senses still lack a definition, and over 60% of senses do not have any sense use example. this area might be addressed by utilising large language models pre-trained on text generation tasks. adding missing definitions and sense use examples is a crucial task for further wordnet development. the definition generation problem is tightly interconnected with word sense disambiguation (wsd) problem, as the words have different meanings in different contexts. the modern language models have significantly improved wsd performance in recent years. transformer-based models such as bert @xcite have proved to be very effective in contextual word sense recognition @xcite . while very effective, large language models require at least a small data sample to effectively fine-tune them for the wsd task. nevertheless, large pre-trained language models with billions of parameters have been shown to require less training data to effectively tune them for downstream tasks @xcite . in this paper, we investigate generation abilities of large pre-trained language models in the task of wordnet gloss generation for the polish language. we treat this problem as a data augmentation problem, as some senses in under-resourced wordnets are missing their definitions. we evaluate gloss generation performance on the example of polish wordnet -słowosieć @xcite -in the version 4.2. foot_0 ## related work the acquisition and completion of missing sense glosses has been addressed in the literature in many different ways. enrichment of synset glosses in wordnets can be partially achieved by utilising machine translation models @xcite . however, these approaches do not take into account the discrepancy between sense inventories in different languages, as some senses do not exist in the source or target languages. thus, an automated translation of princeton wordnet glosses @xcite to other language might not be able to completely solve the task of gloss completion. the other approaches rely on interlinking the wordnets with external resources and semantic networks such as multilingual thesauri in linked open data, wikipedia foot_1 , wikidata foot_2 , babelnet @xcite , or with open multilingual wordnet grid @xcite . some solutions solve the problem as a joint task in which translations and potential glosses available in large semantic networks are analysed with wsd algorithms to increase the accuracy of gloss acquisition @xcite . still, an overall coverage of senses is strongly dependent on the target domain of application, and for specific domains the wsd models are biased towards more frequent senses. the closest to our work are generative approaches in which the encoder-decoder architectures are used to generate definitions in an autoregressive manner and treating the language models as knowledge bases @xcite @xcite . the approaches such as @xcite utilise large pre-trained transformers, mainly t5 @xcite and bart @xcite ) models, to generate definitions. the solution proposed in @xcite is the closest to our work since it's based on the same pre-trained t5 transformer architecture, but the authors have added reranking models to control the specificity of generated sense definitions. in our work we expand the research on generative definition acquisition and investigate the performance of raw generative language models for the polish language. the japanese corpus for definition generation @xcite ) also provides words with usage and definition, but it was generated via linking wikidata items with sentences in wikipedia articles. ## evaluation output of generative models was a definition for a given word in relation to the particular context and the evaluation of such an output is a nontrivial task. in language generation different evaluation metrics are used. we chose bleu @xcite and rouge @xcite metrics which are widely applied in many benchmarks. this automatic evaluation gave us information, if a model is overfitting to provided data or not. we could also estimate the difference between basic and large models performance on the test set. but to evaluate definitions properly, syntactic-level metrics are not sufficient. that is why we also performed manual validation of the generated definitions together with doing error analysis of the model's predictions. the manual validation was performed by professional lexicographers specialising in wordnets. we used a subset of error tags from @xcite as a basis for our manual evaluation, namely: to decrease memorisation impact on our evaluation, we evaluated the predictions by ensuring both the lemmas and the definitions in our test data were not included in the training dataset. we also provide the results with respect to part-of-speech of analysed lemmas. ## results and discussion we also provided some examples of errors in the generated definitions (see the overall results are presented in ## conclusion the plwordnet definitions and usage examples are perfect data source for fine-tuning t5 generative models. fine-tuned models are able to understand the new task and partially to generate definitions of unseen word senses. there is a significant difference in performance for seen and unseen words. models are able to memorize definitions for examples in training set and modify them. large model performs better than base, not only in automatic evaluation but also when evaluated by trained linguist. it is able to generalize better and recall more from it's memory. as our results are promising, we believe that further research in this area will lead to models that are able to generalize better and define polysemic words properly in different contexts. those models can be applied to wsd, entity linking and other knowledge intensive tasks.