A sk2 T ransformers: Zero-Shot Domain labelling with Pretrained Language Models (2021)

Abstract: In this paper we present a system that exploits different pre-trained Language Models for assigning domain labels to WordNet synsets without any kind of supervision. Furthermore, the system is not restricted to use a particular set of domain labels. We exploit the knowledge encoded within different off-the-shelf pre-trained Language Models and task formulations to infer the domain label of a particular WordNet definition. The proposed zero-shot system achieves a new state-of-the-art on the English dataset used in the evaluation.

## introduction the whole natural language processing (nlp) research area have been accelerated with the advent of the unsupervised pre-trained language models. first with elmo @xcite and then with bert @xcite the paradigm of using pre-trained language models for finetuning on a particular nlp task has became the new standard approach, replacing the more traditional knowledge-based and fully supervised approaches. currently, as the size of the corpus and models increase, the research community has observed that the transfer learning approach has the capacity to work without any or with a very small fine-tuning. some examples of the strength of this approach are gpt-2 @xcite or more recently @xcite ) that shows the ability of these huge pre-trained language models to solve tasks for which have not even trained. recently, with the arrival of the gpt-3 new ways to perform zero and few shot approaches have been discovered. these approaches propose the inclusion of a small number of supervised examples in the input as a hint for the model. the model then, just by looking a small set of examples, is able to complete successfully the task at hand. @xcite report that they solve a wide range of nlp tasks just following the previous approach. however, this approach only looks appropriate when the model is large enough. in this paper we exploit the domain knowledge already encoded within the existing pretrained language models to enrich the wordnet @xcite synsets and glosses with domain labels. we explore and evaluate different pretrained language models and pattern objectives. for instance, consider the example shown in interestingly, without any training on the task at hand, the proposed zero-shot system obtains an f1 score of 92.4% on the english dataset used in the evaluation. all the implementation code along with the experiments is freely available on a github repository foot_0 . after this short introduction, the next section presents previous work on domain labelling of wordnet. section 3 presents our approach, section 4 the experimental setup and section 5 the results from our experiments. finally, section 6 revises the main conclusions and the future work. ## related work building large and rich lexical knowledge bases is a very costly effort which involves large research groups for long periods of development. starting from version 3.0, princeton wordnet has associated topic information with a subset of its synsets. this topic labeling is achieved through pointers from a source synset to a target synset representing the topic. wordnet uses 440 topics and the most frequent one @xmath2, jurisprudence>. in order to reduce the manual effort required, a few semi-automatic and fully automatic methods have been applied for associating domain labels to synsets. for instance, wordnet domains foot_1 (wnd) is a lexical resource where synsets have been semi-automatically annotated with one or more domain labels from a set of 165 hierarchically organized domains @xcite . the uses of wnd include the possibility to reduce the polysemy degree of the words, grouping those senses that belong to the same domain @xcite . but the semiautomatic method used to develop this resource was far from being perfect. for instance, the noun @xmath3, frogman, underwater @xmath4, @xmath5. moreover, many synsets have been labelled as factotum meaning that the synset cannot be labelled with a particular domain. wnd also provides mappings to wordnet topics and also to wikipedia categories. more recently, babeldomains foot_5 @xcite propose an automatic method that propagates the knowledge categories from the wikipedia to wordnet by exploiting both distributional and graph-based clues. as domains of knowledge, babeldomains opted for domains from the wikipedia featured articles page 7 . this page contains a set of thirty-two domains of knowledge. when labelling wordnet synsets with these domains, babeldomains reports a precision of 81.7, a recall of 68.7 and an f1 score of 74.6. unfortunately, as these numbers suggest not all wordnet synsets have been labelled with a domain. for instance, the @xmath6, @xmath7. it is worth to note that all these methods depart from a particular set of domain labels (or categories) manually assigned to a set of wordnet synsets (or wikipedia pages). then, these labels are propagated through the wordnet structure following automatic or semi-automatic methods. in contrast, our zero-shot method does not require an initial manual annotation. furthermore, it is not designed for a particular set of domain labels. that is, it can be applied to label from scratch any dictionary or lexical knowledge base (or wordnet) with distinct sets of domain labels. ## using pre-trained lms for domain labelling recent studies such as the one of @xcite shows that when increasing the size of the model, the capacity to solve different tasks with just a few positive examples also increases (few-shot learning). however, very large language models also have important hardware requirements (i.e. large ram gpus). thus, we decided to keep the size of the models used manage-definition: hospital: a health facility where patients receive treatment. the domain of the sentence is about medicine 0.77 biology 0.08 business 0.04 culture 0.02 the task where we focused on is the domain labelling of wordnet glosses. this task consist in the following. given a wordnet gloss g to predict the corresponding domain d of the wordnet concept defined. in this paper, the domains are taken from babeldomains @xcite . supervised domain labelling can be solved as any other multiclass problem, where the output of the model is a class probability distribution. in our zero-shot experiments we did not modify any of the pre-trained models. we just reformulate the domain labelling task to match with the lms training objective. ## experimental setting this section describes our experimental setup. we introduce the pre-trained language models and the dataset used. for the case of the language models, we have tested bert @xcite , roberta @xcite and bart @xcite . for the dataset, we have used the one released by @xcite based on wordnet. ## evaluation and results on the other hand, as one of our system is based on a generative approach (mlm) the applied restrictions may not show the real performance of the method. so, we decided to at least do an small qualitative review of the approach. ## conclusions and future work in this paper we have explored some approaches for domain labelling of wordnet glosses by exploiting pre-trained lm in a zero-shot manner. we have presented a simple approach that achieves a new state-of the art on the babeldomain dataset. even if we have focused on domain labelling of wordnet glosses, our method seems to be robust enough to be adapted to work on tasks such as sentiment analysis or other type of text classification. in particular, we think that the approach can be very useful when no annotated data is available. for the future, we have considered three main objectives. first, we plan to apply this approach to other sources of domain information such as wordnet topics and wordnet domains. we will also explore how to deal with definitions with generic domains (with no babeldomains labels or with wordnet domains factotum label). second, we also aim to explore the cross-lingual capabilities of pre-trained language models for domain labelling of non-english wordnets and other lexical resources. finally, we also plan to explore the utility of these findings in the word sense disambiguation task.