No clues good clues: out of context Lexical Relation Classification (2023)

Abstract: The accurate prediction of lexical relations between words is a challenging task in Natural Language Processing (NLP). The most recent advances in this direction come with the use of pre-trained language models (PTLMs). A PTLM typically needs “well-formed” verbalized text to interact with it, either to fine-tune it or to exploit it. However, there are indications that commonly used PTLMs already encode enough linguistic knowledge to allow the use of minimal (or none) textual context for some linguistically motivated tasks, thus notably reducing human effort, the need for data pre-processing, and favoring techniques that are language neutral since do not rely on syntactic structures. In this work, we explore this idea for the tasks of lexical relation classification (LRC) and graded Lexical Entailment (LE). After fine-tuning PTLMs for LRC with different verbalizations, our evaluation results show that very simple prompts are competitive for LRC and significantly outperform graded LE SoTA. In order to gain a better insight into this phenomenon, we perform a number of quantitative statistical analyses on the results, as well as a qualitative visual exploration based on embedding projections.

## introduction lexical relation classification (lrc) is the task of predicting which lexical relation exists between two given words (e.g., 'tall' and 'small' are related by the antonymy relation), from a finite catalogue of lexical relations. discovering lexico-semantic relations between words has received attention in the nlp community since hearst's seminal research in 1992 on the automatic acquisition of hyponyms from large text corpora based on pre-designed patterns @xcite . despite many recent advancements, lrc continues to be an open research topic in the nlp field @xcite . applications of the task are numerous: automatic thesauri creation, paraphrasing, textual entailment, sentiment analysis, ontology learning, and ontology population, among others @xcite . the most recent advances in lrc come with the use of pre-trained language models (ptlms) based on the transformers architecture @xcite , which have been proven to capture a large amount of lexico-semantic knowledge from text successfully. one of the main benefits of the adoption of pltms is that, while they were trained for a general task (text generation) following a masked language model (mlm) objective in an unsupervised way, they can be easily adapted to different downstream tasks (e.g., text classification, text summarization, sentiment analysis) by introducing additional parameters and fine-tuning them using objective functions specific to the task. that avoids the need to train the model from scratch, still obtaining sota results, while decreasing computational costs and the need for very large amounts of data @xcite . more recently, the "pre-train, fine-tune" procedure is shifting in nlp tasks towards the "pre-train, prompt, and predict" paradigm @xcite . in that case, instead of adapting ptlms to the downstream task via fine-tuning, the task is reformulated to look more like those solved during the original model training with the help of a textual prompt. following the example in @xcite , when recognizing the emotion of a sentence, "i missed the bus today.", we may continue with a prompt "i felt very", and ask the ptlm to fill the blank with an emotion-bearing word. a ptlm typically needs "well-formed" verbalized text to interact with it, either to fine-tune it or to exploit it via prompt engineering. while some authors claim that longer, more complex verbalizations of the input data work best for real-world text classification tasks @xcite , or relation classification @xcite , other authors @xcite have collected indications in the opposite direction for a wide range of nlp tasks (such as paraphrasing, textual similarity, or sentiment analysis). we share the hypothesis that commonly used ptlms already encode enough linguistic knowledge to allow the use of minimal (or none) textual context for some linguistically motivated tasks. in such cases, very simple prompts work almost as well or even better than hand-crafted, more complex verbalizations. reducing the need of complex prompting notably reduces the need of human effort and the need for data pre-processing, and favors techniques that are language neutral since they do not rely on syntactic structures. in this work foot_0 , we explore this idea for the lrc task, and we extend it to graded lexical entailment (le), i.e., discovering the strength of the taxonomical asymmetric hyponymy-hypernymy relation between two words @xcite . in previous works, other authors have explored complex verbalizations for lrc @xcite while others have essayed shorter ones @xcite . however, there has been no systematic study on the impact of long/short prompting for lrc so far. to that end, we have experimented with different verbalizations of the training and test data in an lrc experiment. then, we analysed which verbalization produces better predictions for at least one of the lexico-semantic relations entailed between a pair of words. we experiment with widely used benchmarks for lrc namely, cogalexv @xcite , bless @xcite , evalution @xcite , k&h+n @xcite , and root9 @xcite . besides, we evaluate such models with the hyperlex @xcite dataset for graded le. our main contributions are: our paper is structured as follows: first, in section 3, we formally describe both the lrc task and the le task. secondly, in section 4, we describe the chosen templates for the input verbalizations, the used datasets and baselines we compare with, as well as the hyper parameter and fine-tuning setting of our models. then, in section 5, we analyze our results showing: a) our quantitative results, analyzing which template, model, and method work best on each dataset, b) the error analysis, checking how the distribution and linguistic characteristics of the different datasets affected the performance of our models and what examples and categories were the most difficult ones, and c) a visualization of the embedding projection, highlighting which layers are more informative for relation classification and how the model learns them through the different epochs. finally, in section 6, we summarize the conclusions and possible future work, stating the limitations of our work. ## related work in this section we give an overview of some related approaches that are relevant to our work. ## problem statement let @xmath0, . . . , w n } be a set of words (our vocabulary), and a sentence s be any finite sequence of words from v . the set of all sentences over v is denoted by s. given a word @xmath1, a context c of w is any sentence such that @xmath2. the set of all contexts of a word w is denoted by c w . a binary relation r between words is a subset of v × v . let us denote by r the set of all binary relations over the vocabulary v , that is, r is the power set of v × v . we say that a set of relations, @xmath3, . . . , r k }, where r @xmath4, is mutually exclusive if the relations in r are disjoint; and we say that r is complete if the union of the relations is equal to v × v . note that we can make a relation set r complete by adding a relation named unknown, which is the complementary of all the relations in r. we consider that any context of two words induces a relation from a predefined set of relations, that is, there exists a function f r : @xmath5, where for instance, given the set of relations @xmath6, unknown}, the common context for the words bank and river, "i play by the bank of the river", induces the relation partof, while "i will deposit the money in the bank beside the river" would induce the unknown relation. thus, relation classification (rc) is the task of using a function f r that estimates f r . lexical relation classification (lrc) is a subtype of rc where the relation between words is a lexical one. the most usual and important lexical relations are hyponymy, hyperonymy, antonymy, synonymy, and meronymy. among these relations, hyponymy and, its counterpart, hyperonymy are especially important in nlp and ontology engineering. finally, lexical entailment (le) is the task of detecting the hyponymy relationship between two words. this task becomes graded le when we have to calculate the numerical degree to which a word w 1 is a type of w 2 , becoming a more challenging regression task. ## experimental setup the main goals of our experiments are: 1) to check if lrc can be conducted without adding artificial context when just a pair of words out of context is given, 2) if so, to analyze which verbalization works best for model fine-tuning, and 3) to check the generalizability of our model to other languagerelated tasks such as graded le. ## results in this section, we report the qualitative and quantitative results of our experiments. ## conclusions and future work our experiments show that minimal prompts work equally well to more complex ones for the lrc task, thus, allowing less human effort and computational cost, and following a language-neutral approach. moreover, we show that minimal prompting outperforms sota results in graded le. we conducted an extensive error analysis showing that: synonymy remains the hardest category to classify, there is some domain and pos bias, and polysemy was proven to be an issue. we highlight the need of crafting more balanced datasets in terms of pos and domain, with finer-graded annotations for the different types of synonyms. as future work, we would like to a) address lrc as a multilabel classification task to alleviate the polysemy challenge, b) check the approach with other languages, c) extend the study to other semantic relations, and d) gain insights in why null prompting improves the sota for lrc and if this line of research could be generalized to other relations, or if not, what characterizes lexico-semantic relations to fit this well the null prompting approach. 1. computational cost: for our experiments, we used almost 850h of gpus. in future research, we could try to lower this cost by experimenting with prompting for lrc task in few-shot scenarios, which would also help when conducting the task for low-researched languages. once again, few-shot tuning would decrease the annotation cost, making it possible to train with, although less, better-annotated examples. additionally, synonymy remains the most difficult relation to capture, a more fine-graded annotation of the different kinds of synonyms could improve their classification.