IRB - NLP at S em E val-2022 Task 1: Exploring the Relationship Between Words and Their Semantic Representations (2022)

Abstract: What is the relation between a word and its description, or a word and its embedding? Both descriptions and embeddings are semantic representations of words. But, what information from the original word remains in these representations? Or more importantly, which information about a word do these two representations share? Definition Modeling and Reverse Dictionary are two opposite learning tasks that address these questions. The goal of the Definition Modeling task is to investigate the power of information laying inside a word embedding to express the meaning of the word in a humanly understandable way – as a dictionary definition. Conversely, the Reverse Dictionary task explores the ability to predict word embeddings directly from its definition. In this paper, by tackling these two tasks, we are exploring the relationship between words and their semantic representations. We present our findings based on the descriptive, exploratory, and predictive data analysis conducted on the CODWOE dataset. We give a detailed overview of the systems that we designed for Definition Modeling and Reverse Dictionary tasks, and that achieved top scores on SemEval-2022 CODWOE challenge in several subtasks. We hope that our experimental results concerning the predictive models and the data analyses we provide will prove useful in future explorations of word representations and their relationships.

## introduction the comparing dictionaries and word embeddings (codwoe) task @xcite is aimed at explaining two different types of semantic descriptions of words: dictionary glosses and word embeddings. a dictionary gloss is a brief textual explanation of a word and a word embedding is a vector representation that captures the word's semantic and syntactic properties @xcite ). ## system overview both the defmod and the revdict models rely on unigram subword tokenizers @xcite trained on glosses from the train datasets. ## model selection and experimental setup in this section we describe the technical details of data preprocessing and model selection that comprise our methods of constructing the defmod and revdict models. the conceptual description of the methods is given in section 3. ## results definition modeling on the defmod task, the models were evaluated using three metrics: bleu score @xcite , lemma-level bleu score, and moverscore @xcite . while the bleu score is based on matching token n-grams between the reference and the modelproduced text, moverscore calculates a measure of distance between texts embedded in a semantic space, i.e., between two sets of contextual word embeddings computed using a transformer model. results in our approach shows inter-language variation, both in relative (ranks) and absolute (score values) terms. the full results provided by the organizers 4 show that this is also true for other teams -for example, few of the high-performing models perform markedly better for italian and russian than for other languages. however, some approaches yield more stable results across all languages. all of the models yielded by the codwoe shared task perform weakly in terms of bleu. namely, the bleu scores of the existing defmod approaches commonly achieve bleu scores in the range of 20 to 30 @xcite , with some settings yielding bleu as high as 60 @xcite . the experiments with the weakest reported bleu scores @xcite reports bleu scores of approx. 12, while the best codwoe scores are below bleu 10. codwoe defmod models perform better in terms of moverscore, a metric designed for machine summarization @xcite ). an analysis of a number of summarization systems showed that moverscore values range between 15 and 24, with an absolute minimum of 10 and an average slightly below 20 @xcite . in comparison, top codwoe systems reach scores between 12 and 15, except in the case of french, which puts them on the lower end of the summarization scale. we hypothesize that the main reason for the described weak performance is comparatively small amount of codwoe training data (for each indi- as for the representativeness of the test data, the visual analysis performed in a.1 shows that the distribution of test gloss embeddings matches the train distribution well. another factor that potentially influences performance is word rarity. we observed that the english test examples contain a significant amount of rare words (such as "pelta", "akimbo", "gothy", or "dungarees"), while some defmod experiments explicitly focus on the most frequent words @xcite . the greatest performance gains for the models we used come from using all three vector embeddings to form a context vector. this suggests that future approaches can benefit from leveraging several distinct embeddings types as input for gloss generation. we believe that the question of the influence of various factors on the performance of defmod systems is important and under-explored. these factors include model structure and parameters, performance metric, dataset size (both for training and pre-training), and the semantic relation between training and test data. closely related is the question of the nature of semantic generalization that defmod systems are capable of -what kind of examples (and relations contained within them) can inform a successful inference of glosses for unseen embeddings. further performance-related analyses can be found in appendix a.2. appendix a.3 contains a qualitative analysis of glosses that shows that generated glosses can capture varying levels of semantic properties of the correct glosses. we hypothesize that these variations in similarity are hard to capture with metrics such as moverscore and bleu. reverse dictionary we used the following metrics for internal validation of our revdict solutions (described in section 4.2): mean squared error (mse), cosine similarity (cos), and central kernel alignment (cka) @xcite . cos measure has noted drawbacks @xcite . therefore, we use the linear cka similarity measure to gain another perspective on model performance. validation scores can be found in appendix b.2, test predictions were scored by the following metrics: mse, cos, and cosine-based ranking (rnk). the rnk measure is defined as the proportion of test samples with cosine similarity to the model output embedding higher than the ground truth embedding. the final results of our solutions can be found in compared to other solutions, our systems have average or below-average performance in terms of mse and cos test scores. however, they perform significantly better than the other approaches in terms of rnk test scores, from which we conclude that our solutions are better suited for the retrieval task. this is an interesting situation which we elaborate with the following example, shown in figure 1 . it depicts two different predictions, v 1 and v 2 , the first with better mse and cos scores, and the second with a better rnk score. the second solution prefers a vector subspace with a lower density of test samples even if the absolute distance from the correct vector is greater. with a smaller set of possible surrounding solutions, retrieving the vector v gt from the vector v 2 is more precise than retrieving it from the vector v 1 . ## conclusion definition modeling and reverse dictionary are two opposite learning tasks for exploring the relationship between different semantic representations of words. codwoe semeval task @xcite is designed to investigate these tasks on five different languages using three different types of word embeddings. we propose an adaptation of an existing defmod model and analyze its performance and the glosses generated by the model. we believe that defmod is a theoretically interesting problem and that further investigations should focus on discovering which types of semantic generalization the models are able to perform, and how this generalization ability is influenced by both the data and the models' structure. the existing defmod experiments are largely incomparable since they are based on different data and setups. we believe that a contribution of the codwoe task is the creation of a multilingual evaluation setting, as well as the use of the flexible moverscore as an evaluation metric. our revdict systems are based on deep regression models based on transformer architecture that achieved top scores for the difficult-to-predict sgns (word2vec) embeddings. in most cases our revdict solutions perform significantly better then the other systems in terms of the rnk score. these results imply that our solutions could be the appropriate approach for retrieving the right word from its description, a problem crucial for solving the tot problem @xcite in machine-assisted text synthesis. in summary, the models that we produced for the codwoe task perform competitively when compared to other participants' models, and can therefore serve as a reasonable starting point for future tackling of defmod and revdict problems. we believe that the promising directions for future optimizations include the construction of multilingual and multi-task models, as well as investigations of the influence of the external data, primarily in the form of huge pre-training corpora.