E dinburgh at S em E val-2022 Task 1: Jointly Fishing for Word Embeddings and Definitions (2022)

Abstract: This paper presents a winning submission to the SemEval 2022 Task 1 on two sub-tasks: reverse dictionary and definition modelling. We leverage a recently proposed unified model with multi-task training. It utilizes data symmetrically and learns to tackle both tracks concurrently. Analysis shows that our system performs consistently on diverse languages, and works the best with sgns embeddings. Yet, char and electra carry intriguing properties. The two tracks’ best results are always in differing subsets grouped by linguistic annotations. In this task, the quality of definition generation lags behind, and BLEU scores might be misleading.

## introduction we describe the university of edinburgh's participation in semeval 2022 task 1 on comparing dictionaries and word embeddings (codwoe), organized by @xcite . 1 the task features two directions: reverse dictionary and definition modelling. the former is to construct the embedding of a word given its definition gloss, and the latter is to generate the definition from a word embedding. the organizers provide datasets of word embedding-definition pairs across three types of embeddings and five languages. the training data has a size of 43.6k for each language, which is smaller than the data released in prior research @xcite . however, it provides a precious chance for a comprehensive study of lower-resourced reverse dictionary and definition modelling on languages other than english, as well as on different embedding architectures. as our system architecture, we use a recently proposed unified model, which deals with both tracks concurrently and achieves superior results @xcite . the model enables multitask training by using word embeddings and definitions symmetrically. we also create ensembles 1 https://competitions.codalab.org/competitions/34022 and handcrafted phrases. our code implementation builds on the organizers' and is publicly available. 2 we submit to both reverse dictionary and definition modelling tracks, and cover all language and embedding combinations. furthermore, we examine model generations and scores from three aspects: embedding architectures, languages, and linguistic annotations, aiming to figure out how these affect performance, subject to the models we have adopted. we finally show the information captured by different word embeddings and discuss the limitations in task evaluation and ranking. regarding the shared task outcome, we are the team with the most "gold medals": out of 18 subtracks, we attain first place in 8, second place in 4 and third place in 4. our final ranks in the subtracks are detailed in ## performances across embeddings reverse dictionary mse and cosine are incomparable across different embedding types, whereas ranking scores can tell which embedding architecture is preferred for indexing and retrieving a word. a random baseline ranking score is 0.5, and most char and electra figures, unfortunately, fall between 0.4 and 0.5. on the other hand, sgns is more useful as its baseline scores start at around 0.45, and our models can improve these up to 0.25. we employ principal component analysis (pca) to reduce the gold and output embeddings to 2 dimensions. then in figure 2 we visualize en, fr, and ru, which come with all embeddings. the unified model usually outputs to a larger space than the baseline, hinting at a positive correlation between output spread and performance. gold electra has the most isotropic space, but neither model could imitate the distribution. char has a crescent shape with several clusters inside, which is unlikely to be cosine-friendly. these problems are alleviated on sgns, which witnesses the best ranking scores. definition modelling sgns is again the winner, as models trained with it reach the top in many metrics. char is also favourable. this is counterintuitive as electra should be fitter, for it retains more sense-specific knowledge. a possible reason is that electra needs to go through more training data than sgns and char to reach perfection. ## performances across languages as seen in the result and rank tables, our system's behaviour is relatively consistent on various languages, except that english is more challenging. assuming that the datasets are of similar quality, it is questionable to conclude that our model suits other languages more than english. moreover, figure 2 confirms that the english embeddings are not more peculiar than those of other languages. we guess that other teams have focused on english (e.g. only submitted english), as it is a centred language in the research community. instead, our hyperparameter search is based on the average loss from all languages, neglecting that the losses are not directly comparable. ## performances across linguistic features we look into the unified model's trial set predictions, to interpret how scores vary across diverse linguistic annotations: polysemy, part of speech (pos), word length in characters, definition length in words, and word frequency. for categorical features, we group data by annotations; for numerical features, we divide the data into three subsets, by percentile ranges: 0-33, 33-67, and 67-100. statistics of the subsets are in word length we then make three partitions according to different word length ranges. results in after pca retains the most distinguishing components, figure 2 shows interesting patterns, especially for char. we randomly label 25 english words and present them in figure 3 and figure 4 , respectively for char and electra. the sub-clusters in char's crescent are perfectly in tune with word lengths; for electra, more frequent words are closer to the origin. we do not notice a clear trend for sgns, for which a plot is attached as figure 5 . we attribute the distinct patterns to the training paradigms: character-level word autoencoding for char, and contextualized modelling for electra. this accounts for the largest cosine gap on char between long and short words, seen earlier in within char embeddings, words are grouped by lengths, so we may utilize this for word retrieval in future work. nonetheless, we are unsure of how length or frequency information aids sense-based tasks, like definition generation in our context. cocktail schooner stalwart pattern flare coal dentistry mafia impregnation landslide hear finger passing noise caravan runny different secular abdominal deplorable pizzicato buckler speckless dread hoover figure 4: gold english electra embeddings with word labels. cocktail schooner stalwart pattern flare coal dentistry mafia impregnation landslide hear finger passing noise caravan runny different secular abdominal deplorable pizzicato buckler speckless dread hoover