Rosetta Stone at KSAA - RD Shared Task: A Hop From Language Modeling To Word–Definition Alignment (2023)

Abstract: A Reverse Dictionary is a tool enabling users to discover a word based on its provided definition, meaning, or description. Such a technique proves valuable in various scenarios, aiding language learners who possess a description of a word without its identity, and benefiting writers seeking precise terminology. These scenarios often encapsulate what is referred to as the “Tip-of-the-Tongue” (TOT) phenomena. In this work, we present our winning solution for the Arabic Reverse Dictionary shared task. This task focuses on deriving a vector representation of an Arabic word from its accompanying description. The shared task encompasses two distinct subtasks: the first involves an Arabic definition as input, while the second employs an English definition. For the first subtask, our approach relies on an ensemble of finetuned Arabic BERT-based models, predicting the word embedding for a given definition. The final representation is obtained through averaging the output embeddings from each model within the ensemble. In contrast, the most effective solution for the second subtask involves translating the English test definitions into Arabic and applying them to the finetuned models originally trained for the first subtask. This straightforward method achieves the highest score across both subtasks.

## introduction the tip-of-the-tongue phenomena, as explained by the authors of @xcite , is "a state in which one cannot quite recall a familiar word but can recall words of similar form and meaning". a straightforward way to solve this problem, is to have a reverse dictionary; a system that takes a description provided by the user as an input, and outputs the word @xcite . the initial solutions were heuristic-based. in their work, @xcite suggested a method where the tokens in the user-provided description are compared to all dictionary definitions. the system then returns the word with the highest token match. their method implements different retrieval efficiency tweaks to overcome the issue of excessive time complexity resulting from the comparison operation. recent approaches employ neural-based models, since that are capable of better capturing the semantics of an input description, in contrast to the earlier solutions mentioned, which relied on word overlap. in their work, @xcite suggest utilizing a recurrent-neural-network (rnn) to generate a vector representation based on the provided definition. this representation is then compared against a set of word embeddings to select the closest word to return to the user. the issue of low-frequency words is one of the main challenges of building a reverse dictionary, since these words are the ones that are the less trained and thus have a worse representation compared to more frequent words. @xcite tackle this problem by handcrafting predictors that extract features inspired by the thought process undergone by humans to get a word given its description. polysemy, which is the coexistence of many possible meanings for a word, is another obstacle when building reverse dictionaries. most of the previously mentioned solutions rely on a a set of static word representation builders such as word2vec, which hinders the accuracy of such models. this motivates the use of pretrained language models to produce embeddings that vary based on context. the authors of @xcite probed bert @xcite to predict the word representation, alleviating the issue of polysemy. reverse dictionaries can also be cross-lingual; where one aims to retrieve a word in language x based on a description provided in language y. employing any of the previously mentioned solutions ## datasets the provided data can be categorised into three distinct datasets. the first and third datasets are split into train , dev and test sets by the organizers. the english language dictionary however, isn't provided with such divisions. therefore, we manually split the english dictionary ourselves. ## results ## conclusion in this paper, we present our winning solution to the arabic reverse dictionary shared task. the objective is to derive an arabic word representation based on a provided definition, which can be in either arabic or english. our approach simply leverages several language models pretrained on arabic datasets. through finetuning and ensembling the trained models, our method is capable of capturing the underlying semantics of the input definitions as well as correcting small errors done by single models. for the first subtask, we achieve the best results by fine-tuning four arabic pretrained language models twice, one for predicting the electra embedding and once for the sgns one. this involves minimizing the discrepancy between the predicted embedding and the model's final representation using an mse loss function. in the second subtask, our most effective solution is to repurpose the models initially developed for the first subtask by translating the english test set definitions into arabic.