U ppsala U niversity at S em E val-2022 Task 1: Can Foreign Entries Enhance an E nglish Reverse Dictionary? (2022)

Abstract: We present the Uppsala University system for SemEval-2022 Task 1: Comparing Dictionaries and Word Embeddings (CODWOE). We explore the performance of multilingual reverse dictionaries as well as the possibility of utilizing annotated data in other languages to improve the quality of a reverse dictionary in the target language. We mainly focus on character-based embeddings.In our main experiment, we train multilingual models by combining the training data from multiple languages. In an additional experiment, using resources beyond the shared task, we use the training data in Russian and French to improve the English reverse dictionary using unsupervised embeddings alignment and machine translation. The results show that multilingual models occasionally but not consistently can outperform the monolingual baselines. In addition, we demonstrate an improvement of an English reverse dictionary using translated entries from the Russian training data set.

## introduction in a reverse dictionary, one can look up a gloss, an explanation of a word's meaning, to find the most relevant word or word form. the applications of reverse dictionaries are numerous, as they can help language learners in expanding their vocabulary, authors and writers in looking for the most suitable word, and avid cruciverbalists in taking on some of the most challenging crosswords. reverse dictionary modelling has seen approaches ranging from traditional information retrieval using relevance scores @xcite to ones involving node-graph architectures @xcite . as a general rule, the quality of a reverse dictionary appears to largely depend on the availability of annotated data. however, annotated data are scarcely available and expensive to produce for low-resource languages. we therefore explore the viability of multilingual approaches to improve the quality of a reverse dictionary. this work is performed in the context of the reversed dictionary subtask of the semeval 2022 task 1, comparing dictionaries and word embeddings @xcite . unlike standard reverse dictionaries, the target is to predict a word embedding vector for each gloss, rather than a word form. three types of word embeddings are available: character-based embeddings (char), skipgrams (sgns), and contextual embeddings (electra). no additional resources are allowed in the shared task. in this paper, we do present additional experiments, though, where we also used an external machine translation engine. while five languages were made available in the shared task, we mainly focus on english, but also give some results for russian and french. the main research question of this study is thus whether the performance of a monolingual reverse dictionary can be improved using data in other language(s) in a low supervision setup. we first explore what are the most suitable type of embeddings for a transformer-based reverse dictionary. having found the best-performing embeddings, we use them to train a joint model for multilingual reverse dictionary, which can map glosses to words in multiple languages. finally, we use the training data in french and russian to improve the quality of an english reverse dictionary by means of unsupervised embeddings alignment and machine translation. we did not submit our results in the evaluation period since in one of the experiments we used a pre-trained neural machine translation model, which is prohibited in the shared task. nevertheless, we report the performance of our jointly trained multilingual models on the test sets, as no additional data or pre-trained models were involved in training. for character-level embeddings, our best multilingual models, when tested on english, would rank 25th in terms of mean squared error (mse), 20th in terms of cosine similarity (cos), and 9th in terms of cosine-based ranking (crk); on french: 22nd (mse), 10th (cos), 3rd (crk); on russian: 7th (mse), 7th (cos), 13th (crk). ## related work recent research has explored bilingual and crosslingual reverse dictionaries, the task of which is to map a gloss in a source language to a word in target language. an implementation by @xcite involved a machine translation api and bilingual dictionaries to re-direct a query in the source language through the target language pipeline. yan et al. (2020) implemented the first cross-lingual reverse dictionary based on mbert @xcite , a transformer-based language model trained on wikipedia articles in 104 languages. their study revealed that unaligned cross-lingual reverse dictionary achieves best performance when mbert is tuned on unaligned multilingual data; its quality is substantially worse than that of a monolingual model. @xcite thus concluded that it remains unclear how multilingual data is to be utilized to improve the quality of unaligned reverse dictionary, which is to be explored in this project. joint multilingual models, which are trained on multiple languages at once, offer a solution for lowresource languages that often have little to none annotated data. this has for example been explored for dependency parsing, with positive results @xcite . cross-lingual embeddings are of central importance in word meaning similarity across languages @xcite , and are thus a crucial component of cross-lingual reverse dictionaries. as noted by @xcite , the applicability of cross-lingual embeddings relies on their quality, which, in turn, depends on the availability of bilingual corpora and dictionaries. nevertheless, an unsupervised cross-lingual embeddings alignment method proposed by @xcite enables high quality cross-lingual embeddings with no or little supervision, further allowing for unsupervised machine translation. unsupervised cross-lingual embeddings alignment thus offers a solution for both mapping the word embeddings and its glosses from one language to another. ## system description we focus on the strategies of utilizing the data in foreign languages to improve reverse dictionary rather than the choosing of most suitable model. therefore, we use the semeval 2022 task 1 baseline system, a transformer-based architecture with all parameters unchanged for all of our models. ## conclusions this project has investigated whether an english reverse dictionary can be improved using data in foreign languages. this research question was addressed by firstly determining the most suitable type of embeddings for a transformer-based reverse dictionary. secondly, multilingual joint models were trained to see the affects on the performance of english as target language and two source languages, namely french and russian. lastly, the embeddings from source language were aligned to the target embedding space, followed by machine translation of the respective glosses. three key findings emerged. firstly, characterlevel features lead to best performance of an english transformer-based reverse dictionary. secondly, multilingual reverse dictionaries perform comparably with monolingual ones, as no substantial improvement or decline was observed. thirdly, an english reverse dictionary can be improved using the available data in foreign languages, such as french and russian, though the improvement is rather small. in the reported experimental setup, russian was found to be a more suitable source language in enhancing an english reverse dictionary. there are numerous possible extensions of the present study. one could, for instance, recreate the study in a fully supervised or fully unsupervised setup so as to see to what extent the lack of supervision affected the results. it would also be interesting to investigate whether combinations of embeddings, e.g. contextual and character-level, would lead to better performance of reverse dictionary models. overall, the improvements recorded in this study were, arguably, hardly significant. it may therefore be productive to search for more successful ways of using data in foreign languages in creating or improving reverse dictionaries.