JSI at S em E val-2022 Task 1: CODWOE - Reverse Dictionary: Monolingual and cross-lingual approaches (2022)

Abstract: The reverse dictionary task is a sequence-to-vector task in which a gloss is provided as input, and the output must be a semantically matching word vector. The reverse dictionary is useful in practical applications such as solving the tip-of-the-tongue problem, helping new language learners, etc. In this paper, we evaluate the effect of a Transformer-based model with cross-lingual zero-shot learning to improve the reverse dictionary performance. Our experiments are conducted in five languages in the CODWOE dataset, including English, French, Italian, Spanish, and Russian. Even if we did not achieve a good ranking in the CODWOE competition, we show that our work partially improves the current baseline from the organizers with a hypothesis on the impact of LSTM in monolingual, multilingual, and zero-shot learning. All the codes are available at https://github.com/honghanhh/codwoe2021 .

## introduction the codwoe 2021 shared task on dictionary glosses and word embedding representations, organized as part of the semeval workshop, presented one of the first opportunities to systematically study and compare these semantic descriptions by two sub-tracks: model definition and reverse dictionary. while definition modeling consists in using the vector representation of e.g. "giraffe" to produce the associated gloss, e.g. "a tall, long-necked, spotted ruminant of africa", the reverse dictionary is the mathematical inverse: reconstruct an embedding for the word "giraffe" from the corresponding gloss. in this paper, we dive into the reverse dictionary task modelling to learn the ability to infer word embeddings from dictionary resources. a reverse dictionary is useful in real-world applications. first of all, it can effectively solve the tip-of-the-tongue problem @xcite : the inability to retrieve a word from memory. people who suffer from this problem such as copywriters, novelists, researchers, students, etc. can quickly and easily find the words they need thanks to reverse dictionary. furthermore, new language learners who grasp a limited number of words can also take advantage of the reverse dictionary to express correctly. besides, it plays an important role in word selection for anomia patients @xcite , who can recognize and describe an object but fail to name it due to neurological disorder. the contributions of this paper are as follows: this paper is organised as follows: section 2 presents the related works in reverse dictionary. next, we introduce our methodology in section 3, and the experimental details in section 4. the results are discussed in section 5, before we conclude and present future works in section 6. ## related work the reverse dictionary systems tend to employ two distinct approaches. the first approach takes advantage of sentence matching @xcite @xcite to return the words whose dictionary definitions are most similar to the corresponding gloss. the second approach focuses on neural language models to encode the glosses into a vector representation and returns the words with the closest embeddings to the vector of the glosses @xcite , 2016; @xcite @xcite . as a result, the performance depends largely on the word representation's quality. however, many words are low-frequency and usually have poor embeddings regarding zipf's law. to tackle the above issue, a multi-channel reverse dictionary model has been proposed @xcite . the system includes a sentence encoder (e.g. a bilstm @xcite , bert @xcite ) with attention @xcite , and diverse characteristic predictors that are useful to find the target words with poor representations and exclude wrong words with similar embeddings to the target words, for example, antonyms. in terms of production, onelook foot_0 and reverse-dictionary foot_1 are two successful commercial english reverse dictionary systems. however, their architectures are undisclosed and their performance is far from perfect. meanwhile, open-sourced wantwords foot_2 @xcite is a rising star with state-of-the-art (sota) performance in english and even competitive results in a cross-lingual chinese-english and english-chinese setting. ## methodology as the competition does not allow the use of external data or pretrained language models in order to make approaches easily comparable, we start by experimenting with the simplest form of transformer, a deep learning model that adopts the selfattention mechanism, differentially weighting the significance of each part of the input data. this is also the baseline shared by codwoe's organizers. then we experiment by adding an additional lstm layer (model 1), bilstm layer (model 2), and combining the prediction from these two mentioned layers (model 3). the overall architecture is presented in figure 1 . the objective of the model is to map the glosses to the vector representation of the word that the gloss defines. the target embeddings are learned by a skip-gram with negative sampling (sgns) approach (word2vec). during training, the input is the gloss, which is tokenized using the byte pair encoding (bpe) algorithm foot_3 and then converted into word embeddings. the positional encoding is applied to each embedding to inject meaningful information about the position of the tokens in the sequence. after that, they are fed into a transformer encoder, which is a stack of four identical encoder blocks. as illustrated in figure 2 , each block includes the following layers in the same order: a multi-head self-attention layer that explores the word correlations followed by a normalization layer (both of them are surrounded by a residual connection), and then a linear layer followed by a second normalization layer (both of them are also surrounded by a residual connection). a dropout layer is then added to avoid overfitting. in the baseline model suggested by the codwoe's organizers, the results from the above architecture are then passed into a linear layer to achieve the final model. we propose three settings regarding three different models constructed from the baseline architecture. we hypothesize that with an additional lstm or bilstm layer, we can improve the modeling of the word-level sequential context, same as in @xcite , and therefore improve the performance of the model. in model 1, we add one additional lstm layer after the linear one. we take advantage of the bilstm layer in model 2 to capture the information bidirectionally. we combine the result from the two mentioned layers by averaging their weights in model 3. in the final step, we fed the lstm or bilstm outputs into a linear layer to obtain the final vector representation. during the prediction phase, for each new data example, we feed the gloss into the trained model to obtain the vector presentation similar to the sgns. the proposed three models are first tested in a monolingual setting, to determine which architecture achieves the best performance. next, we explore if the target sgns embedding spaces may already be aligned to some degree across languages, even though the codwoe organizers did not explicitly mention any cross-lingual alignment in the shared task description. we first attempt a multilingual experiment to examine the degree to which training in multiple languages affects performance. finally, the best performing monolingual models are tested in a zero-shot cross-lingual setting, where we train the model in a specific language and evaluate it in different languages that the model has never seen before. the implementation details are in section 4.2. ## results the test set results of our approach on the reverse dictionary task are presented in in terms of cosine similarity, the model with an additional lstm layer proves to have better performance in english, spanish, and french compared 7 https://competitions.codalab.org/ competitions/34022#results to other tested models. this model also surpasses the baseline model on spanish and french test sets. in addition, the multilingual model also achieves a slightly better cosine similarity than the baseline on the spanish test set. in terms of cosine ranking, all models demonstrate a slightly higher ranking in comparison to the baseline on the spanish test set, with the multilingual model achieving the best ranking. in other languages, the baseline model performs the best. overall, training the additional lstm layer on a multilingual training set does not seem to improve the results compared to the monolingual settings, the only exception being the performance of the multilingual model on the spanish test set in terms of cosine ranking. given the fact that the transformer-based model with an additional lstm performs the best in a monolingual setting, we use this model for the zero-shot cross-lingual experiments. the results for these experiments are displayed in in general, if the model is trained on a language matching the language of the test data, it performs better except in the french corpus. however, the interesting exception is that, for example, the spanish test set, on which all models, no matter on which language they were trained, offer very consistent performance according to all measures. it is also interesting that the models trained in english and spanish have exactly the same results on french, italian, and russian test sets. this might suggest that these models were not able to make sense of the examples in the test set and that their performance is on par with a random baseline. further analysis of this behavior will be left for the future. ## conclusion in this paper, we have investigated the performance of monolingual and multilingual transformerbased models on the reverse dictionary problem, a sequence-to-vector task where a word representation needs to be constructed from the corresponding gloss. we have experimented with two additions to the original architecture, namely adding either an additional lstm or a bilstm layer on top of the original architecture. we have also ex-plored whether combining these two architectures improves the performance. besides that, we explored the cross-lingual performance of the monolingual models and compared them to monolingual and multilingual classifiers. on the task of reconstructing sgns embeddings, the monolingual transformer-based model with an additional lstm layer in most cases offers the best performance for english, spanish, and french according to mse and cosine similarity. the model also offers competitive performance in terms of mse for italian and russian compared to the baseline. therefore, the results to some extent confirm the initial hypothesis that with an additional lstm layer, we can improve the modeling of the wordlevel sequential context. nevertheless, the improvements are worse than expected and the multilingual and zero-shot experiments yield unexpected results that require further analysis. we can therefore summarize our findings by saying that the reverse dictionary task of restoring sgns embeddings seems to be very challenging, and none of our models (and also other models in the competition) were able to successfully solve it, at least according to the scores achieved during the competition. this means that there remains a lot of room for improvement. in the future, we would like to investigate the effect of different text representations on the performance of the model, e.g., by feeding the model graph representations. combinations of several text representations will also be explored. furthermore, the effectiveness of multilingual models compared to monolingual ones should be additionally explored. despite zero-shot learning not working well in our studies, it is worth evaluating the performance of one-shot learning and few-shot learning with the hypothesis that the models can understand new concepts from only one or a few examples. further experiments on the topic of adapting the transformer architecture for the specific task at hand will also be conducted.