BL . R esearch at S em E val-2022 Task 1: Deep networks for Reverse Dictionary using embeddings and LSTM autoencoders (2022)

Abstract: This paper describes our two deep learning systems that competed at SemEval-2022 Task 1 “CODWOE: Comparing Dictionaries and WOrd Embeddings”. We participated in the subtask for the reverse dictionary which consists in generating vectors from glosses. We use sequential models that integrate several neural networks, starting from Embeddings networks until the use of Dense networks, Bidirectional Long Short-Term Memory (BiLSTM) networks and LSTM networks. All glosses have been preprocessed in order to consider the best representation form of the meanings for all words that appears. We achieved very competitive results in reverse dictionary with a second position in English and French languages when using contextualized embeddings, and the same position for English, French and Spanish languages when using char embeddings.

## introduction distributed representations of words (or word embeddings) @xcite @xcite have shown to provide useful features for various tasks in natural language processing (nlp) and computer vision. while there seems to be a consensus concerning the usefulness of word embeddings and how to learn them, this is not yet clear with regard to representations that carry the meaning of a full sentence. that is, how to capture 1 https://github.com/jln-brtn/bl.research-atsemeval-2022-task-1 the relationships among multiple words and phrases in a single vector remains a question to be solved. much recent research in computational semantics has focused on learning representations of arbitrary-length phrases and sentences. the reverse dictionary represents one of the most common cases to solve this problem of learning sequence representations. that said, the reverse dictionary is the task to find the proper target word given the word description @xcite @xcite . for example, the composed meaning of the words in a dictionary definition (a mixture of other substances or things) should correspond to the meaning of the word that define it (cocktail). as mentioned by @xcite , this bridge between lexical and phrasal semantics is useful because high quality vector representations of single words can be used as a target when learning to combine the words into a coherent phrasal representation. in this paper, we present our contributions to solve the reverse dictionary problem using very specific neural architectures and applying supervised learning. for more information on task 1 of semeval-2022 as described by its organizers, we invite the reader to consult the paper of @xcite . our approaches require a model able of learning to map between arbitrary-length phrases and fixedlength continuous-valued word vectors. for this purpose, we experiment with two broad classes of neural language models (nlms): recurrent neural networks ( rnns term memory (bilstm and lstm), which naturally encode the order of input words (or characters), and simpler (feedforward) lexical units embedding models. these lexical units can be bagof-words (bow) or a sequence of characters. after having described in section 2 in more detail the problem to be solved in the semeval-2022 evaluation campaign and the data provided, we present in section 3 the previous works of reverse dictionary. then, in section 4, we present our neural architectures with all data preprocessing having been performed. thereafter, in section 5, we describe the experimental setup implemented before presenting the results in section 6 and concluding in section 7. ## related work in the field of natural language processing, word embeddings have been the subject of several research problems for many years. indeed, a text contains various information, and the idea is to resorb the target information in a continuous vector representation. embedding's methods improve significatively the results against standard statistical approaches and justified the interest these last years. in order to create embeddings, the scientific research community is experimenting with two approach types: unsupervised and supervised learning of embeddings. the unsupervised approaches are the most common and consist in using a pretrained language model on a large corpus such as word2vec @xcite or glove @xcite . from the word embeddings we can obtained, we must then choose a technique in order to find the right combination of words that will best convey the desired information. for example, it's possible to apply combinations such as: average, sum, or centroid to obtain a vector that reflect the representation for a sequence of words. we can find an example of centroid usage with @xcite for news summarization extraction or a centroid vector weighted by idf (inverse document frequency) @xcite . for the supervised approaches to produce sequence text vector representation, the idea is consisting of modeling the link between a content and an embedding representation. the reverse dictionary is a common case of sequence or short content representation. some state-of-the-art models are used to perform this task, for example neural networks lstm @xcite and bert (pre-training of deep bidirectional transformers for language understanding) @xcite . notably in the paper of yan, li and qiu (2020), they experiment with word generation from a definition using bert multilingual architecture. as it's mentioned in their paper, the use of a bert model is a great idea and can, at least, achieves state-of-the-art performances for both monolingual and cross-lingual reverse dictionary task. even better, the proposed framework here can perform cross-lingual reverse dictionary task without aligned data. we can also talk about the work of morinaga and yamaguchi (2020), malekzadeh, gheibi and mohades (2021) which are based on a long short-term memory (lstm) architecture. with always our objective to produce a vector of a contextualized text, the lstm offers great prospect in this field of research. indeed, the recurrent neural network architecture of the lstm allow models to perform on sequential data which is exactly our case study in this task. ## system overview in this section, we describe the models we proposed in the codwoe -reverse dictionary shared task. in order to keep comparable and linguistically significant the results submitted by the different participants, the organizers of codwoe disallowed any use of external resources, including standard datasets as well as pretrained models that could be used for this task (such as word2vec models or contextual pretrained models based on transformer's architectures like bert). given this condition, we decided to explore the sequential models, and particularly the lstm and bilstm models. all the following models' architecture we created are based on tensorflow foot_1 and keras foot_2 (chollet and others, 2015) libraries. ## experimental setup in this section, we describe different variants we tested. since there were 3 types of vector representations proposed to us in this shared task, we used the same architectures to produce the 3 types of vectors. however, the data format given as input to the model is not the same for the 3 types. for the 'electra' and 'sgns' representation types, we prepare a vocabulary containing the words of the glosses of the 'training dataset', the words of this vocabulary were obtained by following the preprocessing described in section 4.1. for the 'char' vector type, we construct a vocabulary of all the characters used in the glosses without preprocessing the data. the idea being that, for the 'char' type representation, the model encodes the characters of the glosses into vectors and then produce the vectors encoding the glosses based on the vectors of the characters constituting the glosses. the model that we propose is a monolingual model, i.e., we trained it separately on the training dataset of each language provided. however, in order to evaluate the impact of using a multilingual model, we trained the same neural networks on five languages (with character vector) at the same time and compared the results obtained with those obtained by the monolingual models. for the 'sgns' and 'electra' representation types, we built a vocabulary containing the words of all glosses on the five languages, which contains in total 121,147 words. we did the same with the 'char' vectors but with preparing a vocabulary of characters instead containing 405 characters, in total. the ## results and analysis in this section, we present the performance results on using architectures that we described in section 4 and try to give clue to understand them. our main goal was to outperform the organizers' baseline model and results @xcite . with our second model, the results are completely opposite. we performed in mse and cosine measure. with these two measures, we're as we mentioned earlier, we tried to create a multilingual model. unfortunately, after trained this model on all five languages and test on french character embeddings, the model gave us poor results: 0.67 for mse and 0.48 for cosine measure. these are the worst results we've had in this competition, so we decided to drop this architecture and focus on the models presented in the system overview section. given the set of results obtained, we find that the best cosine score was obtained by using electra (contextualized) vector embeddings and the best mse score was obtained by using character vector embeddings. more generally, the use of bilstm-lstm architecture neural network has been beneficial in having results that surpass baselines when cosine and mse are used as evaluation measures. ## conclusion in this paper, we have presented our contributions to solve the task 1 problem of the semeval-2022 evaluation campaign. we studied the effects of training sentence embeddings with supervised data by testing on five different languages, namely: english, french, spanish, italian and russian language. we showed that models learned with char embeddings or contextualized embeddings can perform better than models learned with skip-gram word embeddings. by exploring various architectures, we showed that the combination of embedding/dense/bilstm/ dense/lstm layers can be beneficial than the simple use of embedding layer. we believe that the neural architecture of our advanced model can be used to solve other tasks such as definition modeling @xcite , where the objective would be to reverse the inputs/outputs of the model, or other natural language processing tasks where the objective is to add a specific output layer to adopt the specific problem like sequence classification, for example.