L ing J ing at S em E val-2022 Task 1: Multi-task Self-supervised Pre-training for Multilingual Reverse Dictionary (2022)

Abstract: This paper introduces the approach of Team LingJing’s experiments on SemEval-2022 Task 1 Comparing Dictionaries and Word Embeddings (CODWOE). This task aims at comparing two types of semantic descriptions and including two sub-tasks: the definition modeling and reverse dictionary track. Our team focuses on the reverse dictionary track and adopts the multi-task self-supervised pre-training for multilingual reverse dictionaries. Specifically, the randomly initialized mDeBERTa-base model is used to perform multi-task pre-training on the multilingual training datasets. The pre-training step is divided into two stages, namely the MLM pre-training stage and the contrastive pre-training stage. The experimental results show that the proposed method has achieved good performance in the reverse dictionary track, where we rank the 1-st in the Sgns targets of the EN and RU languages. All the experimental codes are open-sourced at https://github.com/WENGSYX/Semeval .

## introduction the codwoe shared task invites the participants to compare two types of semantic descriptions: dictionary glosses and word embedding representations. the intuitions come from the questions: "are these two types of representation equivalent? can we generate one from the other?". to study this question, the codwoe proposes two sub-tracks: a definition modeling track @xcite , where participants have to generate glosses from vectors, and a reverse dictionary track @xcite , where participants have to generate vectors from glosses. these two tracks are fairly challenging @xcite , where more efficient methods are required to be designed for implementation. these tasks are also useful for explainable ai, since they involve converting human-readable data into machine-readable data and back @xcite . in this paper, we focus on the reverse dictionary track. as shown in figure 1 , given the gloss "a meal consisting of food normally eaten in the morning, which may typically include eggs, sausages, toast, bacon, etc.", the reverse dictionary task requires us to generate corresponding three sets of 256-dimensional word vectors. the sgns @xcite , char @xcite , and electra @xcite are skip-gram with negative sampling embeddings, character-based embeddings, and transformer-based contextualized embeddings, respectively. it is noted that this task comprises datasets in 5 languages: english, spanish, italian, french, and russian. the reverse dictionary task is difficult due to the significant inborn differences between word vectors and glosses and the vast differences between languages @xcite . to solve the above problems, we use a multi-task self-supervised pre-training approach with masked language modeling (mlm) @xcite and contrastive learning @xcite . on the one hand, mlm can better capture the semantic representation of the input text @xcite . on the other hand, contrast learning can further improve the performance of downstream regression tasks @xcite . specifically, we use a randomly initialized mdeberta-base @xcite model to perform mlm pre-training on five text datasets in different languages. contrastive pre-training @xcite is then performed using vectors with and without dropout @xcite . afterward, the model is fine-tuned using the reverse dictionary dataset. the experimental results show that the proposed method has achieved good performance in the reverse dictionary track. we achieve the top three results on the sgns evaluation metrics in all languages. specifically, we get first place in english and russian, second place in spanish and french, and third place in italian. ## main method in this section, we will elaborate on the main methods for the reverse dictionary track. as the pre-training method can enhance the performance of semantic representation @xcite , we adopt masked language modeling (mlm) task @xcite and contrastive pre-training task @xcite for implementing this regression task. ## results and discussions in this section, we introduce the experimental results of the sgns, the char and the electra tracks. the online results and further discussions are also presented. ## conclusion in this paper, it is mainly introduced that in order to solve the reverse dictionary track in semeval-22 codwoe, the lingjing team makes the model have the ability of semantic understanding through the mlm task with contrastive learning in the randomly initialized mdeberta model. after that, we report the performance of our model in cod-woe, and obtain the best performance in english and russian tasks of sgns dataset, which proves that our method is effective. in the future, we will further study how to make full use of the characteristics of different languages and make the model embed the text into a more accurate vector space.