1 C ademy at S emeval-2022 Task 1: Investigating the Effectiveness of Multilingual, Multitask, and Language-Agnostic Tricks for the Reverse Dictionary Task (2022)

Abstract: This paper describes our system for the Se- mEval2022 task of matching dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track of the competition, which maps multilingual glosses to reconstructed vector representations. More specifically, models convert the input of sentences to three types of embeddings: SGNS, Char, and Electra. We pro- pose several experiments for applying neural network cells, general multilingual and multi-task structures, and language-agnostic tricks to the task. We also provide comparisons over different types of word embeddings and ablation studies to suggest helpful strategies. Our initial transformer-based model achieves relatively low performance. However, trials on different retokenization methodologies indicate improved performance. Our proposed Elmo- based monolingual model achieves the highest outcome, and its multitask, and multilingual varieties show competitive results as well.

## introduction reverse dictionary task is defined as word generation based on user descriptions @xcite . following competition rules, pre-trained models and external information should be avoided, and large-scale language models are unsuitable for the task. our paper is devoted to the performance comparison of different neural network structures, multilingual and multitask tricks, and elaborating on language-agnostic or bidirectional structure helpfulness. the competition @xcite has significant potential in contributing pretraining process acceleration, low-resource language model development, and commonsense using. furthermore, the task is of high importance for explainable ai and natural language processing since it models direct mapping from human-readable data to machine-readable data. * the two authors contributed equally to this work. known word representation methods using dictionaries, knowledge databases, or glosses have been a common approach for years. related models can be divided into two major groups. in the former, category methods highly rely on large-scale model construction. @xcite develop sensebert, introducing super-senses from wordnet @xcite into general bert model. ernie @xcite combines node embeddings from knowledge graph and matched entities to enhance word representations. knowbert @xcite subsumes the entity connection and bert models, which are trained together. there are similar research works relevant to the topic @xcite @xcite . still, their models' performances are dependent on the basic largescale language model trained by sentence samples. in the latter group, traditional dependency-based language models learn directly from word dependency and glosses. they have two major disadvantages: incompatibility with modern language models and relatively low performance @xcite @xcite . there is ambiguity about whether recent embeddings and dictionary glosses are mappable from each other. the paper specifically focuses on progressing utilization of the glosses, different word representations, and languages. first, we discuss ablation studies for language-agnostic trick, bidirectional, multilingual, and multitask models and explain the experimental results. second, we apply and analyze different re-tokenization methods. finally, we give instructive conclusions about encoder structures, distinctive word representation relations, and cross-lingual dictionary performance based on our experiment results. we find that (1) transformerbased model performance is inferior to other models for its high complexity, (2) bidirectional models with similar parameter size outperform the unidirec-tional model because of their better understanding of context-environments even in the low-resource condition, and (3) different word embeddings have a potential relations and can be collaboratively learnt from glosses using a multitask learning structure. we make our codes and results publicly available 1 . ## task description the competition, comparing dictionaries and word embeddings, proposes definition modeling @xcite and reverse dictionary sub-tracks @xcite . these sub-tracks are designed to test the equivalency of dictionary glosses and word embedding representations. this paper focuses on the reverse dictionary direction. the task refers to word recalling using gloss input and provides word representations that are separately generated by word2vec (sgns) @xcite , character @xcite , and electra @xcite embeddings as training data. external data and large-scale language models are strictly restricted from this competition since the models might learn the word embeddings majorly from the sentence samples instead of the dictionary glosses. the words matched with the dictionary glosses are hidden in the datasets, implying that dependencybased word representation algorithms cannot be applied directly. ## methodology to clarify, we affirm that we only refer to the model structures instead of the trained models when we mention elmo and mbert in the section and use no external data. ## conclusion the paper proposes a model showing competitive results in most cases of the reverse dictionaries task. several conclusions are provided about the reverse dictionaries task by the paper based on the ablation studies. first, the transformer-based model, for its high complexity, performs worse compared to rnn-or lstm-based models. multilingual transformer-based model benefits from specifying languages and including language-related grammar positional information. second, bidirectional models with similar parameter sizes outperform the unidirectional one since they better grasp the context in low-resource conditions. third, different word representations are potential connections and can be collaboratively learned from glosses using a multitask learning structure. sgns embedding is much harder to model compared to character embedding and electra embedding.