BLCU - ICALL at S em E val-2022 Task 1: Cross-Attention Multitasking Framework for Definition Modeling (2022)

Abstract: This paper describes the BLCU-ICALL system used in the SemEval-2022 Task 1 Comparing Dictionaries and Word Embeddings, the Definition Modeling subtrack, achieving 1st on Italian, 2nd on Spanish and Russian, and 3rd on English and French. We propose a transformer-based multitasking framework to explore the task. The framework integrates multiple embedding architectures through the cross-attention mechanism, and captures the structure of glosses through a masking language model objective. Additionally, we also investigate a simple but effective model ensembling strategy to further improve the robustness. The evaluation results show the effectiveness of our solution. We release our code at: https://github.com/blcuicall/SemEval2022-Task1-DM .

## introduction word embeddings @xcite @xcite are dense and low dimensional vectors used in many nlp tasks because they are found to be useful representations of words and often lead to better performance in various tasks. in recent years, large pretrained language models (plms), such as bert @xcite and gpt @xcite families of models, have taken the nlp field by storm, achieving state-of-the-art performance on many tasks @xcite . the contextual embeddings generated by plms are proven to capture syntax and semantic features of words @xcite . but for human beings, word embeddings containing these information is still a black box and unexplainable. there have been many efforts devoted to evaluating the word embeddings' lexical information, such as the word similarity @xcite and analogical relation figure 1 : architecture of the cross-attention multitasking framework. @xcite tasks. however, these tasks can only serve as indirect evaluation methods. in light of this, @xcite proposed the task of definition modeling to evaluate whether a word embedding can be employed to generate a dictionary gloss. since the gloss is a direct and explicit statement of word meaning, this task provides a more transparent view. the semeval-2022 task 1 comparing dictionaries and word embeddings @xcite aims at comparing the two types of semantic descriptions: dictionary glosses and word embeddings. the subtrack 1 is a definition modeling task, which requires models to generate glosses from word embeddings. the task provides data from 5 languages (english, spanish, french, italian, russian) as well as static, character, and contextual embeddings. our team propose a transformer-based @xcite cross-attention multitasking framework to explore the task and apply the framework to all 5 languages. we integrate the multiple embedding architectures through a cross-attention mechanism, which allows the model to query all the embeddings at each time step during generation. to better capture the structure of glosses, we employ an additional masking language model (mlm) @xcite into the framework. we also investigate the ensemble strategies to further enhance the robustness. therefore, the contributions of our system lie in: ## background the definition modeling subtrack provides participants with a multilingual dataset in the form of {e, g}, where e is a set including sgns @xcite , character @xcite , and electra @xcite embeddings, and g is a dictionary gloss. this task takes e as the input, and requires models to generate g. note that all the embeddings have 256 dimensions, and the electra embeddings are only available for 3 of the 5 languages. more detailed statistics of the dataset are listed in many previous work used additional data to improve the performance of generation, such as example sentences @xcite @xcite and semantic features @xcite . some studies also investigated how to employ plms for this task @xcite @xcite . differently, to keep the results linguistically significant and easily comparable, the semeval-2022 task 1 prohibits the usage of external data and plms. therefore, our system focuses on effectively integrating all given embeddings and modeling the glosses. ## system overview figure 1 illustrates the entire architecture of our system, which is a cross-attention multitasking framework based on transformer. the framework consists of two objectives, namely the generation and reconstruction objectives. this section introduces the system in detail. ## results and analysis in this section, we present the evaluation results and discuss our analysis of the generated definitions. ## conclusion in this paper, we present the implementation of the blcu-icall system submitted to the semeval-2022 task 1, definition modeling subtrack. we propose a cross-attention multitasking framework that leverages multiple embedding architectures and jointly trains two objectives. we also investigate a simple but effective ensembling strategy to enhance the robustness. in future efforts, we plan to further improve our system to better handle the problems of redundancy and incorrect glosses.