Sememe Prediction for B abel N et Synsets using Multilingual and Multimodal Information (2022)

Abstract: In linguistics, a sememe is defined as the minimum semantic unit of languages. Sememe knowledge bases (KBs), which are built by manually annotating words with sememes, have been successfully applied to various NLP tasks. However, existing sememe KBs only cover a few languages, which hinders the wide utilization of sememes. To address this issue, the task of sememe prediction for BabelNet synsets (SPBS) is presented, aiming to build a multilingual sememe KB based on BabelNet, a multilingual encyclopedia dictionary. By automatically predicting sememes for a BabelNet synset, the words in many languages in the synset would obtain sememe annotations simultaneously. However, previous SPBS methods have not taken full advantage of the abundant information in BabelNet. In this paper, we utilize the multilingual synonyms, multilingual glosses and images in BabelNet for SPBS. We design a multimodal information fusion model to encode and combine this information for sememe prediction. Experimental results show the substantial outperformance of our model over previous methods (about 10 MAP and F1 scores). All the code and data of this paper can be obtained at https://github.com/thunlp/MSGI .

## introduction a word is the smallest unit of language that can stand on its own (o' @xcite , but its meaning can be further divided into smaller components. in linguistics, a sememe is defined as the minimum semantic unit @xcite . it is believed by some linguists that the meanings of all the words in any language can be decomposed of a limited set of language-independent sememes, * equal contribution † corresponding author. email: sms@tsinghua.edu.cn husband human economize family male spouse word sense sememe "married man" "carefully use" figure 1 : sememe annotations of the english word "husband" in hownet. for succinctness, we only show the english notations of sememes, although sememes have both english and chinese notations in hownet, e.g., family|家庭. hownet is a special lexical knowledge base (kb). different from other lexical kbs like word-net @xcite , which explain meanings of words by relations between words, e.g., hyponym and meronym, hownet provides intensional definitions of words using infra-word sememes. this distinctness gives hownet unique advantages. first, sememes can be easily incorporated into neural networks as semantic labels @xcite , which displays the particular suitability of hownet in knowledge integration into deep learning. second, the nature that limited sememes can represent meanings of unlimited words endows hownet with the ability to handle low-data regimes, e.g., sememes can improve the embeddings of rare words @xcite . because of these advantages, hownet has been successfully utilized in various nlp tasks @xcite . hownet is distinctive and useful, but it covers only two languages (english and chinese). plus there are no sememe kbs like hownet in other languages, which hinders nlp of most languages from benefiting by sememes. manually building a sememe kb for each language is an obvious solution. but it is not realistic at all because the building process would be extremely time-consuming and labor-intensive -it takes several linguistic experts more than two decades to build hownet. to solve this problem, @xcite pioneeringly propose to build a multilingual sememe kb based on babelnet @xcite , a multilingual encyclopedic dictionary. the entries of babelnet are synsets composed of synonyms in almost 500 languages, as illustrated in figure 2 . all the multilingual synonyms in a synset have the same meaning and thus should be annotated with the same sememes. therefore, sememe annotations of words in many languages would be simultaneously obtained by annotating sememes for babelnet synsets. for example, suppose we annotate four sememes human, family, spouse and male to the synset in figure 2 , all the multilingual synonyms in the synset ("husband", "époux", "丈夫", etc.) would be simultaneously annotated with these sememes. 1 further, @xcite build a seed dataset by manually annotating sememes for some synsets, and present the task of sememe prediction for ba-belnet synsets (spbs), which is aimed at automatically predicting sememes for the other unannotated synsets. in addition, they put forward two spbs methods that utilize different information in ba-belnet synsets, namely synset-related wikipedia articles and relations between synsets. in this paper, we argue that some other information contained in babelnet can be exploited for spbs. as shown in figure 2 , in addition to the multilingual synonyms, each babelnet synset comprises multilingual glosses that are extracted from different sources including wordnet and wiktionary. 2 besides, many synsets contain images from wikipedia and wikidata @xcite . the multilingual synonyms, glosses and images of a synset convey the meaning of the synset, thus naturally helpful in predicting 1 if a word is polysemous, it would be included in multiple babelnet synsets and have multiple sets of sememes. we design a multimodal information fusion model named msgi (sememe prediction with multilingual synonyms and glosses as well as images), which comprises a multilingual text encoder, an image encoder and a multi-label classifier. the text encoder is based on a cross-lingual pre-trained language model that encodes the multilingual synonyms and glosses. to adapt the general pretrained language model for the task of spbs, we introduce a new pre-training task named masked contextual sememe prediction to it. the image encoder learns the embeddings of the images, and we adopt the attention-based multi-instance learning mechanism to process multiple images. in experiments, we find that our msgi model substantially outperforms previous spbs methods (by about 10 map and f1 scores). we also conduct a series of quantitative and qualitative analyses of the sememe prediction results of msgi, aiming to explain the effectiveness of msgi. ## methodology in this section, we elaborate on our msgi model. before that, we first introduce the formalization of the spbs task. then we describe the details of msgi, and finally we present the training strategy of msgi. figure 3 illustrates the framework and training strategy of msgi. ## experiments in this section, we evaluate the sememe prediction performance of our msgi model. ## analysis in this section, we conduct some quantitative and qualitative analyses of the spbs results of msgi. all the experiments are conducted on the validation set of babelsememe. ## conclusion and future work in this paper, we propose to utilize the multilingual and multimodal information in babelnet, i.e., multilingual synonyms, multilingual glosses and images, to predict sememes for babelnet synsets. we design the msgi model and it achieves absolute outperformance over previous methods. in the future, we will try to leverage more information in babelnet, e.g., semantic relations, to better predict sememes. we will also consider expanding babelsememe with the prediction results of our model after manual examination.