BERT for Monolingual and Cross-Lingual Reverse Dictionary (2020)

Abstract: Reverse dictionary is the task to find the proper target word given the word description. In this paper, we tried to incorporate BERT into this task. However, since BERT is based on the byte-pair-encoding (BPE) subword encoding, it is nontrivial to make BERT generate a word given the description. We propose a simple but effective method to make BERT generate the target word for this specific task. Besides, the cross-lingual reverse dictionary is the task to find the proper target word described in another language. Previous models have to keep two different word embeddings and learn to align these embeddings. Nevertheless, by using the Multilingual BERT (mBERT), we can efficiently conduct the cross-lingual reverse dictionary with one subword embedding, and the alignment between languages is not necessary. More importantly, mBERT can achieve remarkable cross-lingual reverse dictionary performance even without the parallel corpus, which means it can conduct the cross-lingual reverse dictionary with only corresponding monolingual data. Code is publicly available at https://github.com/yhcc/BertForRD.git .

## introduction reverse dictionary @xcite is the task to find the proper target word given the word description. fig. 1 shows an example of the monolingual and the cross-lingual reverse dictionary. reverse dictionary should be a useful tool to help writers, translators, and new language learners find a proper word when encountering the tip-of-the-tongue problem @xcite . moreover, the reverse dictionary can be used for educational evaluation. for example, teachers can ask the students to describe a a situation where neither can go forward nor backward. word, and the correct description should make the reverse dictionary model recall the word. the core of reverse dictionary is to match a word and its description semantically. early methods @xcite firstly extracted the handcrafted features and then used similaritybased approaches to find the target word. however, since these methods are mainly based on the surface form of words, they cannot extract the semantic meaning, resulting in bad performance when evaluated on the human-written search query. recent methods usually adopt neural networks to encode the description and the candidate words into the same semantic embedding space and return the word which is closest to the description @xcite . although current neural methods can extract the semantic representations of the descriptions and words, they have three challenging issues: (1) the first issue is the data sparsity. it is hard to learn good embeddings for the low-frequent words; in this paper, we leverage the pre-trained masked language model bert @xcite to tackle the above issues. firstly, since bert tokenizes the words into subwords with byte-pairencoding (bpe) @xcite , the common subwords between low-frequent and highfrequent words can alleviate the data sparsity problem. secondly, bert can output contextualized representation for a word. thus the polysemy problem can be much relieved. thirdly, the mbert is suitable to tackle the cross-lingual reverse dictionary. because bert shares some subwords between different languages, there is no need to align different languages explicitly. therefore, we formulate the reverse dictionary task into the masked language model framework and use bert to deal with the reverse dictionary task in monolingual and cross-lingual scenarios. besides, our proposed framework can also tackle the cross-lingual reverse dictionary task without the parallel (aligned) corpus. our contributions can be summarized as follows: ## related work the reverse dictionary task has been investigated in several previous academic studies. @xcite proposed using the information retrieval techniques to solve this task, and they first built a database based on available dictionaries. when a query came in, the system would find the closest definition in the database, then return the corresponding word. different similarity metrics can be used to calculate the distance. @xcite enhanced the retrieval system with wordnet @xcite . @xcite was the first to apply rnn into the reverse dictionary task, making the model free of handcrafted features. after encoding the definition into a dense vector, this vector is used to find its nearest neighbor word. this model formulation has been adopted in several papers @xcite @xcite @xcite , their difference lies in usage of different resources. @xcite ; @xcite used wordnet to form graphs to tackle the reverse dictionary task. the construction of the bilingual reverse dictionary has been studied in @xcite . lam and kalita (2013) relied on the availability of lexical resources, such as wordnet, to build a bilingual reverse dictionary. @xcite built several bilingual reverse dictionaries based on the wiktionary foot_0 , but this kind of online data cannot ensure the data's quality. building a bilingual reverse dictionary is not an easy task, and it will be even harder for low-resource language. other than the low-quality problem, the vast number of language pairs is also a big obstacle, since if there are n languages, they will form n 2 pairs. however, by the unaligned cross-lingual reverse dictionary, we can not only exploit the high-quality monolingual dictionaries, but also avoid the preparation of n 2 language pairs. unsupervised machine translation is highly correlated with the unaligned cross-lingual reverse dictionary @xcite @xcite . however, the unaligned cross-lingual reverse dictionary task differs from the unsupervised machine translation at least in two aspects. firstly, the target for the crosslingual reverse dictionary and machine translation is a word and a sentence, respectively. secondly, theoretically, the translated sentence and the original sentence should contain the same information. nevertheless, in the cross-lingual reverse dictionary task, on the one hand, the target word might contain more senses when it is polysemous. on the other hand, a description can correspond to several similar terms. the polysemy also makes the unsupervised word alignment hard to solve this task @xcite . last but not least, the pre-trained language model bert has been extensively exploited in the natural language processing (nlp) community since its introduction @xcite . owing to bert's ability to extract contextualized information, bert has been successfully utilized to enhance various tasks substantially, such as the aspect-based sentiment analysis task @xcite , summarization @xcite , named entity recognition @xcite and chinese dependency parsing @xcite . however, most works used bert as an encoder, and less work uses bert to do generation @xcite . @xcite showed that bert is a markov random field language model. therefore, sentences can be sampled from bert. @xcite used pre-trained bert to initialize the unsupervised machine training model an achieve good performance. different from these work, although a word might contain several subwords, we use a simple but effective method to make bert generate the word ranking list with only one forward pass. ## methodology the reverse dictionary task is to find the target word w given its definition @xmath0, w 2 , . . . , w n ], where d and w can be in the same language or different languages. in this section, we first introduce bert, then present the method we used to incorporate bert into the reverse dictionary task. ## analysis 6.1 performance for number of senses following @xcite , we evaluate the accuracy of words with a different number of senses through wordnet @xcite . the results are shown in fig. 4 . bert and roberta significantly improve the accuracy of words with single and multiple senses, which means they can alleviate the polysemous issue. ## conclusion in this paper, we formulate the reverse dictionary task under the masked language model framework and use bert to predict the target word. since definition someone who studies secret code systems in order to obtain secret information mul-channel cryptographer cryptologist spymaster snoop bert cryptanalyst codebreaker cryptographer coder roberta codebreaker cryptanalyst cryptographer snooper definition el punto que esta a mitad del camino entre dos extremos. (the point that is halfway between two ends) spanish centro mitad medio punta trans. core middle middle tip unaligned center centre middle mid definition pièce où l ' on prépare et fait cuire les aliments (room where food is prepared and cooked) french cuisine restaurant pièce cuire trans. cookery restaurant room cook unaligned kitchen cook office restaurant bert decomposes words into subwords, the score of the target word is the sum of the scores of its constituent subwords. with the incorporation of bert, our method achieves state-of-the-art performances for both the monolingual and cross-lingual reverse dictionary tasks. besides, we propose a new cross-lingual reverse dictionary task without aligned data. our proposed framework can perform the cross-lingual reverse dictionary while being trained on monolingual corpora only. although the performance of unaligned bert is superior to the translation and word vector alignment method, it still lags behind the supervised aligned reverse dictionary model. therefore, future work should be conducted to enhance performance on the unaligned reverse dictionary.