Learning and Evaluating C hinese Idiom Embeddings (2021)

Abstract: We study the task of learning and evaluating Chinese idiom embeddings. We first construct a new evaluation dataset that contains idiom synonyms and antonyms. Observing that existing Chinese word embedding methods may not be suitable for learning idiom embeddings, we further present a BERT-based method that directly learns embedding vectors for individual idioms. We empirically compare representative existing methods and our method. We find that our method substantially outperforms existing methods on the evaluation dataset we have constructed.

## introduction chengyu (成语, literally meaning "set phrases") are a type of idiomatic expressions in chinese that usually consist of four chinese characters. they are mostly derived from ancient chinese literature and many of them are based on historical stories. the semantic meanings of chengyu are often non-compositional and sometimes metaphoric. for example, the chengyu 瓜田李下 literally means "melon field, beneath the plums," but its idiomatic meaning is to warn people to avoid situations where a person may be easily suspected of wrongdoing. chengyu are commonly used in modern chinese language, and using computational methods to understand chengyu plays an important role in chinese language understanding. for example, a recent work studied how to improve essay writing with recommending chinese idioms @xcite , and others studied how to improve reading comprehension by correcting usage of chinese idioms @xcite and differentiating synonyms of chinese idioms @xcite . in this paper, we refer to chengyu as chinese idioms, although there are also other types of idioms in chinese. recent years have witnessed the success of deep neural networks for many nlp tasks. a central idea behind deep neural networks for nlp is to use dense embedding vectors to represent language units including words, phrases and sentences, and such embeddings have been shown to be useful for many tasks such as sentiment analysis @xcite , question answering @xcite and machine translation @xcite . we therefore believe that it is also desirable to derive embedding vectors for chinese idioms that can accurately capture their semantic meanings. however, it is not clear whether existing methods for chinese word embeddings are effective in deriving good chinese idiom embeddings, and there are at least two reasons for this. first, existing chinese word embedding evaluation datasets do not have sufficient coverage of idioms. for example, in the commonly used wordsim-240 @xcite and wordsim-296 @xcite datasets for chinese word relatedness, no idiom is found. more recently, @xcite released a cos960 dataset with similarities of multiword expressions (mwes). although cos960 covers 150 chinese idioms, this is still a relatively small number, and only 20 mwe pairs in cos960 consist of both idioms. for the word analogy task, another commonly used evaluation task, @xcite created the first chinese dataset with 1,125 analogies, but no idiom is included. @xcite released a large and balanced dataset ca8 for word analogy. although ca8 has 400 entries that contain idioms, they only cover 32 unique idioms and no idiom pairs are included. with this lack of coverage of idioms in existing evaluation datasets, we cannot judge whether existing chinese word embedding methods work well for chinese idioms. second, it is reasonable to suspect that existing word embedding methods for chinese have limitations that make them less suitable for chinese idioms. for non-contextualized word embedding methods such as continuous-bag-of-words (cbow) and skip-gram with negative sampling (sgns), they treat contexts as bags of words, but given the complex meanings of chinese idioms, learning their embeddings from bagof-word representations of contextual words without considering the order and interactions between these contextual words may not be sufficient. existing pre-trained non-contextualized chinese word embeddings are also usually trained with a relatively small context window, but the semantic meaning of a chinese idiom is often based on a larger context where the idiom appears. in fact, it has been observed that larger context windows result in more topicality @xcite , and we suspect that for learning chinese idiom embeddings a larger context window helps. therefore, existing pre-trained noncontextualized chinese word embeddings may not capture the semantic meanings of chinese idioms well. on the other hand, recent contextualized word embedding methods such as bert @xcite and its variants (e.g., ernie @xcite ) consider longer contexts and use attention mechanism to model interactions between words, but since they do not focus on learning word embeddings, they do not learn a single embedding vector for each chinese idiom. although we can aggregate the character-level representations of the characters inside an idiom and treat the aggregated representation as the idiom embedding, since many chinese idioms' semantics are non-compositional, this simplified approach is likely not ideal. in this paper, we study the problem of learning and evaluating chinese idiom embeddings. to overcome the first challenge stated above, i.e., the lack of suitable evaluation dataset for chinese idiom embeddings, we construct an evaluation dataset that contains chinese idiom synonyms and antonyms. we also define two evaluation metrics to measure how close the ground truth idiom synonyms are in an embedding space in order to quantify the quality of the embedding space. to overcome the second challenge stated above, i.e., the potential limitations of existing word embedding methods for chinese idioms, we propose to adapt a method @xcite for chinese idiom recommendation to learn idiom embeddings. this method learns a single embedding vector directly for each idiom and encodes the contextual information using bert. with the evaluation dataset we have created, we empirically compare a sgns-based noncontextualized word embedding method for chinese, two variants of bert for chinese, and our chinese idiom embedding method. we find that based on the two metrics we have defined to measure closeness of synonyms in an embedding space, our method performs substantially better than existing methods. we also find that our method can better distinguish idiom antonyms from idiom synonyms than existing embedding methods. we also conduct further analysis to demonstrate that embedding methods that rely more on chinese character information show advantages only when the synonyms share many common characters. the contributions of our work are twofold: (1) we construct an evaluation dataset to facilitate the evaluation of chinese idiom embeddings. code and data are released on github foot_0 . (2) we present a bert-based method that directly learns chinese idiom embeddings, and we empirically compare this method with existing chinese word embedding methods to demonstrate both the importance of learning a single embedding vector for an entire idiom and the importance of using bert to encode the context when learning these idiom embeddings. ## related work word embeddings word embedding is an important technique in nlp. it computes dense meaning representations for discrete words. it is built upon the distributional hypothesis that linguistic items with similar distributions have similar meanings. several methods have been proposed to learn non-contextualized word embeddings efficiently, including continuous bag-of-words (cbow), skip-gram with negative sampling (sgns) and glove @xcite . in this paper, we use an sgns-based chinese word embedding method as a representative non-contextualized word embeding method for evaluation. contextualized word embeddings such as elmo, gpt and bert have been developed in recent years and shown their high effectiveness for many nlp tasks. in this paper, we use two representative bert variants, bert-wwm and ernie, to evaluate chinese idiom embeddings derived from pre-trained chi-nese bert models. evaluation of chinese word embeddings for word embeddings, existing evaluation methods can be categorized into intrinsic and extrinsic methods @xcite . commonly used intrinsic methods include word similarity and word analogy, while extrinsic methods rely on downstream nlp tasks @xcite . in this paper, we use an intrinsic method to evaluate chinese idiom embeddings. several benchmark datasets for evaluating chinese word embeddings have been released @xcite @xcite @xcite @xcite . but as we pointed out in section 1, existing datasets have low coverage of chinese idioms. neural network models for chinese idiom understanding despite the importance of chengyu in chinese language understanding, there have been only a few pieces of work on chengyu using neural models @xcite @xcite . chinese chengyu recommendation (ccr) has been addressed in recent years @xcite @xcite . in this paper, we adapt a method for ccr @xcite to learn chinese idiom embeddings. ## construction of the evaluation dataset a standard intrinsic task for evaluating word embeddings is word similarity @xcite . for chinese idioms, a natural choice of idiom pairs that are semantically similar are synonyms or near-synonyms foot_1 . although previously @xcite constructed a chinese idiom knowledge base that contains idiom synonyms, this knowledge base is not publicly available. on the other hand, there exist online resources containing synonyms and near-synonyms of chinese idioms. we choose two websites, kxue.com (快学 网) foot_2 and baidu baike (百度百科) foot_3 , as the sources from which to crawl idiom synonyms and nearsynonyms. we also collect idiom antonyms from these two websites because an antonym of an idiom is often topically related to that idiom and therefore may be also close to that idiom in an embedding space. however, we expect a good idiom embedding method to be able to separate antonyms from synonyms. idiom vocabulary: according to @xcite , there are in total around 38k chinese idioms, among which around 3.5k are commonly used. in order to obtain a vocabulary of chinese idioms with high coverage, we merge the idioms found in the following four resources: (1) chengyu daquan foot_4 , (2) xinhua chengyu dictionary foot_5 , (3) chengyu cloze test foot_6 , and (4) chid. 8 . this gives us a chinese idiom vocabulary with 33,237 idioms. chidsyn: as we have pointed out earlier, we believe idiom synonyms can help us evaluate idiom embeddings. to construct a large dataset of chinese idiom synonyms, we crawled synonyms from two websites: (1) kxue.com is an online chinese thesaurus. it has a dedicated page where chinese idiom synonyms are listed. each entry in this list consists of a key and a value, where the key is a chinese idiom and the value is one or more other chinese idioms that are near-synonyms of the key. we crawled all the entries from this idiom synonym page on kxue.come foot_8 . baidu baike is an online encyclopedia in chinese. for each idiom, there is a section called 成语辨析 (chengyu differentiation) that lists its synonyms and antonyms. 10 we crawled the synonyms of those idioms in our vocabulary that can be found on baidu baike. in total, we obtained around 30k entries of chinese synonyms. we then removed those idioms in the data that are not in our idiom vocabulary as described earlier. in the end we obtained a total of around 21k entries in our synonym dataset, where each entry consists of a query idiom and a set of other idioms that are the query idiom's synonyms or near-synonyms. we observe that a significant portion of the synonyms share common characters with the query idioms. for example, 山盟海誓 (oath of eternal love) and 海誓山盟 are treated as near-synonyms in our dataset, but these two idioms contain exactly the same set of chinese characters. in fact, they are variants of the same basic form. another example is 挨家挨户 (door to door) and 挨门挨户, which share three common characters. in general, it is not uncommon for chinese idioms to have such variants due to historical reasons such as misuse (including literary malapropism). although these are valid near-synonyms, we suspect that they may affect the evaluation of idiom embeddings. this is because those idiom embeddings that rely more on character-level information are likely to gain advantages when evaluated on these near-synonym pairs sharing common characters. for example, if an idiom embedding is obtained by averaging the character embeddings of its component characters, then it is very easy for this type of idiom embeddings to recognize that 山盟海誓 and 海誓山盟 are nearsynonyms (because they would have the same average character embedding), but we would not be able to know whether such embeddings truly capture the semantic meanings. we also suspect that for those idioms that have near-synonyms sharing common characters, their semantic meanings are more likely to be compositional and thus less idiomatic. for example, for the idiom 挨家挨户, the character 挨 means "in sequence" and both 家 and 户 mean "household." the meaning of the idiom, which is "door to door," can be directly inferred from the meanings of the characters. therefore, when the character 家 (household) is replaced with the character 门 (door), the meaning of the idiom remains the same. consequently, we move those synonyms that share at least two common characters with the query idioms into a separate dataset, which we will not use as the main evaluation dataset. the remaining synonyms always have no more than one common character with their query idioms. we refer to this cleaned synonym dataset as chidsyn, and the separate dataset containing synonyms sharing two or more common characters is referred to as chidsyn-com. we will use chidsyn-com for additional analysis in our experiments. statistics of chidsyn and chidsyn-com can be found in chidant: from the same two websites, we have also collected around 10k entries in an antonym dataset which we refer to as chidant. similarly, each entry in this dataset consists of a query idiom and its antonyms. although antonyms are idioms having opposite meanings, they are often topically closely related. for example, the idiom 饱学之士 means "a scholarly man," and its antonym 胸无 点墨 means "uneducated." we can see that their meanings are topically closely related. we therefore suspect that they are still close in an embedding space, but ideally a good idiom embedding method should be able to distinguish the synonyms of a query idiom from its antonyms. ## learning chinese idiom embeddings existing chinese word embedding methods can be used to derive idiom embeddings. however, as we have discussed in section 1, they may not be ideal for learning chinese idiom embeddings. in this section, we first briefly review existing chinese word embedding methods and how we use them to obtain idiom embeddings. we then present a method to learn chinese idiom embeddings based on bert. our proposed method is adapted from a method for chinese idiom recommendation @xcite . ## conclusion in this paper, we constructed a new evaluation dataset that contains chinese idiom synonyms and antonyms to facilitate the evaluation of chinese idiom embeddings. we presented a method that learns chinese idiom embeddings by predicting idioms based on bert-encoded contexts. we also propose two metrics to measure closeness of synonyms in the embedding space. our method performs substantially better than existing methods.