A BERT -based Dual Embedding Model for C hinese Idiom Prediction (2020)

Abstract: Chinese idioms are special fixed phrases usually derived from ancient stories, whose meanings are oftentimes highly idiomatic and non-compositional. The Chinese idiom prediction task is to select the correct idiom from a set of candidate idioms given a context with a blank. We propose a BERT-based dual embedding model to encode the contextual words as well as to learn dual embeddings of the idioms. Specifically, we first match the embedding of each candidate idiom with the hidden representation corresponding to the blank in the context. We then match the embedding of each candidate idiom with the hidden representations of all the tokens in the context thorough context pooling. We further propose to use two separate idiom embeddings for the two kinds of matching. Experiments on a recently released Chinese idiom cloze test dataset show that our proposed method performs better than the existing state of the art. Ablation experiments also show that both context pooling and dual embedding contribute to the improvement of performance.

## introduction in this paper, we study chinese idiom prediction, a language understanding problem that has not been extensively explored before in computational linguistics. chinese idioms, mainly chengyu (成语) (set phrases) @xcite , have fixed forms in structure; the component characters (mostly four) cannot be changed. chinese idioms are characterized by rich contents, concise forms and frequent use @xcite with properties of structural regularity, semantic fusion, and functional integrity @xcite . chinese idioms are commonly used in both written and spoken chinese, and understanding chinese idioms is important for learning chinese as a second language. the meaning of each chinese idiom may not be literally understood through the composition of its characters, especially for those which are derived from historical stories or formulated using ancient chinese grammars. for example, "一定不易" is literally interpreted as "it must be not easy" in modern chinese. however, the idiom is constructed from grammars and word senses of ancient chinese. its idiomatic meaning is "once decided, never change", which is not even close to the literal meaning. as a result, the usage of chinese idioms poses a challenge on language understanding not only for humans but also for artificial intelligence. due to their pervasive usage, chinese idiom prediction is an important task in chinese language understanding. there have been several studies focusing on representing chinese idioms using neural network models @xcite ), but they were limited by the amount of data available for training. recently, @xcite released a large-scale chinese idiom dataset (chid) to facilitate machine comprehension of chinese idioms. the chid dataset contains more than 500k passages and 600k blanks, making it possible for researchers to train deep neural network models. the dataset is in cloze test style that target chinese idioms in passages are replaced by blanks. for each blank, a set of candidate chinese idioms is provided and the task is to pick the correct one based on the context. candidates: in this paper, we propose a bert-based dual embedding model for the chinese idiom prediction task. we first present two baseline models that use bert to process and match passages and candidate answers in order to rank the candidates. observing that these baselines do not explicitly model the global, long-range contextual information in the given passage for chinese idiom prediction, we propose a context-aware pooling operation to force the model to explicitly consider all contextual words when matching a candidate idiom with the passage. furthermore, we propose to split the embedding vector of each chinese idiom into two separate vectors, one modeling its local properties and the other modeling its global properties. we expect the embedding for local properties to capture the syntactic properties of an idiom, while the embedding for global properties to capture its topical meaning. in addition, using idiom embeddings makes it possible for us to consider the entire chinese idiom vocabulary as the candidate set, which is computationally intractable compared to pretrained bert models with multiple-sequence classification. we apply this enlarged candidates heuristic to all the models with idiom embeddings to further strengthen the performance. to evaluate the effectiveness of the bert-based dual embedding model, we conduct experiments on the chid dataset. our experiments show that our method can outperform several existing methods tested by @xcite as well as our baseline methods. we also find that both context-aware pooling and dual embedding contribute to the performance improvement. to prove the effectiveness of our model, we also evaluate it against a public leaderboard of chid competition. the results show that our model is competitive compared to the top-ranked systems. we can also achieve better performance with a large margin compared with several methods using pretrained language models. we also conduct further analysis using a gradient-based attribution method to check if our model can indeed capture global information to make correct predictions. some case studies show that indeed our method makes use of more global contextual information to make predictions. ## experiments in this section, we evaluate our proposed dual embedding method using the chid dataset. we also use an attribution method to visualize how each proposed method works on some selected cases. ## conclusion in this paper, we proposed a bert-based dual embedding method to study chinese idiom prediction. we used a dual-embedding to not only capture local context information but also match the whole context passage. our experiments showed that our dual-embedding design can improve the performance of the base model, and both the idea of context-aware pooling and the idea of dual embedding can help improve the idiom prediction performance compared to the baseline methods on the chid dataset.