Enhancing Transformer with Sememe Knowledge (2020)

Abstract: While large-scale pretraining has achieved great success in many NLP tasks, it has not been fully studied whether external linguistic knowledge can improve data-driven models. In this work, we introduce sememe knowledge into Transformer and propose three sememe-enhanced Transformer models. Sememes, by linguistic definition, are the minimum semantic units of language, which can well represent implicit semantic meanings behind words. Our experiments demonstrate that introducing sememe knowledge into Transformer can consistently improve language modeling and downstream tasks. The adversarial test further demonstrates that sememe knowledge can substantially improve model robustness.

## introduction self-supervised pretraining has significantly improved the performance of transformer @xcite on a wide range of nlp tasks @xcite @xcite . while no explicit linguistic rules and concepts are introduced, models can achieve remarkable performances with extensive training signals provided by large-scale data. nonetheless, recent works still demonstrate that external syntactic information can improve various nlp tasks, including machine translation @xcite @xcite and semantic role labeling @xcite . can external semantic information benefit the widely-adopted pretraining and fine-tuning framework as well? in response, we explore incorporating sememe knowledge into transformer @xcite . sememes are the minimum semantic units of meaning for natural language, as some linguists assume that a limited closed set of sememes can be composed to represent the semantic meaning of each word @xcite . in this work, we adopt a highquality sememe-based lexical knowledge base, hownet @xcite , which can provide powerful support for models to understand chinese word semantics @xcite . some examples of sememe annotations can be found in figure 1 . we propose to combine two simple methods to incorporate sememe knowledge into our framework: 1) based on the linguistic assumption, we add aggregated sememe embeddings to each word embedding to enhance its semantic representation; 2) we use sememe prediction as an auxiliary task to help the model gain deeper understandings of word semantics. we verify the effectiveness of our methods on several chinese nlp tasks that are closely related to word-level and sentence-level semantics. following general settings of pretraining and fine-tuning, our experiments show consistent improvements on all the tasks with sememeenhanced transformer. we also find that the sememe-enhanced model can achieve the same performance with less fine-tuning data, which is desirable as data annotation processes are always time-consuming and expensive. we further demonstrate that, by incorporating sememe knowledge using our methods, model robustness can be significantly improved towards adversarial examples, which are generated by replacing nouns, adjectives and adverbs with their synonyms in our experiment. our case studies further interpret why sememe knowledge can help model defend adversarial attacks. masked multi self attention sememe prediction language modeling supervised learning 水域 ██████ water area 抢劫 ████ rob 水果 █ fruit 海盗 ██████ pirate 罪犯 ███ criminal 苹果 █ apple 新闻 ██████ news 财经 ███ finance 娱乐 █ entertain sp loss lm loss sl loss + large 大 vast 泛 ship 船 transport 运送 human 人 suffer 遭受 thing 事情 country 国家 place 地方 africa 非洲 government 政 proper noun 专 大型 large-scale 客船 passenger ship 遭遇 encounter 索马里 somali layer norm feed forward layer norm 6x predict sememes of next word predict next word downstream task mean mean mean mean transformer + + position embedding + + + + figure 1 : our proposed model architecture. for each word, we enhance word representation by adding aggregated sememe embeddings. we use multitask learning with three tasks: sememe prediction (predicting sememes of next word), language modeling (predicting next word) and supervised learning (only for downstream tasks). ## methodology in this section, we propose two simple methods to incorporate sememe knowledge into our framework: aggregated sememe embeddings and sememe prediction auxiliary task. ## experiments we experiment across a diverse set of five benchmark nlp tasks and demonstrate the effectiveness of introducing sememe knowledge. ## conclusion in this work, we introduce sememe knowledge into transformer and verify the effectiveness of external semantic knowledge for data-driven models. we further demonstrate the robustness of our methods via data ablation study and adversarial test. for future work, we would like to explore more ways to leverage semantic knowledge and generate different adversarial examples for evaluation.