CN - HIT - IT . NLP at S em E val-2020 Task 4: Enhanced Language Representation with Multiple Knowledge Triples (2020)

Abstract: This paper describes our system that participated in the SemEval-2020 task 4: Commonsense Validation and Explanation. For this task, it is obvious that external knowledge, such as Knowledge graph, can help the model understand commonsense in natural language statements. But how to select the right triples for statements remains unsolved, so how to reduce the interference of irrelevant triples on model performance is a research focus. This paper adopt a modified K-BERT as the language encoder, to enhance language representation through triples from knowledge graphs. Experiments show that our method is better than models without external knowledge, and is slightly better than the original K-BERT. We got an accuracy score of 0.97 in subtaskA, ranking 1/45, and got an accuracy score of 0.948, ranking 2/35.

## introduction in recent years, language models trained on large scale corpora @xcite @xcite have performed exceptionally well on many benchmarks @xcite , reaching or even surpassing human performance. it seems to show that natural language understanding (nlu) is becoming easier. and the level of nlu may be seen in the ability to understand commonsense in natural language statements. therefore, it is important to be able to evaluate how well a model can do for sense making @xcite . for more direct research on commonsense in natural language statements, commonsense validation and explanation (comve) is proposed by @xcite . the comve task consists of three subtasks, and we participated in subtaska and subtaskb. subtaska is validation, requiring the model to identify which statement makes sense from the given two. then for the against-common-sense statement, three optional sentences are provided to explain why the statement does not make sense. in subtaskb, named explanation (multi-choice), the only one correct reason is required to be identified from two other confusing ones. intuitively, knowledge graphs (kgs) can help language models understand commonsense. for example, for the sentence, "all whales are small", the @xmath0, hasproperty, @xmath1. under such motivation, k-bert @xcite injects triples into sentences as domain knowledge. however, how to select the helpful triples from kg remains a problem. when we fuse too much external knowledge, irrelevant knowledge will adversely affect the model, which is called knowledge noise (kn) issue. to overcome kn issue, k-bert introduces soft-position and visible matrix to limit the influence of the triples. but kn issue is still serious, especially when the number of injected triples increases. therefore, this paper introduce a variant of k-bert for further reduction of kn. besides, we choose conceptnet @xcite as the commonsense repository. different from the domain-specific kg, conceptnet contains more than 1,500,000 english nodes. this means that you can find almost every word in conceptnet, which make it more difficult to choose the relevant triples. so this paper adopt a simple and junior threshold-based method to deal with this problem. tokens basic-level masking whole-word masking entity-level masking [mask] [mask] [mask] [mask] [mask] is a character al ##bus [mask] [mask] [mask] is a character al ##bus [mask] ##led ##ore is a character al ##bus dumb ##led ##ore is a character sentence: albus dumbledore is a character. 2 related work ## encoder enhanced by triples in this section, we describe our encoder, which is a variant of k-bert @xcite . its general framework is the same as k-bert presented in figure 1(b) , but some details are modified. ## conclusion in this paper, we propose a variant of k-bert as the language encoder to help the model understand commonsense. we adopt a simple threshold-based method for triple selection and apply our encoder on semeval-2020 task 4: commonsense validation and explanation, and achieve good performance (ranking 1st place in subtska). this shows that our system can effectively enhance the language representation with multiple knowledge triples. however, when the number of injected triples is further increased, or triples of different relations are injected into the sentence, the performance of our model will deteriorate. therefore, we believe that future work can be carried out in the following points: (1) pre-training language models on sentences injected with triples; (2) recognizing multi-word entities an d developing better selection methods, such as those based on similarity; (3) designing a robust and noise-resistant knowledge integration model to integrate more triples.