Leveraging Capsule Routing to Associate Knowledge with Medical Literature Hierarchically (2021)

Abstract: Integrating knowledge into text is a promising way to enrich text representation, especially in the medical field. However, undifferentiated knowledge not only confuses the text representation but also imports unexpected noises. In this paper, to alleviate this problem, we propose leveraging capsule routing to associate knowledge with medical literature hierarchically (called HiCapsRKL). Firstly, HiCapsRKL extracts two empirically designed text fragments from medical literature and encodes them into fragment representations respectively. Secondly, the capsule routing algorithm is applied to two fragment representations. Through the capsule computing and dynamic routing, each representation is processed into a new representation (denoted as caps-representation), and we integrate the caps-representations as information gain to associate knowledge with medical literature hierarchically. Finally, HiCapsRKL are validated on relevance prediction and medical literature retrieval test sets. The experimental results and analyses show that HiCapsRKLcan more accurately associate knowledge with medical literature than mainstream methods. In summary, HiCapsRKL can efficiently help selecting the most relevant knowledge to the medical literature, which may be an alternative attempt to improve knowledge-based text representation. Source code is released on GitHub.

## introduction knowledge is known as a triple to describe the relationship (r) between head entity (e h ) and tail entity (e t ) with the format @xmath0, r, e t >. the popular neural models can improve the ability of learning text representation by integrating knowledge, because they usually lack the ability to learn entities and their relationship in the text. however, the medical literature usually contains multiple knowledge, and not all knowledge is beneficial to its subject. the integration of undifferentiated knowledge into the neural models may reduce the accuracy of medical literature representation. for example, in figure 1 , it lists three knowledge existed in the medical literature. but, the third knowledge is redundant to the subject of the literature. therefore, it is an essential step to determine the hierarchical association between knowledge and medical literature before integrating knowledge. the hierarchical association between knowledge and medical literature refers to the definition of the degree of their relevance according to how the subject of the medical literature covers the knowledge. given a knowledge and a medical literature, the task asks to predict their relevance from four levels, namely "highly relevance (h r )", "fairly relevance (f r )", "marginally relevance (m r )", "irrelevance (i r )" @xcite . the public four-point scale graded relevance assessment @xcite from text retrieval conference (trec) is commonly used for the hierarchical association (see however, it is difficult for the mainstream methods to capture these two information, mainly because 1) the subject of medical literature is usually multi-knowledge entangled, and these methods seldom can learn the unique knowledge from it; 2) the expression of the relationship information in medical literature is complex and abstract, which requires methods with strong distinguishing ability. this paper proposes leveraging the capsule routing algorithm to extract rcor and kimp information. the capsule routing algorithm is proposed for the capsule network by @xcite , and is an efficient algorithm for decoupling multiple object feature. for the multiple entangled knowledge and complex relationship, the capsule routing algorithm splits the input feature into multiple capsules, and the capsule in a lower-level hands out its output to higher-level capsules through routing algorithm, completing the extraction and aggregation of information flow. after multi-layer capsule calculation, the final layer capsule represents unique knowledge or relation information to solve the issues of knowledge entanglement and complex relationship in medical literature, and then determine the hierarchical association of knowledge and medical literature. in summary, in this paper our contributions include: 1) proposing hierarchically associating knowledge with medical literature from relationship correlation and knowledge importance, and recording these information through two empirically designed text fragments in the medical literature; 2) proposing leveraging capsule routing algorithm to model the rcor and kimp text fragments (called hicapsrkl), and taking them as information gain to judge the hierarchical association of knowledge and medical literature; 3) building a weakly supervised training set, a relevance prediction test set and a medical literature retrieval test set, and using these sets to test and analyze the proposed hicapsrkl and other comparison methods. the experimental results and analyses prove the efficiency of hicapsrkl in associating knowledge and medical literature hierarchically. ## related works the neural information retrieval (ir) models are available techniques for associating knowledge with medical literature because of their powerful deep neural architectures, like cnn @xcite , rnn @xcite , and pre-trained bert @xcite . for example, @xcite proposed a joint deep architecture to associate knowledge with medical literature at the query term level. @xcite proposed the positionaware model considering position-dependent interactions. @xcite incorporated information from the word space, the entity space, and the cross-space connections through the knowledge. @xcite used a translation matrix to model word-level similarities and multi-level soft match features for their association. @xcite used cnn for n-grams of various lengths and soft matched them in a unified embedding r kimp pairwise operation language encoder r rcor r c kimp r c rcor r medl r' medl kimp label prediction rcor label prediction relevance prediction feature integration rcor text fragment medical literature kimp text fragment knowledge (head entity relationship, tail entity) input texts figure 2: the brief architecture of the proposed hicapsrkl model. space. liu et al. (2018b) integrated knowledge to neural models by representing texts with words and entity annotations. dai and callan (2019) simulated detecting, determining, and aggregating of human judgement process to associate knowledge and medical literature. macavaney et al. (2019a) relatively, the capsule network is newly proposed neural architecture in recent years and still being explored for its applications in nlp area @xcite @xcite . several researches have explored to apply the capsule network to various nlp tasks, e,g., sentiment classification @xcite @xcite , relation extraction @xcite ), text classification @xcite @xcite , intent detection @xcite @xcite , document translation @xcite , word sense disam-biguation @xcite , etc. most of these works followed the convolution and dynamic rooting architecture in capsule network and did not explore the effectiveness of the capsule routing algorithm for nlp tasks alone @xcite . in this work, the proposed hicapsrkl model uses the capsule routing algorithm to learn the relationship correlation and knowledge importance information in texts, and it can semantically explore the hierarchical association of knowledge and medical literature. ## method description figure 2 shows the brief architecture of the proposed hicapsrkl model. first, the model inputs include the input texts (i.e. the rcor text fragment, the medical literature, and the kimp text fragment) and the knowledge triple. second, each text and knowledge are pair-wised and passed into the language encoder to learn the contextual representation for each pair, namely r m edl , r rcor , and r kimp . third, two representations r rcor and r kimp go through two capsule routing algorithms, respectively. the algorithm in each branch outputs the corresponding new caps-representation r c rcor and r c kimp . finally, the model integrates three representations for the relevance prediction. besides, to learn accurate rcor and kimp features, the model uses each caps-representation to predict its rcor or kimp label as defined in ## conclusion in this paper, we proposed hicapsrkl to leverage capsule routing to associate knowledge with medical literature hierarchically. this is a worthy research work for better integrating knowledge to learn rich text representation. on two manually labeled test sets, namely the relevance prediction test set and medical literature retrieval test set, the proposed hicapsrkl model has shown sota performances than other comparison methods. exhaustive experimental results and analyses have proven the excellent ability of the proposed model, and showed its potential on learning association features. in the future, we will focus on applying this work to improve the text representation of the knowledge integration methods by the hierarchical knowledge. for example, the hicapsrkl can be used as multitask, using the relevance of the knowledge, e.g. the softmax probability or relevance label, as a weight or filter to control the integrating process. hicap-srkl will help to reduce the effect of the noisy knowledge and may further improve the quality of text representation. besides, this work can also contribute to other nlp researches (e.g., the medical information processing, question answering, information retrieval, reading comprehension, etc), which may benefit from integrating knowledge.