Regularized Attentive Capsule Network for Overlapped Relation Extraction (2020)

Abstract: Distantly supervised relation extraction has been widely applied in knowledge base construction due to its less requirement of human efforts. However, the automatically established training datasets in distant supervision contain low-quality instances with noisy words and overlapped relations, introducing great challenges to the accurate extraction of relations. To address this problem, we propose a novel Regularized Attentive Capsule Network (RA-CapNet) to better identify highly overlapped relations in each informal sentence. To discover multiple relation features in an instance, we embed multi-head attention into the capsule network as the low-level capsules, where the subtraction of two entities acts as a new form of relation query to select salient features regardless of their positions. To further discriminate overlapped relation features, we devise disagreement regularization to explicitly encourage the diversity among both multiple attention heads and low-level capsules. Extensive experiments conducted on widely used datasets show that our model achieves significant improvements in relation extraction.

## introduction relation extraction aims to extract relations between entities in text, where distant supervision proposed by @xcite automatically establishes training datasets by assigning relation labels to instances that mention entities within knowledge bases. however, the wrong labeling problem can occur and various multi-instance learning methods @xcite @xcite have been proposed to address it. despite the wrong labeling problem, each instance in distant supervision is crawled from web pages, which is informal with many noisy words and can express multiple similar relations. this problem is not well-handled by previous approaches and severely hampers the performance of conventional neural relation extractors. to handle this problem, we have to address two challenges: (1) identifying and gathering spotted relation information from low-quality instances; (2) distinguishing multiple overlapped relation features from each instance. first, a few significant relation words are distributed dispersedly in the sentence, as shown in figure 1 , where words marked in red brackets represent entities, and italic words are key to expressing the relations. for instance, the clause "evan bayh son of birch bayh" in s1 is sufficient to express the relation /people/person/children of evan bayh and birch bayh. salient relation words are few in number and dispersedly in s1, while others excluded from the clause can be regarded as noise. traditional neural models have difficulty gathering spotted relation features at different positions along the sequence because they use convolutional neural network (cnn) or recurrent neural network (rnn) as basic relation encoders @xcite @xcite , which model each sequence word by word and lose rich non-local information for modeling the dependencies of semantic salience. thus, a well-behaved relation extractor is needed to extract scattered relation features from informal instances. second, each instance can express multiple similar relations of two entities. as shown in figure 1 , changsha and hunan possess the relations /location/location/contains and /location/province/capital in id instances relations s1 senator [evan_bayh], son of former senator [birch_bayh] of indiana, is organizing and testing the waters for a possible presidential bid in 2008. /people/person/children s2 that is one reason that [hunan]'s fast-growing provincial capital, [changsha], is beginning to siphon some workers. /location/location/contains /location/province/capital s3 the land is near calgary; while that is one of [alberta]'s largest cities, the capital is [edmonton]. /location/location/contains /location/province/capital /location/country/capital … … … figure 1 : example of instances from the new york times (nyt). s2, which have similar semantics, introducing great challenges for neural extractors in discriminating them clearly. conventional neural methods are not effective at extracting overlapped relation features, because they mix different relation semantics into a single vector by max-pooling @xcite or self-attention @xcite . although @xcite first propose an attentive capsule network for multi-labeled relation extraction, it treats the cnn/rnn as low-level capsules without the diversity encouragement, which poses the difficulty of distinguishing different and overlapped relation features from a single type of semantic capsule. therefore, a well-behaved relation extractor is needed to discriminate diverse overlapped relation features from different semantic spaces. to address the above problem, we propose a novel regularized attentive capsule network (ra-capnet) to identify highly overlapped relations in the low-quality distant supervision corpus. first, we propose to embed multi-head attention into the capsule network, where attention vectors from each head are encapsulated as a low-level capsule, discovering relation features in an unique semantic space. then, to improve multi-head attention in extracting spotted relation features, we devise relation query multi-head attention, which selects salient relation words regardless of their positions. this mechanism assigns proper attention scores to salient relation words by calculating the logit similarity of each relation representation and word representation. furthermore, we apply disagreement regularization to multihead attention and low-level capsules, which encourages each head or capsule to discriminate different relation features from different semantic spaces. finally, the dynamic routing algorithm and slidingmargin loss are employed to gather diverse relation features and predict multiple specific relations. we evaluate ra-capnet using two benchmarks. the experimental results show that our model achieves satisfactory performance over the baselines. our contributions are summarized as follows: ## related work distantly supervised relation extraction has been essential for knowledge base construction since @xcite propose it. to address the wrong labeling problem in distant supervision, multi-instance and multi-label approaches are proposed @xcite @xcite . with the renaissance of neural networks, increasing researches in distant supervision have been proposed to extract precise relation features. piecewise cnns with various attention mechanisms are proposed @xcite @xcite . reinforcement learning and adversarial training are proposed to select valid instances to train relation extractors @xcite @xcite . recently, multi-level noise reduction is designed by @xcite . nevertheless, the above approaches ignore the effect of noisy words and overlapped relation features in each instance. to reduce the impact of noisy words, tree-based methods attempt to obtain the relevant sub-structure of an instance for relation extraction @xcite @xcite . to discriminate overlapped relation features, @xcite apply the capsule network @xcite for multi-labeled relation extraction. inspired by the ability of multi-head attention in modeling the long-term dependency @xcite , @xcite attempt to reduce multi-granularity noise via multi-head attention in relation extraction. ## methodology as shown in figure 2 , we will introduce the three-layer ra-capnet: (1) the feature encoding layer primarily contains the word encoding layer and blstm encoding layer. (2) the feature extracting layer chiefly includes relation query multi-head attention and disagreement regularization. ## experiments our experiments are devised to demonstrate that ra-capnet can identify highly overlapped relations of informal instances in distant supervision. in this section, we first introduce the dataset and experimental setup. then, we evaluate the overall performance of ra-capnet and the effects of different parts of ra-capnet. finally, we present the case study.