T o HRE : A Top-Down Classification Strategy with Hierarchical Bag Representation for Distantly Supervised Relation Extraction (2020)

Abstract: Distantly Supervised Relation Extraction (DSRE) has proven to be effective to find relational facts from texts, but it still suffers from two main problems: the wrong labeling problem and the long-tail problem. Most of the existing approaches address these two problems through flat classification, which lacks hierarchical information of relations. To leverage the informative relation hierarchies, we formulate DSRE as a hierarchical classification task and propose a novel hierarchical classification framework, which extracts the relation in a top-down manner. Specifically, in our proposed framework, 1) we use a hierarchically-refined representation method to achieve hierarchy-specific representation; 2) a top-down classification strategy is introduced instead of training a set of local classifiers. The experiments on NYT dataset demonstrate that our approach significantly outperforms other state-of-the-art approaches, especially for the long-tail problem.

## introduction knowledge bases (kbs) such as freebase @xcite , dbpedia @xcite , and nell @xcite currently play an essential role in nlp tasks including information retrieval and question answering. as current kbs are still far from complete compared with the infinite real-world facts, relation extraction (re) which aims to extract relations between two entities in texts and enrich kbs has attracted a surge of research interest. most existing supervised re methods require high quality labeled data, which is time-consuming and labor-intensive. therefore, @xcite propose the distant supervision (ds) approach to automatically generate a large amount of training data for re by aligning kbs with large-scale unlabeled corpora. two problems have to be addressed in building an efficient ds model. the main problem, namely the wrong labeling problem, comes from the assumption of ds. ds is based on an assumption: if two entities have a relation in kbs, all sentences that contain these two entities will express this relation. this strong and unrealistic assumption inevitably bring some noise. for example, ds method may mistakenly label the sentence "steve jobs passed away the day before apple unveiled iphone 4s in late 2011" with the relation business/company/founders. secondly, although ds method can automatically generate large-scale training data, this training data can only cover a limited part of relations. nearly 70% of the relations are long-tail and still suffer from data deficiency in the widely used new york times (nyt) dataset. to alleviate the wrong labeling problem, @xcite and @xcite propose the multi-instance learning (mil) framework, which assigns a label to a bag of sentences containing a common entity pair. based on the mil framework, many efforts @xcite @xcite have been devoted to identifying valid sentences from labeled bags. however, they ignore the hierarchical relation information. for example, the relations in freebases are labeled as shown in figure 1 (left). considering the correlations among relations, relations can also be naturally organized as a class hierarchy @xcite -typically like a tree shown in figure 1 to leverage the inherent hierarchical structure of relations, we formulate dsre as a hierarchical classification task, which extracts relations in a top-down manner. intuitively, coarse-grained relations in the top level are easy whereas fine-grained relations in the bottom layer are harder to classify. in this way, we can preferentially extract the relation in the top level and then use the top-level relation to boost the performance of the relation in the bottom level. there exist two challenges when conducting a hierarchical top-down classification: capturing specific bag representations in different levels and training large amounts of classifiers. for the challenge of the bag representation, a bag of sentences expresses different relations in different levels. hence, it is necessary to dynamically adjust the bag representations in different relation levels. to capture the hierarchy-specific bag representation, we propose a hierarchical bag representation method, which incorporates the hierarchically-refined selective attention mechanism to dynamically adjust the bag representation in different levels. for the challenge of massive classifiers, traditional hierarchical classification methods which adopt the top-down manner need to train a set of local classifiers (d' @xcite @xcite . the number of local classifiers depends on the size of the label hierarchies, making hierarchical classification infeasible to scale. to handle the massive local classifiers problem, we introduce a top-down classification strategy which shares most of its parameters in different levels to avoid training massive local classifiers, thus making our methods available for various relation hierarchies. we name our approach a top-down classification strategy for hierarchical relation extraction (tohre). our main contributions are summarized as follows: different from previous methods, our proposed model aims to explore the relation hierarchy h in a top-down manner and output relation probability in each relation level. the entity-pair bag along with the history of parent relations are integrated to predict a relation from pre-defined relation classes r. framework our model consists of two key components: hierarchical bag representation module and a top-down classification strategy. the hierarchical bag representation module is shown in figure 2 . it takes an entity-pair bag as input and outputs the bag representations in different relation levels. first, each sentence in the entity-pair bag is transformed to a matrix with the entity-aware embedding. then, a piecewise convolution layer @xcite is used to obtain the sentence representation. after that, a hierarchically-refined selective attention is leveraged to select sentences in the bag which actually expresses the corresponding hierarchical relation and output hierarchy-specific bag representation. the top-down classification strategy is illustrated in figure 3 . the designed strategy takes the hierarchical bag representation and corresponding relation embeddings as input to discriminate its child relations, then walks down the relation hierarchies to predict the next relation until the leaf node. the details of the two components are introduced in section 2.2 and section 2.3 respectively. ## conclusion in this paper, we take advantage of the inherent hierarchical structure of relations and propose a topdown classification strategy with a hierarchical bag presentation. in this way, we formulate the dsre as a hierarchical classification task. the experimental results indicate that our proposed model outperforms previous state-of-the-art flat methods, especially for long-tail relations. in the future, we plan to explore the following directions: (1) incorporating entity information from external knowledge graphs to enhance the hierarchical bag representation; (2) utilizing more sophisticated training strategies for the long-tail relation problem.