Revisiting Unsupervised Relation Extraction (2020)

Abstract: Unsupervised relation extraction (URE) extracts relations between named entities from raw text without manually-labelled data and existing knowledge bases (KBs). URE methods can be categorised into generative and discriminative approaches, which rely either on hand-crafted features or surface form. However, we demonstrate that by using only named entities to induce relation types, we can outperform existing methods on two popular datasets. We conduct a comparison and evaluation of our findings with other URE techniques, to ascertain the important features in URE. We conclude that entity types provide a strong inductive bias for URE.

## introduction relation extraction (re) extracts semantic relations between entities from plain text. for instance, "jon robin baitz head , born in los angeles tail ..." expresses the relation /people/person/place of birth between the two head-tail entities. extracted relations are then used for several downstream tasks such as information retrieval @xcite and knowledge base construction @xcite . re has been widely studied using fully supervised learning @xcite @xcite and distantly supervised approaches @xcite @xcite . unsupervised relation extraction (ure) methods have not been explored as much as fully or distantly supervised learning techniques. ure is promising, since it does not require manually annotated data nor human curated knowledge bases (kbs), which are expensive to produce. therefore, it can be applied to domains and languages where annotated data and kbs are not available. moreover, ure can discover new relation types, since it is not restricted to specific relation types in the same way as fully and distantly supervised methods. one might argue that open information extraction (openie) can also discover new relations. however, openie identifies relations based on textual surface information. thus, similar relations with different textual forms may not be recognised. unlike openie, ure groups similar relations into clusters. despite these advantages, there are only a few attempts tackling ure using machine learning (ml) @xcite @xcite @xcite . similarly to other unsupervised learning tasks, a challenge in ure is how to evaluate results. recent approaches @xcite @xcite employ a widely used data generation setting in distantly supervised re, i.e., aligning a large amount of raw text against triplets in a curated kb. a standard metric score is computed by comparing the output relation clusters against the automatically annotated relations. in particular, the nyt-fb dataset @xcite which is used for evaluation, has been created by mapping relation triplets in freebase @xcite against plain text articles in the new york times (nyt) corpus @xcite . standard clustering evaluation metrics for ure include b 3 @xcite , v-measure (rosenberg and hirschberg, 2007), and ari @xcite . although the above mentioned experimental setting can be created automatically, there are three challenges to overcome. firstly, the development and test sets are silver, i.e., they include noisy labelled instances, since they are not human-curated. secondly, the development and test sentences are part of the training set, i.e., a transductive setting. it is thus unclear how well the existing models perform on unseen sentences. finally, nyt-fb can be considered highly imbalanced, since only 2.1% of the training sentences can be aligned with freebase's triplets. due to the noisy nature of silver data (nyt-fb), evaluation on silver data will not accurately reflect the system performance. we also need unseen data during testing to examine the system generalisation. to overcome these challenges, we will employ the test set of tacred @xcite , a widely used manually annotated corpus. regarding the imbalanced data, we will demonstrate that in fact around 60% (instead of 2.1%) of instances in the training set express relation types defined in freebase. in this work, we present a simple ure approach relying only on entity types that can obtain improved performance compared to current methods. specifically, given a sentence consisting of two entities and their corresponding entity types, e.g., person and location, we induce relations as the combination of entity types, e.g., person-location. it should be noted that we employ only entity types because their combinations form reasonably coarse relation types (e.g., person-location covers /people/person/place of birth defined in freebase). we further discuss our improved performance in §3. our contributions are as follows: (i) we perform experiments on both automatically/manuallylabelled datasets, namely nyt-fb and tacred, respectively. we show that two methods using only entity types can outperform the state-of-theart models including both feature-engineering and deep learning approaches. the surprising results raise questions about the current state of unsupervised relation extraction. (ii) for model design, we show that link predictor provides a good signal to train a ure model (fig 1 ). we also illustrate that entity types are a strong inductive bias for ure ( ## methods for ure the goal of ure is to predict the relation r between two entities e head and e tail in a sentence s. we will describe three recent ml-based methods tackling ure and our own methods. we divide the ml-based methods into two main approaches: generative and discriminative. ## experiments and results evaluation metrics we use the following evaluation metrics for our analysis: a) b 3 @xcite used in previous work, which is the harmonic mean of precision and recall for clustering task; b) v-measure (rosenberg and hirschberg, 2007), and c) ari @xcite used in @xcite . foot_1 v-measure is analysed in terms of homogeneity and completeness, while ari measures the similarity between two clusterings. we note that v-measure is sensitive to the dependency between the number of clusters and instances. a relatively small number of clusters compared to the number of instances should be used to maintain the comparability of using v-measure. more precisely, we evaluated v-measure of the trivial homogeneity, where there are only singular clusters (i.e., each instance is its own cluster). the v-measure of the trivial homogeneity on nyt-fb reached 43.77%, which is higher than all the implemented methods in hyper-parameters we examine three models rellda1, march, and simon using the reported hyper-parameters @xcite ; marcheggiani 0 5 10 15 20 25 30 epoch 0.1 0.2 0.3 0.4 0.5 0.6 log loss rand10 rand10 with silver frequencies one relation etype silver relations (10) silver relations (full) figure 1: average negative log likelihood losses across three runs of the link predictor on the training data (not including negative instances). each line demonstrates a different relation input setting. and titov, 2016; simon et al., 2019). for comparison, we also evaluate march with the two regularisers of simon, namely march (l s + l d ). to evaluate on tacred, we employed the original march with @xmath0. meanwhile, for march (l s +l d ) and simon, we reimplemented these models and evaluated them on tacred. regarding our methods, etype does not have hyper-parameters, while etype+ uses the same optimiser and entity type dimension as in simon. all the hyper-parameters used in our experiments are listed in appendix b. results ## discussion the results of our evaluation demonstrate that our models outperform previous methods, despite being simpler than them. these results lead us to the ## conclusion we have shown the importance of entity types in ure. our methods use only entity types, yet they yield higher performance than previous work on both nyt-fb and tacred. we have investigated the current experimental setting, concluding that a strong inductive bias is required to train a relation extraction model without labelled data. ure remains challenging, which requires improved methods to deal with silver data. we also plan to use different types of labelled data, e.g., domain specific data sets, to ascertain whether entity type information is more discriminative in sub-languages. mike mintz, steven bills, rion snow, and daniel jurafsky. 2009. distant supervision for relation extraction without labeled data. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 1003-1011, suntec, singapore. association for computational linguistics. makoto miwa and mohit bansal. 2016. end-to-end relation extraction using lstms on sequences and tree structures. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1105-1116, berlin, germany. association for computational linguistics. thien huu nguyen and ralph grishman. 2015. relation extraction: perspective from convolutional neural networks. in proceedings of the 1st workshop on vector space modeling for natural language processing, pages 39-48, denver, colorado. association for computational linguistics. xiang ren, zeqiu wu, wenqi he, meng qu, clare r voss, heng ji, tarek f abdelzaher, and jiawei han. 2017. cotype: joint extraction of typed entities and relations with knowledge bases. in proceedings of the 26th international conference on world wide web, pages 1015-1024. sebastian riedel, limin yao, and andrew mccallum. 2010. modeling relations and their mentions without labeled text. in joint european conference on machine learning and knowledge discovery in databases, pages 148-163. springer. simone romano, nguyen xuan vinh, james bailey, and karin verspoor. 2016. adjusting for chance clustering comparison measures. journal of machine learning research, 17(1):4635-4666. andrew rosenberg and julia hirschberg. 2007. vmeasure: a conditional entropy-based external cluster evaluation measure. in proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (emnlp-conll), pages 410-420, prague, czech republic. association for computational linguistics. evan sandhaus. 2008. the new york times annotated corpus. linguistic data consortium, philadelphia, 6(12):e26752. étienne simon, vincent guigue, and benjamin piwowarski. 2019. unsupervised information extraction: regularizing discriminative approaches with relation distribution losses. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 1378-1387, florence, italy. association for computational linguistics. kiri wagstaff. 2000. refining inductive bias in unsupervised learning via constraints. in aaai/iaai, page 1112. limin yao, aria haghighi, sebastian riedel, and andrew mccallum. 2011. structured relation discovery using generative models. in proceedings of the 2011 conference on empirical methods in natural language processing, pages 1456-1466, edinburgh, scotland, uk. association for computational linguistics. yuhao zhang, peng qi, and christopher d. manning. 2018. graph convolution over pruned dependency trees improves relation extraction. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2205-2215, brussels, belgium. association for computational linguistics. yuhao zhang, victor zhong, danqi chen, gabor angeli, and christopher d. manning. 2017. positionaware attention and supervised data improve slot filling. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 35-45, copenhagen, denmark. association for computational linguistics.