Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priors (2021)

Abstract: We propose a multi-task, probabilistic approach to facilitate distantly supervised relation extraction by bringing closer the representations of sentences that contain the same Knowledge Base pairs. To achieve this, we bias the latent space of sentences via a Variational Autoencoder (VAE) that is trained jointly with a relation classifier. The latent code guides the pair representations and influences sentence reconstruction. Experimental results on two datasets created via distant supervision indicate that multi-task learning results in performance benefits. Additional exploration of employing Knowledge Base priors into theVAE reveals that the sentence space can be shifted towards that of the Knowledge Base, offering interpretability and further improving results.

## introduction distant supervision (ds) is a setting where information from existing, structured knowledge, such as knowledge bases (kb), is exploited to automatically annotate raw data. for the task of relation extraction, this setting was popularised by @xcite . sentences containing a pair of interest were annotated as positive instances of a relation, if and only if the pair was found to share this relation in the kb. however, due to the strictness of this assumption, relaxations were proposed, such as the at-least-one assumption introduced by @xcite : instead of assuming that all sentences in which a known related pair appears express the relationship, we assume that at least one of these sentences (namely a bag of sentences) expresses the relationship. figure 1 shows example bags for two entity pairs. 1 source code is available at https://github.com/ fenchri/dsre-vae @xcite . the usefulness of distantly supervised relation extraction (dsre) is reflected in facilitating automatic data annotation, as well as the usage of such data to train models for kb population @xcite . however, dsre suffers from noisy instances, long-tail relations and unbalanced bag sizes. typical noise reduction methods have focused on using attention @xcite or reinforcement learning @xcite . for long-tail relations, relation type hierarchies and entity descriptors have been proposed @xcite @xcite , while the limited bag size is usually tackled through incorporation of external data @xcite , information from kbs @xcite or pre-trained language models @xcite . our goal is not to investigate noise reduction, since it has already been widely addressed. instead, we aim to propose a more general framework that can be easily combined with existing noise reduction methods or pre-trained language models. methods that combine information from knowledge bases in the form of pre-trained knowledge graph (kg) embeddings have been particularly effective in dsre. this is expected since they capture broad associations between entities, thus assisting the detection of facts. existing approaches either encourage explicit agreement between sentence-and kb-level classification decisions @xcite , minimise the distance between kb pairs and sentence embeddings @xcite or directly incorporate kb embeddings into the training process in the form of attention queries @xcite @xcite . although these signals are beneficial, direct usage of kb embeddings into the model often requires explicit kb representations of entities and relations, leading to poor generalisation to unseen examples. in addition, forcing decisions between kb and text to be the same makes the connection between contextagnostic (from the kb) and context-aware (from sentences) pairs rigid, as they often express different things. variational autoencoders (vaes) (kingma and welling, 2013) are latent variable encoder-decoder models that parameterise posterior distributions using neural networks. as such, they learn an effective latent space which can be easily manipulated. sentence reconstruction via encoder-decoder networks helps sentence expressivity by learning semantic or syntactic similarities in the sentence space. on the other hand, signals from a kb can assist detection of factual relations. we aim to combine these two using a vae together with a bag-level relation classifier. we then either force each sentence's latent code to be close to the normal distribution @xcite , or to a prior distribution obtained from kb embeddings. this latent code is employed into sentence representations for classification and is responsible for sentence reconstruction. as it is influenced by the prior we essentially inject signals from the kb to the target task. in addition, sentence reconstruction learns to preserve elements that are useful for the bag relation. to the best of our knowledge, this is the first attempt to combine a vae with a bag-level classifier for dsre. finally, there are methods for dsre that follow a rather flawed evaluation setting, where several test pairs are included in the training set. under this setting, the generalisability of such methods can be exaggerated. we test these approaches under data without overlaps and find that their performance is severely deprecated. with this comparison, we aim to promote evaluation on the amended version of existing dsre data that can prevent memori-sation of test pair relations. our contributions are threefold: our results suggest that combination of both bag classification and bag reconstruction improves the target task. ## results the results of the proposed approach versus existing methods on the nyt10 dataset are shown in regarding the results on the wikidistant dataset in ## analysis we then check whether the latent space has indeed learned some information about the kb triples, by visualising the t-sne plots of the priors, i.e. the µ kb vectors as resulted from training transe (equation ( 8 )) and the posteriors, i.e. the µ vectors as resulted from the vae encoder (equation ( 1 )). figure 4a illustrates the space of the priors in freebase for the most frequent relation categories in the nyt10 training set 8 . as it can be observed, 7 if a pair does not have a kb prior it will be assigned the normal prior instead. 8 we plot t-snes for the training set instead of the validation/test sets because the wikidistant validation set contains too few pairs belonging to the top-10 categories. nyt10 validation set t-sne can be found in the appendix a.5 the separation is obvious for most categories, with a few overlaps. relations place of birth, place lived and place of death appear to reside in the same region. this is expected as these relations can be shared by a pair simultaneously. another overlap is identified for contains, administrative divisions and capital. again, these are similar relations found between certain entity types (e.g. location, province, city). figure 4b shows the t-sne plot for a collection of latent vectors (random selection of 2 sentences in a positive bag). the space is very similar to that of the kb and the same overlapping regions are clearly observed. a difference is that it appears to be less compact, as not all sentences in a bag express the exact same relation. similar observations stand for wikidata priors, as shown in figure 4c . by looking at the space of the posteriors, we can see that although for most categories separation is achieved, there are 2 relations that are not so well separated in the posterior space. we find that has part (cyan) and part of (orange) are opposite relations, that transe can effectively learn thanks to its properties. however, the model appears to not be able to fully separate the two. these relations are expressed in the same manner, by only changing the order of the arguments. as there is no restriction regarding the argument order in our model directionality can sometimes be an issue. finally, in order to check how the prior constraints affect sentence reconstruction, we illustrate reconstructions of sentences in the validation set of the nyt10 in manual inspection of reconstruction reveals that kb-priors generate longer sentences than the normal prior by repeating several words (especially the unk). in fact, vae with kb-priors fails to generate plausible and grammatical examples for nyt10, as shown in sequence-to-sequence methods. autoencoders and variational autoencoders have been investigated lately for relation extraction, primarily for detection of relations between entity mentions in sentences. marcheggiani and titov (2016) proposed discrete-state vaes for link prediction, reconstructing one of the two entities of a pair at a time. ma et al. (2019) investigated conditional vaes for sentence-level relation extraction, showing that they can generate relation-specific sentences. our overall approach shares similarities with this work since we also use vaes for re, though in a bag rather than a sentence-level setting. vaes have also been investigated for re in the biomedical domain (zhang and lu, 2019), where additional non-labelled examples were incorporated to assist classification. this work also has commonalities with our work but the major difference is that the former uses two different encoders while we use only one, shared among bag classification and bag reconstruction. other seq2seq methods treat re as a sequence generation task. encoder-decoder networks were proposed for joint extraction of entities and relations ( @xcite , generation of triples from sequences @xcite or generation of sequences from triples @xcite . ## conclusions we proposed a probabilistic approach for distantly supervised relation extraction, which incorporates context agnostic knowledge base triples information as latent signals into context aware bag-level entity pairs. our method is based on a variational autoencoder that is trained jointly with a relation classifier. kb information via a link prediction model is used in the form of prior distributions on the vae for each pair. the proposed approach brings close sentences that contain the same kb pairs and it does not require any external information during inference time. experimental results suggest that jointly reconstructing sentences with relation classification is helpful for distantly supervised re and kb priors further boost performance. analysis of the generated latent representations showed that we can indeed manipulate the space of sentences to match the space of kb triples, while reconstruction is enforced to keep topic-related terms. future work will target experimentation with different link prediction models and handling of noninformative sentences. finally, incorporating large pre-trained language models (lms) into vaes is a recent and promising study @xcite which can be combined with kbs as injecting such information into lms has been shown to further improve their performance @xcite . yujin yuan, liyuan liu, siliang tang, zhongfei zhang, yueting zhuang, shiliang pu, fei wu, and xiang ren. 2019. cross-relation cross-bag attention for distantly-supervised relation extraction. in proceedings of the aaai conference on artificial intelligence, volume 33, pages 419-426. daojian zeng, kang liu, yubo chen, and jun zhao. 2015. distant supervision for relation extraction via piecewise convolutional neural networks. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 1753-1762, lisbon, portugal. association for computational linguistics. ningyu zhang, shumin deng, zhanlin sun, guanying wang, xi chen, wei zhang, and huajun chen. 2019. long-tail relation extraction via knowledge graph embeddings and graph convolution networks. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3016-3025, minneapolis, minnesota. association for computational linguistics. yijia zhang and zhiyong lu. 2019. exploring semisupervised variational autoencoders for biomedical relation extraction. methods, 166:112 -119. deep learning in bioinformatics. da zheng, xiang song, chao ma, zeyuan tan, zihao ye, jin dong, hao xiong, zheng zhang, and george karypis. 2020. dgl-ke: training knowledge graph embeddings at scale. in proceedings of the 43rd international acm sigir conference on research and development in information retrieval, sigir '20, page 739-748, new york, ny, usa. association for computing machinery. yaoming zhu, juncheng wan, zhiming zhou, liheng chen, lin qiu, weinan zhang, xin jiang, and yong yu. 2019. triple-to-text: converting rdf triples into high-quality natural languages via optimizing an inverse kl divergence. in proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 455-464.