Unsupervised Relation Extraction: A Variational Autoencoder Approach (2021)

Abstract: Unsupervised relation extraction works by clustering entity pairs that have the same relations in the text. Some existing variational autoencoder (VAE)-based approaches train the relation extraction model as an encoder that generates relation classifications. A decoder is trained along with the encoder to reconstruct the encoder input based on the encoder-generated relation classifications. These classifications are a latent variable so they are required to follow a pre-defined prior distribution which results in unstable training. We propose a VAE-based unsupervised relation extraction technique that overcomes this limitation by using the classifications as an intermediate variable instead of a latent variable. Specifically, classifications are conditioned on sentence input, while the latent variable is conditioned on both the classifications and the sentence input. This allows our model to connect the decoder with the encoder without putting restrictions on the classification distribution; which improves training stability. Our approach is evaluated on the NYT dataset and outperforms state-of-the-art methods.

## introduction relation extraction (re) methods aim to classify relations of entity pairs in a given sentence. re is the basis of many nlp tasks, such as information extraction @xcite , knowledge graph construction @xcite and information retrieval @xcite . conventional re models classify relations based on preset rules @xcite . these rules include syntactic patterns such as dependency structures and part of speech (pos). although these rule-based methods achieve high accuracy, the rules can only be applied to limited types of sentences, which leads to application limitations @xcite . recently, with the rise of deep learning, some work introduced deep neural networks, such as piece-wise convolutional neural network (pcnn) @xcite , graph convolutional networks @xcite , and graph lstm @xcite . these approaches perform well. however, they use supervised model training which requires large-scale labeled data which is not always available. to reduce the reliance on labeled data, some work propose unsupervised relation extraction methods. traditional unsupervised re approaches compose vectors based on extracted sentence features. these feature vectors represent the relation information of sentences @xcite . these methods then cluster sentences according to these feature vectors to classify sentences into relations. even though these approaches achieve state-of-the-art performance, they heavily rely on hand-craft features and make many simplifying assumptions as mentioned in previous work @xcite . other unsupervised re techniques follow a discriminative approach to train more expressive models. these techniques can provide a sufficient learning signal to train deep neural networks as relation classifiers. for example, some work achieved a discriminative approach using a variational autoencoder (vae)based method @xcite . the main idea of this approach is to train an encoder as a relation classifier, which converts input sentences into relation classifications. a decoder then reconstructs entity pairs (head_entity, tail_entity) of input sentences according to the classifications. the decoder's reconstruction loss provides the encoder with a supervision signal so the encoder can be implemented as a deep neural network. while this approach trains the relation classifier with supervision, the model has been shown to be unstable when tested on the new york times-freebase (nyt-fb) dataset @xcite . specifically, the model always results in one of two training results depending on the hyper-parameter setting: the model either predicts the same relation for all input sentences, or the prediction probability of each relation is always the same @xcite . to improve model stability, recent work proposed adding two regularization losses to the original objective function @xcite . adding these two extra losses skews the prediction probability distribution of the relation of one sentence while pushing the relation distribution of a set of sentences to be uniform. however, during our experimental implementation of this model, we note that even with adding the regularization terms, there is still an unresolved problem that may negatively impact performance. the key problem of this vae-based model is that the relation classification result r is a latent variable. this leads to the following two limitations. 1) the decoder can not follow the traditional vae training process of directly sampling from r as it would result in interrupting the gradient passing. therefore, the decoder is not directly connected to the encoder and can not update it in a timely manner. 2) the additional regularization loss terms serve the role of the kl divergence term in the vae loss function. this requires the classification to follow a pre-defined prior distribution, namely, a uniform distribution. these two limitations may lead to unbalanced vae training of the relation extraction encoder model and decoder. consequently, the vae loss converges, while the relation classification accuracy is not relatively high. we propose a variational autoencoder-based unsupervised relation extraction model (ureva) which overcomes the aforementioned limitations. in particular, our proposed approach treats the relation classification r as an intermediate variable conditioned on the input sentence x, while the latent variable z is conditioned on the joint distribution of x and r. the decoder uses r and data sampled from z to reconstruct entity pair in the input sentence. through the sampling process, the decoder and the latent space establish a connection. this addresses limitation 1 mentioned above. in addition, since r is no longer a latent variable, fitting the latent variable to any prior distribution does not affect the classification results. this overcomes limitation 2. we run experiments on the nyt-fb @xcite and semeval @xcite datasets to compare the performance of our proposed method against other state-of-the-art methods. experimental results show that our model generates more accurate classification results than state-of-the-art methods on both datasets. our key contributions can be summarized as follows: ## variational autoencoder background in this section, we review the variational autoencoder (vae) architecture and its objective function. vae is a probabilistic generative model that describes an observation in latent space. the goal of the vae is to train a decoder via the joint distribution p φ (x, z) = p φ (z)p φ (x|z), where p φ (z) is a prior distribution of latent variable z and the posterior distribution p φ (x|z) is the decoder that generates x given z. in general, since computing the true posterior distribution p φ (z|x) is intractable, an encoder q θ (z|x) is used to approximate it. the objective of the vae is to minimize the kl divergence of the encoder q θ (z|x) and the decoder p φ (z|x) such that the two distributions are similar. expanding p φ (x), it can be expressed as follows: the goal of our model is to extract a relation between an entity pair in a given sentence. the open question is how to train a relation extraction model without labeled data. in order to provide a supervision signal to the relation extractor, the initial idea is to use an encoder-decoder architecture. in this architecture, the relation extractor is an encoder that outputs relation classifications. in addition, a decoder reconstructs the encoder input, i.e., the sentence, given the classifications. the two models are jointly trained and the only way to reduce reconstruction loss is to let the encoder compress all the relation information to the classification results. in this way, the classifications become the bottleneck of the whole architecture. in practice, we choose vae instead of a general autoencoder to implement this because classifications are probabilistic. specifically, as shown in fig 1 , we divide the encoder network into two parts: the relation classifier, and the encoding layer. the relation classifier calculates the probability q θ (r|x) that a given input sentence x is classified into relation r. we let the latent variable z (representing sentence and relation information) be conditioned on x and r, which can be written as q θ (z|x, r). we use the encoding layer to model the relation probability q θ (z|x, r). the decoder network then reconstructs the probability p φ (x|r, z) of input sentence x given samples of z and relations r. finally, as shown in eq. 3, our model has the following mathematical property: note that the relation extractor is not used as the entire encoder. we explain this design choice in sec 3.5. ## related work unsupervised re uses unlabeled data to train a model that clusters sentences with the same relation information together. initially, some work proposed using some linguistic patterns extracted from sentences as the contexts of entity pairs @xcite . once these contexts are vectorized, the model can cluster sentences with similar contexts. follow-up work improved model performance by reducing the frequency of noisy words @xcite . inspired by this approach, some work combined linguistic patterns and surface context to design a two-step clustering algorithm @xcite . some methods deal with unsupervised re from the perspective of probabilistic generative models. rel-lda introduced the latent dirichlet allocation (lda) approach to unsupervised re @xcite . in this method, the topic distribution is replaced by the relation distribution and expectation maximization algorithm is applied to cluster similar relations. some work extended this model for general domain knowledge (de lacalle and lapata, 2013). they applied first order logic rules to extract patterns from knowledge base such that the model can be enhanced by using patterns as the prior knowledge. some work used the idea of autoencoder to provide supervised signal to re. the cure used an encoder to extract relation information from sentences @xcite . the decoder reconstructs the dependency path between the entity pairs based on the relation information. after training, the relation information output by the encoder is used to cluster sentences with similar relations. however, some work proposed to let the encoder output the classification result instead of an information vector @xcite . in this method, the decoder reconstructs the entity pair in the input sentence according to the probability of the encoder output. in order to allow the encoder output a variety of relations, an entropy-based regularization function is added to the original objective function. more recently, some work considered that this model is not stable @xcite . this is because the model only predicts one same relation for all input sentences, or the predicted relations probability distribution for each instance is similar. this recent work proposed that in addition to the original entropy-based regularization, dispersion loss needs to be added. this loss term requires that all relation classification results of the model conform to the uniform distribution. the pcnn architecture in this model is then replaced by entity type feature, since entity type was considered as the most important feature @xcite . ## conclusion in this paper, we present ureva, a variational autoencoder-based unsupervised relation extraction model. we consider relation classification as an intermediate variable to solve the model training instability problem that appeared in previous works. in ureva, the classification is conditioned on the input sentence, and the latent variable is conditioned on the the joint distribution of the classification and the input sentence. then the decoder reconstructs the entity pair in the input sentence by sampling from the latent space and the relation classification. experiments on two public datasets show that ureva outperforms state-ofthe-art models.