Multi-facet Universal Schema (2021)

Abstract: Universal schema (USchema) assumes that two sentence patterns that share the same entity pairs are similar to each other. This assumption is widely adopted for solving various types of relation extraction (RE) tasks. Nevertheless, each sentence pattern could contain multiple facets, and not every facet is similar to all the facets of another sentence pattern co-occurring with the same entity pair. To address the violation of the USchema assumption, we propose multi-facet universal schema that uses a neural model to represent each sentence pattern as multiple facet embeddings and encourage one of these facet embeddings to be close to that of another sentence pattern if they co-occur with the same entity pair. In our experiments, we demonstrate that multi-facet embeddings significantly outperform their single-facet embedding counterpart, compositional universal schema (CUSchema) (Verga et al., 2016), in distantly supervised relation extraction tasks. Moreover, we can also use multiple embeddings to detect the entailment relation between two sentence patterns when no manual label is available.

## introduction relation extraction (re) is a crucial step in automatic knowledge base construction (akbc). a major challenge of re is that the frequency of relations in the real world is a long-tail distribution but collecting sufficient human annotations for every relation is infeasible @xcite . distant supervision is proposed to alleviate the issue @xcite . distant supervision assumes that a sentence pattern expresses a relation if the sentence pattern co-occurs with an entity pair and the entity pair has the relation. for example, we assume the sentence pattern "@xmath0arg2" is likely to express the spouse baseline: compositional universal schema close close neural encoder and decoder neural encoder and decoder pattern embedding 5 entity pair embedding close close multi-facet universal schema table lookup pattern embedding 4 pattern embedding 3 pattern embedding 2 pattern embedding 1 pattern embedding 5 pattern embedding 4 pattern embedding 3 pattern embedding 2 pattern embedding 1 entity pair embedding pattern embedding pattern embedding co-occur sentence pattern @xmath1arg2 sentence pattern @xmath2arg2 entity pair @@xmath12: comparison between the multi-facet and compositional universal schema. in our training loss, we encourage one of the facet embeddings from a sentence pattern to be similar to its co-occurred entity pair. universal schema @xcite extends this assumption by treating every sentence pattern as a relation, which means we assume that sentence patterns or relations in a knowledge base are similar if they co-occur with the same entity pair. for example, we assume "@xmath4arg2" and "@xmath5arg2" are similar if they both co-occur with @xcite . consequently, we can infer that "@xmath6arg2" also implies spouse relation as "@xmath7arg2" even if the knowledge base does not record the spouse relation between kristen bell and dax shepard. compositional universal schema @xcite realizes the idea by using a lstm @xcite to encode each sentence pattern into an embedding and encouraging the embedding to be similar to the embedding of the co-occurred entity pair. as in the lower part of figure 1 , the model makes the embeddings of two sentence patterns similar if they co-occur with the same entity pair. baldini @xcite rely on a similar assumption and achieve state-of-theart results on supervised re tasks by replacing the lstm with a large pre-trained language model. the variants of universal schema have many different applications, including multilingual re @xcite , knowledge base construction @xcite , question answering @xcite , documentlevel re @xcite , n-ary re @xcite , open information extraction @xcite , and unsupervised relation discovery @xcite . nevertheless, one sentence pattern could contain multiple facets, and each facet could imply a different relation. in figure 1 , "@xmath8arg2" could imply the entity pair has the spouse relation, the co-worker relation, or both. "@xmath9arg2" could imply the spouse relation, the parent relation, ..., etc. if we squeeze the facets of a sentence pattern into a single embedding, the embedding is more likely to be affected by the irrelevant facets from other patterns co-occurred with the same entity pair (e.g., "@xmath10arg2" might incorrectly imply the co-worker relation). another limitation is that single embedding representation can only provide symmetric similarity measurement between two sentence patterns. thus, an open research challenge is to predict the entailment direction of two sentence patterns only based on their co-occurring entity pair information. to overcome the challenges, we propose multifacet universal schema, where we assume that two sentence patterns share a similar facet if they cooccur with the same entity pair. as in figure 1 , we use a neural encoder and decoder to predict multiple facet embeddings of each sentence pattern and encourage one of the facet embeddings to be similar to the entity pair embedding. as a result, the facets that are irrelevant to the relation between the entity pairs are less likely to affect the embeddings of entity pairs and other related sentence patterns. for example, the parent facet of "@xmath11arg2" could be excluded when updating the embeddings of (angelina jolie, brad pitt). in our experiments, we first compare the multifacet embeddings with the single-facet embedding in distantly supervised re tasks. the results demonstrate that multiple facet embeddings significantly improve the similarity measurement between the sentence patterns and knowledge base relations. besides re, we also apply multi-facet embeddings to unsupervised entailment detection tasks. in a newly collected dataset, we show that multi-facet universal schema significantly outperforms the other unsupervised baselines. ## methods our method is illustrated in figure 2 . in section 2.1, we first provide our problem setup: we are given a knowledge base (kb) and a text corpus during training. our goal is to extract relations by measuring the similarity between kb relations and an (unseen) sentence pattern or to detect entailment between two sentence patterns. in section 2.2, we introduce our neural model, which predicts multifacet embeddings of each sentence pattern. next, in section 2.3, we describe our objective function, which encourages the embeddings of co-occurred entity pairs to be close to the embeddings of their closest pattern facets. finally, in section 2.4, we explain that multi-facet embeddings could be viewed as the cluster centers of possibly co-occurred entity pairs, and in section 2.5, we provide our scoring functions for distantly supervised re and unsupervised entailment tasks. ## experiments we primarily compare our method with compositional universal schema (cuschema) @xcite because cuschema is one of the state-of-theart re methods in the small model regime (without using large pre-trained language models) @xcite . 1 in section 3.1, we visualize and analyze the facet embeddings. next, we use distant-supervised re tasks to evaluate our symmetric similarity measurement in section 3.2, and detect entailment between sentence patterns to evaluate our asymmetric similarity measurement in section 3.3. ## related work relation extraction (re) is widely studied. @xcite summarize the trend of recent studies and point out one of the major challenges is the cost of collecting the labels. distant supervision @xcite and its follow-up work enable us to collect a large amount of training data at a low cost, but the violation of its assumptions often introduces substantial noise into the supervision signal. our goal is to alleviate the noise issue by representing every sentence pattern using multiple embeddings. other noise reduction methods have also been proposed @xcite . for instance, we can adopt multi-instance learning techniques @xcite @xcite , global topic model @xcite , or both @xcite . we can also reduce the noise by counting the number of shared entity pairs between a sentence pattern and a kb relation @xcite . nevertheless, the studies focus on mitigating the noise caused by assuming similarity between the sentence patterns and kb relations that co-occur with the same entity pairs, while our method can also reduce the noise from two sentence patterns sharing the same entity pair. besides, our method is complementary to popular noise reduction methods because our improvement is shown in the training data that have been cleaned @xcite . our method is conceptually related to some studies for lexical semantics. for example, word sense induction or unsupervised hypernymy detection can be addressed by clustering the co-occurring words @xcite @xcite . however, the clustering-based methods do not apply to re because the co-occurring matrix for re is much sparser (see section 2.4 for more details). finally, our work is inspired by @xcite , but they focus on improving the sentence representation rather than re. we encourage the facet embeddings to become the centers in kmeans clustering instead of nnsc (non-negative sparse coding) clustering used in @xcite , due to its simplicity, efficiency, and better re performance. moreover, we discover that an additional regularization described in the appendixis crucial to overcome the sparsity challenge in re. ## conclusion in this work, we address the limitation of representing each sentence pattern using only a single embedding, and our approach improves the distantlysupervised re performances of compositional universal schema. relying on only a very sparse co-occurrence matrix between the sentence patterns and entity pairs, we show that it is possible to predict reasonable cluster centers of entity pair embeddings and to predict the entailment relation between two sentence patterns without any labels.