Relation Extraction with Contextualized Relation Embedding ( CRE ) (2020)

Abstract: This submission is a paper that proposes an architecture for the relation extraction task which integrates semantic information with knowledge base modeling in a novel manner.

## introduction relation extraction (re) is a sub-task under the broad category of information extraction (ie) that aims to identify relationship(s) between named entities based on textual information (corpus). the groundtruth relationship(s) between an entity pair can be either internal, if each sentence in the corpus is explicitly labeled, or external, in the form of relation instances in a standalone knowledge base (kb). such knowledge bases, like freebase @xcite , are collaboratively edited by human beings and thus offer high informational fidelity. in this paper, we focus on the problem of predicting the collection of relations for a new entity pair based on the usage of that entity pair in its respective collection of sentences. to this end, we assume the general construction that given a dataset with a set of named entity pairs and each entity pair's respective collection of sentences as input along with entity pair's respective collection of relations as output, our goal is to train a model with the objective of predicting collection of relevant relation(s) for a new entity pair based on its contextual usage. distant supervision @xcite is the most popular approach for this problem, that has achieved its success by leveraging knowledge base information for relation extraction tasks. however, this restricts the usage of semantic information present in knowledge bases, since the distant supervision work mainly incorporates the knowledge base information as labels instead of treating it as a graph and thereby losing the dense relationships between different entities. knowledge base modelling is an independent area of research and there has been some recent work on utilizing knowledge base modeling and incorporating internal structural information from them to the relation extraction task. @xcite , @xcite . @xcite conducted the first research work that utilized knowledge bases as a structural graph by training a transe @xcite ) kb model alongside a traditional distant supervision extractor. however, this has been a simple combination of both ideas but despite its simplicity, the model was able to beat the thenstate-of-the-art models in relation extraction tasks. the work by @xcite , on the other hand, such contextualized embeddings represent a natural link between the textual input and the knowledge base modeling objective, so that the end-to-end information transformation is completely internal to the model. the entity embeddings, on the other hand, are not contextualized and will still be trained as parameters of the same model, because they need to serve as bridges across different entity pairs and globalize the knowledge base modeling which is now based on contextualized relation embeddings localized to each entity pair. a significant innovation of this work is that each sentence in which a pair of entities occurs, we represent each possible relation between them as a function of that entire sentence and score the relation for that entity pair and sentence using the representation of the relation. ## contextualized relation embedding let r stand for the set of all relations and e stand for set of all entities. then, for the i-th entity pair, h i and t i stand for the subject and object respectively, both belonging to e. r i ⊂ r stand for the set of relations existing between the i-th entity pair. s i stands for the collection of sentences that contain h i as the subject and t i as the object; s jk i stands for the k-th word of the j-th sentence in s i , where each s jk i is delimited by spaces except for the subject and the object, whose phrasal integrities are preserved (e.g. if it is the subject/object of a sentence, the united states will be considered as a single "word"). we propose a novel neural network model (called cre model) that implicitly combines knowledge base to detect relationships between entities, given a corpus of sentences containing those entities. the overall architecture of a cre model is shown in figure 1 , which consists of a sentence encoding step and a knowledge base modeling step. during the sentence encoding step, the model transforms each s jk i into e-dimensional embeddings. specifically, we apply pretrained skipgram embeddings @xcite , treating all named entities as unknown, which results in a w -dimensional embedding for each word. we apply a positional embedding based on zeng et al. 2014 , where each word's index in the positional "vocabulary" is defined as its relative distance to one of the named entities. consider the sentence "a bear entered the fridge" and named entities "bear" and "fridge". the word "a" will have -1 as its positional embedding index with respect to "bear" and -4 with respect to "fridge", whereas "entered" will have +1 with respect to "bear" and -2 with respect to "fridge". for each word (including the two named entities), we represent it as its skip-gram word embedding of dimension w , concatenated with its positional embeddings (each of dimension p ) with respect to the two named entities, to form a final embedding of dimension e, where @xmath0. the skip-gram embeddings are fixed during training, whereas the positional embeddings are learnt. this leads to a sentence embedding of l * e, where l is the maximum length of each sentence. we utilize the above mentioned sentence embeddings and learn an encoder that encodes each sentence into a r * k embedding, which can be represented as |r| k-dimensional embeddings, each of which corresponds to a distinct relation in r. figure 2 illustrates the sentence encoding process for a single sentence. it is worth noting that the sentence encoder is a generic framework where we can plug in any model architecture (both for positional encoding and sentence encoding). in our experiments, we found that neural network based encoders tend to result in better performance. we think this is due to their ability to extract both semantic and syntactic information. we denote sentence j's representation of r's relation for entity pair i as cre jr i . in order to provide the model more context, we explicitly apply a k-dimensional embedding for each part of the entity pair represented by h i and t i respectively. these entity embeddings are learnt as parameters during training, similar to the positional embeddings above. we utilize the relation embeddings (contextualized relation embeddings) from the sentences and these entity embeddings to form relation triplet embeddings. specifically, this will result in r triplets for each sentence, since we computed r relation embeddings for each sentence. this allows us to apply different knowledge based models and further enhances the generalization of our model. depending on the choice of knowledge based model, we use the relation triplet embeddings to compute its score based on the scoring function of the respective knowledge base model. for example, if transe @xcite is chosen as the underlying knowledge base model with h i and t i as head and tail entity embeddings, then embeddings for relation r derived from sentence j will be scored as: since most knowledge base models are characterized by their scoring functions, it is straightforward to swap in any knowledge base model here for scoring purposes. finally, the model aggregates scores for the same relation across all sentences so that we can obtain a single score for each relation given an entity pair: for experiments in this paper, we adopted summation as the aggregation function since we do not have prior knowledge about our dataset and thus want to take into account the "opinion" of all sentences equally, but it is by no means prescriptive. one can certainly use maximum, minimum or mean as the aggregation functions instead of sum in case the dataset is such that any single sentence's outcome can be regarded as reliable. we train the model with a binary cross entropy loss, after normalizing relation scores and targets and we use an l2 regularization that helps prevent overfitting. we use the normalized scores to predict the relations for the given entity pair. note that normalization is still necessary for quantitative evaluation. we pool together scores for different entity pairs to form a precision-recall curve, so scores need to be comparable and normalization brings them to the same scale. the top-k prediction can formulated as: ## discussion our experiments from figure 3 clearly demonstrate that models utilizing contextualized relation embeddings that internalize both relation extractor modeling and knowledge base modeling tend to perform much better on relation extraction task than architectures like @xcite and @xcite that join a relation extractor and a knowledge base model in some arbitrary way. we believe this is due to the fact that the "internal knowledge base models" within cre models are context-aware. as a result, cre models can take advantage of the contextual information contained in the corpus more effectively. we demonstrate the generality and flexibility of the proposed cre model that can work with different encoders and knowledge base models. moreover, it can be observed from it can be observed from figure 4 that more complex configurations for our model tend to achieve stellar results in the low recall arena but lose steam when recall levels are high. this may be explained by the insufficient training epochs due to resource constraints. as a result, these complex models did not get the chance to optimize for the less seen prediction targets. however, it can been seen from ## conclusion and future work in this paper, we introduced a novel contextualized relation embedding (cre) model for relation extraction that incorporates knowledge base modeling in a comprehensive and efficient manner. we demonstrate both empirically and qualitatively that the cre model is able to achieve state of the art results on relation extraction task on the new york times dataset with freebase as a knowledge