Entity or Relation Embeddings? An Analysis of Encoding Strategies for Relation Extraction (2024)

Abstract: Existing approaches to relation extraction obtain relation embeddings by concatenating embeddings of the head and tail entities. Despite the popularity of this approach, we find that such representations mostly capture the types of the entities involved, leading to false positives and confusion between relations that involve entities of the same type. Another possibility is to use a prompt with a [MASK] token to directly learn relation embeddings, but this approach tends to perform poorly. We show that this underperformance comes from the fact that information about entity types is insufficiently captured by the [MASK] embeddings. We therefore propose a simple model, which combines such [MASK] embeddings with entity embeddings. Despite its simplicity, our model consistently outperforms the state-of-the-art across several benchmarks, even when the entity embeddings are obtained from a pre-trained entity typing model. We also experiment with a self-supervised pre-training strategy which further improves the results.

## introduction relation extraction consists in identifying the relationships between entities that are expressed in text. it is a fundamental natural language processing (nlp) task, which enables the learning of symbolic representations such as knowledge graphs. while large language lodels (llms) are highly effective in interpreting natural language inputs, the stateof-the-art in relation extraction is still based on fine-tuned language models of the bert family @xcite . moreover, relation extraction is often applied to large document collections, such as corpora of news stories, scientific articles or social media, which means that relying on llms is often not feasible in practice, due to their high cost. the study of how smaller lms can be most effectively used for this task thus remains important. in high-level terms, the standard approach is to fine-tune a bert-based language model to learn a relation embedding, i.e. a vector representation of the relationship that is expressed between two entities in a given sentence, and to train a classifier to predict a discrete relation label from this embedding. to obtain these relation embeddings, we cannot simply train a sentence embedding model, since a single sentence may express several relationships. moreover, we cannot easily identify which tokens in the sentence express the relationship, which makes learning relation embeddings fundamentally different from learning embeddings of text spans. in their seminal work, baldini @xcite proposed to encapsulate the head and tail entity with special tokens. consider the following example, where we are interested in extracting the relationship between paris and france: the olympics will take place in [e1] paris [/e1], the capital of [e2] france [/e2]. the corresponding relation embedding is obtained by concatenating the final-layer embedding of the special tokens @xcite . this strategy was found to outperform other alternatives by @xcite , and has remained the most popular approach for learning relation embeddings the success of this strategy is somewhat surprising: [e1] and [e2] are intuitively designed to represent the head and tail entities, rather than their relationship even though their contextualised representations may still capture the relational context to some extent. in practice, however, knowing the semantic types of the entities often allows us to "guess" the relationship between them, especially if the types are fine-grained. for instance, knowing that the head entity is a capital city and the tail entity is a country, we may reasonably assume that the relationship being expressed is capital of. however, as our analysis in this paper shows, such approaches have at least two important limitations. first, they often struggle to distinguish relations between entities of the same type. second, they sometimes lead to false positives (e.g. incorrectly assuming that a sentence mentioning a country and a capital city expresses the capital-of relationship). a possible alternative is to add a prompt which includes the @xcite token, e.g.: the olympics will take place in paris, the capital of france. the relation between paris and france is @xcite . we can then fine-tune the language model (lm) such that the contextualised embedding of @xcite corresponds to a relation embedding. this strategy is popular for zero-shot and few-shot relation extraction @xcite , but is not normally used for the standard supervised relation extraction setting. our experiments indeed confirm that this approach performs poorly when used alone. crucially, we find that this strategy struggles because it is not able to accurately characterise the semantic types of the entities, which makes relation prediction much harder. this approach is thus complementary to the entity embedding based strategy: in one case, the model nearly entirely concentrates on entity types, whereas in the other case, it mostly overlooks them. exploiting this fact, we show that by combining both approaches, we arrive at a simple but highly effective strategy for relation extraction which improves the state-of-the-art in several relation extraction benchmarks. our main contributions can be summarised as follows: learning relation embeddings the standard approach for relation extraction with lms uses special tokens to indicate the head and tail entities (also known as the subject and object). such approaches then predict relation labels from the contextualised representations of these two entities. for instance, the matching-the-blanks model (baldini soares et al., 2019) encapsulates the head entity using the special tokens [e1]...[/e1] and the tail entity using separate special tokens [e2]...[/e2]. luke (yamada et al., 2020) simply replaces the entities by the special tokens [head] and [tail], omitting the actual entity spans from the input. wang et al. (2021) encapsulate the head and tail entity with the markers @...@ and #...#, thus avoiding the introduction of new tokens. some approaches use typed markers, which encode the semantic type of the entity, either as special tokens zhong and chen (2021) or by verbalising the entity type as part of the input. it should be noted, however, that entity types are typically not available in practice, which limits the applicability of such approaches. another possibility is to append the input with a prompt containing the [mask] token. this strategy is popular for zero-shot and few-shot relation extraction @xcite @xcite . rather than training a classifier on top of a relation embedding, the aim is then to compare the contextualised representation of the mask token with verbalisers, i.e. tokens from the lm's vocabulary that describe the relationship. this strategy has rarely been considered in the fully supervised setting, with knowprompt (chen et al., 2022) being a notable exception. zhong and chen (2021) already discussed the idea that representing entity types is not sufficient for relation classification. they highlight in particular that jointly training an entity typing and relation extraction system hurts performance, suggesting that both tasks need different kinds of latent representations. we take this idea further, based on the hypothesis that representing relations by concatenating the embeddings of the head and tail entity is inherently limited, even if the entity encoders are specifically trained for relation extraction. pre-training relation encoders several approaches have been proposed for pre-training or adapting language models to make them more suitable for the task of relation extraction. the matching-the-blanks model @xcite uses entity linking to find sentences that refer to the same entities, and then pre-trains a relation encoder based on the idea that sentences mentioning the same entity pairs are likely to express the same relationship. more recently, variants of this approach based on distant supervision have also been considered. two sentences are then assumed to have the same relation if their entity pairs are asserted to have the same relation in some knowledge base, a strategy that has a long tradition in relation extraction @xcite . for instance, @xcite implement this strategy using contrastive learning with the infonce loss. while the aforementioned approaches are focused on finetuning a pre-trained lm, the idea of changing the lm model itself has also been explored. for instance, spanbert @xcite changes the standard masked token prediction task with the aim of learning better span-level representations, while luke @xcite uses entity linking and distinguished entity tokens to improve the representation of entities. relation extraction with llms llms such as chatgpt perform surprisingly poorly on relation extraction benchmarks, and information extraction tasks more generally @xcite . @xcite discuss some of the challenges involved in using llms for such tasks, which include the difficulty in selecting suitable in-context demonstrations. @xcite make the observation that llms with in-context learning (icl) struggle in particular on specification-heavy tasks, i.e. tasks where even human annotators need to carefully study a non-trivial set of annotation guidelines to correctly solve the task, as is often the case in information extraction. due to the challenges of using icl for relation extraction, most approaches involving llms use models that can be fine-tuned. for instance, @xcite proposed a reformulation of relation extraction as a natural language inference (nli) problem and included experiments with the 1.5b parameter deberta xxl model @xcite . while this allowed them to improve the state-of-the-art at the time, it should be noted that the nli based formulation is highly inefficient when a large number of relation labels need to be considered. it also relies on manually defined verbalisations of the relation labels. @xcite fine-tuned a 10b parameter model on a range of tasks that can be formulated as triple prediction, including relation extraction. @xcite found that while gpt-3 (brown et al., 2020) performed poorly when used directly, it was useful for generating chain-of-thought @xcite explanations. in particular, they showed that by fine-tuning a flan-t5 (chung et al., 2022) model on these explanations, the resulting model performed substantially better than when fine-tuning flan-t5 on the relation extraction task directly. as another strategy for leveraging llms indirectly, @xcite use chatgpt for data generation in few-shot settings. overall, however, the state-of-the-art in relation extraction, for the fully supervised setting, is still based on fine-tuned models of the bert family @xcite . while this might change in future, e.g. with novel prompting techniques or better models, the need for efficient information extraction models means that such smaller models are likely to remain important. ## relation extraction we consider the standard sentence-level relation extraction setting, where we are given a sentence in which two entities are highlighted, which we refer to as the head entity and tail entity. the goal is to predict which relationship holds between these two entities, given a pre-defined set of candidate relation labels. we focus on strategies that first learn a relation embedding, which describes the relationship between the two entities in a continuous space, and then use a classifier to predict the actual label based on that relation embedding. in section 3.1, we first explain the pre-training strategies that we use for learning high-quality relation embeddings. section 3.2 then describes how the pre-trained relation encoder is fine-tuned for the relation classification task. finally, in section 3.3 we explain how relation embeddings can be obtained by concatenating the contextualised embeddings of the head and tail entities or by using a prompt-based strategy with a [mask] token, among others. ## experiments we compare the effectiveness of the considered relation embedding strategies on a number of standard relation extraction benchmarks. our main hypothesis is that the common [h,t] strategy essentially leads models to focus on the semantic types of the head and tail entity. we are thus interested in comparing the [h,t] and encore strategies. furthermore, we hypothesise that the information captured by the mask embeddings is complementary to that captured by the [h,t] embeddings. accordingly, we are interested in comparing [h,t] with [h,t,mask] and [h,t]+mask. ## conclusions the primary aim of this paper was to analyse two different strategies for training relation encoders. on the one hand, most work in supervised relation extraction relies on contextualised embeddings of the head and tail entity for predicting relationships. on the other hand, prompt-based strategies can be used to obtain embeddings that represent the relationship itself. the latter strategy is arguably more intuitive, but we found it to perform poorly in practice. rather than suggesting that such relation embeddings are not useful, however, we found that they capture information that is highly complementary to what is captured by contextualised entity embeddings. indeed, we considered a hybrid strategy, which substantially outperforms either of the two individual strategies, allowing us to improve the state of the art in each of the five considered benchmarks. remarkably, we found that this remains true if we use entity embeddings from an off-the-shelf entity encoder. finally, as a secondary contribution, we also found that coreference chains, which were used for training entity encoders in a self-supervised way by @xcite , can be successfully leveraged for self-supervised training of relation encoders.