Distilling Relation Embeddings from Pretrained Language Models (2021)

Abstract: Pre-trained language models have been found to capture a surprisingly rich amount of lexical knowledge, ranging from commonsense properties of everyday concepts to detailed factual knowledge about named entities. Among others, this makes it possible to distill high-quality word vectors from pre-trained language models. However, it is currently unclear to what extent it is possible to distill relation embeddings, i.e. vectors that characterize the relationship between two words. Such relation embeddings are appealing because they can, in principle, encode relational knowledge in a more fine-grained way than is possible with knowledge graphs. To obtain relation embeddings from a pre-trained language model, we encode word pairs using a (manually or automatically generated) prompt, and we fine-tune the language model such that relationally similar word pairs yield similar output vectors. We find that the resulting relation embeddings are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. Source code to reproduce our experimental results and the model checkpoints are available in the following repository: https://github.com/asahi417/relbert

## introduction one of the most widely studied aspects of word embeddings is the fact that word vector differences capture lexical relations @xcite . while not being directly connected to downstream performance on nlp tasks, this ability of word embeddings is nonetheless important. for instance, understanding lexical relations is an important prerequisite for understanding the meaning of compound nouns @xcite . moreover, the ability of word vectors to capture semantic relations has enabled a wide range of applications beyond nlp, including flexible querying of relational databases @xcite , schema match-ing @xcite , completion and retrieval of web tables @xcite , ontology completion @xcite and information retrieval in the medical domain @xcite . more generally, relational similarity (or analogy) plays a central role in computational creativity @xcite , legal reasoning @xcite , ontology alignment @xcite and instance-based learning @xcite . given the recent success of pre-trained language models @xcite @xcite , we may wonder whether such models are able to capture lexical relations in a more faithful or fine-grained way than traditional word embeddings. however, for language models (lms), there is no direct equivalent to the word vector difference. in this paper, we therefore propose a strategy for extracting relation embeddings from pre-trained lms, i.e. vectors encoding the relationship between two words. on the one hand, this will allow us to gain a better understanding of how well lexical relations are captured by these models. on the other hand, this will also provide us with a practical method for obtaining relation embeddings in applications such as the ones mentioned above. since it is unclear how lms store relational knowledge, rather than directly extracting relation embeddings, we first fine-tune the lm, such that relation embeddings can be obtained from its output. to this end, we need a prompt, i.e. a template to convert a given word pair into a sentence, and some training data to fine-tune the model. to illustrate the process, consider the word pair paris-france. as a possible input to the model, we could use a sentence such as "the relation between paris and france @xmath0. note that our aim is to find a strategy that can be applied to any pair of words, hence the way in which the input is represented needs to be sufficiently generic. we then fine-tune the lm such that its output corresponds to a relation embedding. to this end, we use a crowdsourced dataset of relational similarity judgements that was collected in the context of semeval 2012 task 2 @xcite . despite the relatively small size of this dataset, we show that the resulting fine-tuned lm allows us to produce high-quality relation embeddings, as confirmed in our extensive evaluation in analogy and relation classification tasks. importantly, this also holds for relations that are of a different nature than those in the semeval dataset, showing that this process allows us to distill relational knowledge that is encoded in the pre-trained lm, rather than merely generalising from the examples that were used for fine-tuning. another notable work focusing on link prediction is @xcite , where gpt is fine-tuned to complete triples from commonsense knowledge graphs, in particular conceptnet @xcite and atomic @xcite . while their model was able to generate new knowledge graph triples, it is unclear to what extent this is achieved by extracting commonsense knowledge that was already captured by the pre-trained gpt model, or whether this rather comes from the ability to generalise from the training triples. for the conceptnet dataset, for instance, @xcite found that most test triples are in fact minor variations of training triples. in this paper, we also rely on fine-tuning, which makes it harder to determine to what extent the pre-trained lm already captures relational knowledge. we address this concern by including relation types in our evaluation which are different from the ones that have been used for fine-tuning. unsupervised relation discovery modelling how different words are related is a long-standing challenge in nlp. an early approach is dirt @xcite , which encodes the relation between two nouns as the dependency path connecting them. their view is that two such dependency paths are similar if the sets of word pairs with which they co-occur are similar. @xcite cluster named entity pairs based on the bag-of-words representations of the contexts in which they appear. along the same lines, @xcite proposed a generative probabilistic model, inspired by lda @xcite , in which relations are viewed as latent variables (similar to topics in lda). @xcite proposed a method called latent relational analysis (lra), which uses matrix factorization to learn relation embeddings based on co-occurrences of word pairs and dependency paths. matrix factorization is also used in the universal schema approach from riedel et al. @xcite , which jointly models the contexts in which words appear in a corpus with a given set of relational facts. the aforementioned works essentially represent the relation between two words by summarising the contexts in which these words co-occur. in recent years, a number of strategies based on distributional models have been explored that rely on similar intuitions but go beyond simple vector operations of word embeddings. 2 for instance, @xcite introduced a variant of the glove word embedding model, in which relation vectors are jointly learned with word vectors. in seven (espinosa-anke and schockaert, 2018) and relative (camacho-collados et al., 2019), relation vectors are computed by averaging the embeddings of context words, while pair2vec @xcite uses an lstm to summarise the contexts in which two given words occur, and washio and kato (2018) learn embeddings of dependency paths to encode word pairs. another line of work is based on the idea that relation embeddings should facilitate link prediction, i.e. given the first word and a relation vector, we should be able to predict the second word @xcite . this idea also lies at the basis of the approach from @xcite , who train a relation encoder by fine-tuning bert @xcite with a link prediction loss. however, it should be noted that they focus on learning relation vectors from individual sentences, as a pre-training task for applications such as few-shot relation extraction. in contrast, our focus in this paper is on characterising the overall relationship between two words. ## relbert in this section, we describe our proposed relation embedding model (relbert henceforth). to obtain a relation embedding for given a word pair (h, t), we first convert it into a sentence s, called the prompt. we then feed the prompt through the lm and average the contextualized embeddings (i.e. the output vectors) to get the relation embedding of (h, t). these steps are illustrated in figure 1 and explained in more detail in the following. ## experimental setting in this section, we explain our experimental setting to train and evaluate relbert. ## results in this section, we present our main experimental results, testing the relation embeddings learned by relbert on analogy questions (section 5.1) and relation classification (section 5.2). ## analysis to better understand how relation embeddings are learned, in this section we analyze the model's performance in more detail. ## conclusion we have proposed a strategy for learning relation embeddings, i.e. vector representations of pairs of words which capture their relationship. the main 11 we used bert-large-cased and albertxlarge-v1 from the huggingface model hub.