Solving Hard Analogy Questions with Relation Embedding Chains (2023)

Abstract: Modelling how concepts are related is a central topic in Lexical Semantics. A common strategy is to rely on knowledge graphs (KGs) such as ConceptNet, and to model the relation between two concepts as a set of paths. However, KGs are limited to a fixed set of relation types, and they are incomplete and often noisy. Another strategy is to distill relation embeddings from a fine-tuned language model. However, this is less suitable for words that are only indirectly related and it does not readily allow us to incorporate structured domain knowledge. In this paper, we aim to combine the best of both worlds. We model relations as paths but associate their edges with relation embeddings. The paths are obtained by first identifying suitable intermediate words and then selecting those words for which informative relation embeddings can be obtained. We empirically show that our proposed representations are useful for solving hard analogy questions.

## introduction many applications rely on knowledge graphs (kgs) to model the relationship between concepts. for instance, kgs have been used to characterise how an answer candidate is related to the concepts that appear in a question @xcite and to help interpret visual scenes @xcite . in such applications, relations are modelled as kg paths, which has two key advantages: (i) we can easily inject domain-specific knowledge and (ii) the interpretable nature of kg paths offers a degree of transparency. but kgs also have important drawbacks. they use a fixed set of relation types, which may not be fine-grained enough. kgs are also inevitably incomplete and, especially in the case of commonsense kgs such as conceptnet @xcite , noisy. there is also a tradition in natural language processing (nlp) to model relations as embeddings. for instance, dirt @xcite and lra @xcite are early examples of methods which used vectors to represent the relation between two words. more recently, relation embeddings have been obtained using fine-tuned language models. for instance, @xcite has achieved state-of-the-art results on several analogy datasets using a fine-tuned robertalarge model @xcite . the strengths and weaknesses of relation embeddings are complementary to those of kgs: relation embeddings lack transparency and cannot easily incorporate external knowledge, but they are capable of capturing subtle differences, making it possible to encode relational knowledge in a way that is considerably richer than what can be achieved with kgs. a final drawback of relation embeddings is that they are best suited for concept pairs that have a clear and direct relationship. for instance, they can encode the relation between umbrella and rain, but are less suitable for encoding the relation between umbrella and cloudy (e.g. if it is cloudy, there is a chance of rain, which means that we might take an umbrella). in this paper, we propose a hybrid approach which combines the advantages of kgs with those of relation embeddings. the idea is conceptually straightforward. we represent relations as paths, where nodes correspond to concepts, but rather than associating edges with discrete types, we label them with relation embeddings. we will refer to such paths as relation embedding chains. clearly, this approach allows us to model indirect relationships, while keeping the flexibility of embeddings, as well as some of the interpretability of kg paths. we are still faced with the challenge of selecting suitable paths. kgs are insufficient for this purpose, given their noisy and incomplete nature. our solution relies on the following idea: to decide whether @xmath0, what matters is whether we have access to an informative relation embedding for the word pairs (a, x) and (x, b). motivated by this, we first develop a classifier to predict whether a given relbert embedding is informative or not. we then generate possible relational paths, using conceptnet as well as standard word embeddings, and filter these paths based on the informativeness classifier. while relation embedding chains are expressive, we sometimes need a simpler representation, especially in unsupervised settings. we therefore also study how the information captured by relation embedding chains can be summarised as a single vector, without relying on task-specific supervision. to evaluate the usefulness of relation embedding chains, we focus on word analogy questions. given a query word pair (e.g. word:language) and a set of candidate word pairs, the task is to select the most analogous candidate (e.g. note,music). we show that relation embedding chains are well-suited for answering hard analogy questions. ## related work modelling analogies has been a long-standing topic of interest in nlp @xcite . recent years have seen a significant increase in interest in this topic, fuelled by the success of large language models (llms). for instance, @xcite used llms for generating explanations of scientific concepts involving analogies, while sultan and shahaf (2022) used llms for identifying analogies between procedural texts. however, most work has focused on modelling analogies between word pairs @xcite @xcite @xcite , which is also the setting we consider in this paper. we focus in particular on @xcite , which is the state-of-the-art on several benchmarks. relbert is a roberta-large model that was fine-tuned on the relational similarity dataset from semeval 2012 task 2 @xcite . given a word pair, relbert computes a relation embedding by feeding that word pair as input to the fine-tuned roberta model and averaging the output embeddings. the use of kg paths for modelling relations between concepts has also been extensively studied. for instance, @xcite proposed the use of conceptnet paths for explaining why two word pairs are analogous. @xcite highlighted the noisy nature of many con-ceptnet paths. to address this issue, they trained a system to predict path quality based on crowdsourced judgments of naturalness. more recently, conceptnet paths have been used to provide external knowledge to nlp systems, for instance in question answering systems @xcite @xcite @xcite . in such cases, a graph neural network is typically used to process the paths, and the resulting representations are then integrated with those obtained by a language model. an important distinction with our work is that we focus on unsupervised settings. kg paths are especially helpful for question answering over kgs. however, most approaches rely on matching chains of discrete relation types @xcite . chains of relation embeddings, as we study in this paper, make it possible to model finer relationships. ## scoring relbert embeddings given a word pair (a, b), we write r @xmath1. while relbert can provide a relation embedding for any pair of words, not all these embeddings are equally informative. in particular, we would like to distinguish embeddings r ab which encode a specific relationship from embeddings that rather reflect the lack of any (known) relationship between a and b. unfortunately, the relation embeddings of different unrelated word pairs are typically not similar. to address this, we want to train a classifier to predict whether a given relation embedding r ab is informative or not. however, this requires access to a set of related word pairs pos and a set of unrelated word pairs neg. the set neg can simply be constructed by choosing random word pairs. while we may occasionally end up with word pairs that are actually related in some way, the probability that this is the case for randomly chosen words is sufficiently low for this not to be problematic. however, constructing the set pos is less straightforward, since we are not looking for words that have a particular relation, but rather for words that have any kind of (clear) relationship. if the examples are not sufficiently diverse, then our classifier will simply detect whether r ab corresponds to one of the relations that were considered during training. one of the few relevant large-scale resources we have at our disposal is conceptnet. however, during initial experiments, we quickly found conceptnet to be too noisy for this purpose. we therefore instead relied on gpt-4 foot_1 to generate a diverse set of around 11,000 positive examples. the prompt we used to obtain a sufficiently diverse set of examples is shown in the appendix d. to create a set of negative examples, of the same size, we corrupted the positive examples. we then trained a logistic regression classifier on the resulting training set. throughout this paper, we will write inf(r ab ) for the probability that r ab expresses an informative relationship, according to this classifier. we will refer to this value as the informativeness of the embedding r ab . ## connecting concepts the relationship between two concepts a and b is sometimes easiest to explain using an intermediate concept x. for instance, umbrella is related to cloudy because (i) an umbrella protects against rain and (ii) rain is only possible if it is cloudy. a natural strategy for identifying such intermediate concepts is to find paths of length two connecting the target words a and b in conceptnet. however, the coverage of conceptnet is limited, and many relevant intermediate concepts might not be found in this way. one possible solution would be to consider a sequence of intermediate concepts, and model the relation between a and b in terms of a path @xmath2... → x @xmath3. however, longer conceptnet paths are often too noisy to be useful. moreover, in practice a single intermediate concept is usually sufficient, so rather than considering longer paths, we propose two strategies to find additional links between concepts based on word embeddings. they are illustrated in figure 1 . ## condensing relation embedding chains the strategy from section 4 allows us to identify intermediate concepts x 1 , ..., x n that can help to describe the relation between two given words a and b. each intermediate concept x i corresponds to a chain r ax i ; r x i b of two relation embeddings. these relation embedding chains can encode relationships in a fine-grained way, but in practice we often need a more compact representation. we therefore train a simple model to summarise a set of relation em-bedding chains {r ax 1 ; r x 1 b , ..., r axn ; r xnb } for a given concept pair (a, b) as a single vector. our model has the following form: the function ϕ : r d × r @xmath4: it aims to predict a relation embedding for the pair (a, b) from the relation embeddings r ax i and r x i b . these predicted embeddings are then combined using a pooling function f : r m ×...×r @xmath5. finally, the resulting vector is mapped back to a d-dimensional embedding using the decoder ψ : r @xmath6. for the decoder ψ, we simply use a linear layer. for the pooling function f , we experimented with sum-pooling and max-pooling. we found the result to be largely insensitive to this choice. throughout this paper, we will therefore fix f as summation. finally, the composition function ϕ is implemented as follows: training our focus is on solving analogy questions, which is an unsupervised problem. in the absence of task-specific training data, we train the model to predict the relbert embedding r ab , using the following loss: note that while we train the model on concept pairs (a, b) which have an informative relbert embedding, our aim is to use this model for pairs for which this is not the case. we thus rely on the assumption that a composition model which is trained on word pairs with informative relbert vectors will generalise in a meaningful way to word pairs for which this is not the case. ## solving analogy questions analogy questions involve a query pair (a, b) and a number of candidate answers (x 1 , y 1 ), ..., (x k , y k ). the aim is to select the candidate (x i , y i ) whose relationship is most similar to that of the query pair. when using relbert, we simply choose the pair (x i , y i ) for which cos(r ab , r x i y i ) is maximal. ## conclusions relations between concepts are typically modelled either as kg paths or as relation embeddings. in this paper, we have proposed a hybrid approach, which uses paths whose edges correspond the relation embeddings. this is useful to model relationships between concepts that are only indirectly related, and more generally for concept pairs whose relation embedding is unreliable. our approach crucially relies on a classifier which can predict the informativeness of a relbert embedding. this classifier is used (i) to identify analogy questions where we should not rely on relbert embeddings; (ii) to select reliable relbert vectors for training a model for condensing relation embedding chains; and (iii) to identify informative paths beyond those in conceptnet. we have relied on gpt-4 to generate training data for the informativeness classifier, which allowed us to address the lack of suitable existing datasets.