A R el E nt L ess Benchmark for Modelling Graded Relations between Named Entities (2024)

Abstract: Relations such as “is influenced by”, “is known for” or “is a competitor of” are inherently graded: we can rank entity pairs based on how well they satisfy these relations, but it is hard to draw a line between those pairs that satisfy them and those that do not. Such graded relations play a central role in many applications, yet they are typically not covered by existing Knowledge Graphs. In this paper, we consider the possibility of using Large Language Models (LLMs) to fill this gap. To this end, we introduce a new benchmark, in which entity pairs have to be ranked according to how much they satisfy a given graded relation. The task is formulated as a few-shot ranking problem, where models only have access to a description of the relation and five prototypical instances. We use the proposed benchmark to evaluate state-of-the-art relation embedding strategies as well as several publicly available LLMs and closed conversational models such as GPT-4. We find that smaller language models struggle to outperform a naive baseline. Overall, the best results are obtained with the 11B parameter Flan-T5 model and the 13B parameter OPT model, where further increasing the model size does not seem to be beneficial. For all models, a clear gap with human performance remains.

## introduction language models (lms) capture an abundance of factual and commonsense knowledge about the world @xcite @xcite @xcite . given two entities, large language models (llms) can straightforwardly be used to obtain a description of how these entities are related, although with some caveats for less popular entities @xcite . however, relations are often a matter of degree @xcite @xcite . for instance, suppose we are interested in modelling whether one entity has been influenced by another one. while we could argue that most contemporary pop music has been influenced by the beatles, clearly there are some bands that have been influenced more directly than others. graded relations such as influenced by, competitor of or similar to are typically not found in traditional knowledge graphs (kgs), while they can nonetheless be of central importance to applications. for instance, in the context of financial nlp, we may need to know which companies are leaders and which are followers in a given field, who is competing with whom, and what strategic alliances exist. as another example, music recommendation systems often suggest artists based on the user's listening history, but these suggestions would be more helpful if the system could identify artists that have influenced or were influenced by artists the user already likes, as opposed to merely identifying similar artists. studying how such relations can be modelled is thus clearly an important but under-explored research problem. the subjective nature of graded relations makes it difficult to include them in traditional kgs. moreover, for many of these relations, it would simply not be feasible to list all the (graded) instances in a comprehensive way. taking inspiration from existing work on extracting kgs from llms, we therefore ask the following question: are current llms capable of modelling graded relations between named entities in a meaningful way? the task of modelling graded relations offers a number of unique challenges for llms. first, since this is essentially a ranking task, designing suitable prompts is not straightforward. second, the task requires making very fine-grained distinctions. for instance, while we can say that microsoft is known for windows and apple is known for macos, the former statement represents a more prototypical instance of the known for relation, as apple is perhaps best known for its hardware products (e.g. iphone). it is currently unclear to what ex-tent llms are able to capture such subtle differences. finally, modelling graded relations requires comparing entities of different types. for instance, the known for relation has instances such as (microsoft,windows), (the beatles, hey jude) and even @xcite . comparing instances of such a diverse nature poses a particular challenge, as such comparisons are almost never expressed in text. in this paper, we introduce relentless foot_0 , a new dataset aimed at furthering the study of graded relations between named entities. our dataset covers five common graded relations: competitor/rival of, friend/ally of, influenced by, known for, and similar to. we evaluate the ability of llms to rank entity pairs according to how much they satisfy these relations, given a description of the relation and five prototypical examples. analysing the performance of several recent llms @xcite , including gpt-4 (openai, 2023), we find the best models to achieve a spearman rank correlation of around 0.6. this shows that recent llms capture fine-grained relational knowledge to a meaningful extent, while at the same time still leaving a significant gap with human performance. for the open-source llms, we find that while the largest models achieve strong results, smaller models fail to outperform a naive baseline based on fasttext vectors @xcite . gpt-3 performs well, albeit slightly below the best variants of flan-t5 and opt. finally, we found chatgpt and gpt-4 hard to use for this task, since the openai api foot_1 does not allow computing perplexity scores. as a result, we were not able to outperform gpt-3 with these models. ## related work benchmarks for graded relations relent-less was inspired by the semeval 2012 task 2 dataset on modelling relational similarity @xcite , which we will refer to as relsim. rel-sim covers 79 fine-grained relations, which are organised into 10 categories, such as part-whole (e.g. car:engine), attribute (e.g. beggar:poor) and causepurpose (enigma:puzzlement). for each of the finegrained relations, a ranking of concept pairs is provided, which reflects how prototypical these pairs are as instances of the relation. however, relsim only considers concepts, whereas our focus is on named entities. to the best of our knowledge, the problem of modelling relational similarity between named entities has not yet been considered. hyperlex @xcite is focused on modelling hypernymy as a graded relation. it involves ranking concept pairs according to how prototypical they are of the hypernymy relation. as for relsim, named entities were explicitly excluded from this dataset. more broadly, word similarity benchmarks also follow the format of ranking concept pairs according to the degree to which a graded relation is satisfied, i.e. similarity. benchmarks with analogy questions @xcite @xcite also relate to the problem of modelling graded relations. these benchmarks typically follow a multiple-choice format, where one word pair is given (e.g. eye:seeing), and the system has to predict which among a given set of candidate answer pairs is most analogous to the query pair (e.g. ear:hearing). most existing benchmarks again focus on concepts. moreover, where named entities are involved, the task degenerates to predicting whether two entity pairs have the same relation, i.e. the problem of measuring degrees of relatedness is not considered for named entities. language models as knowledge bases the idea of using language models as knowledge bases was popularised by @xcite , and has gained considerable further traction with the advent of llms. for instance, several authors have proposed strategies for extracting knowledge graphs from llms @xcite @xcite . while the idea of modelling graded relations has not been considered, @xcite focused on relations that are not covered by traditional knowledge graphs, such as "is capable of but not good at". similarly, our motivation for studying graded relations between named entities is also to complement what is captured by kgs. ## dataset we consider the five relations which are shown in first phase in the first phase, the annotators were asked to provide 15 entity pairs for each of the five relations. specifically, the aim was to provide 5 prototypical examples (i.e. entity pairs that clearly satisfy the relationship), 5 borderline positive pairs, which only satisfy the relationship to some extent, and 5 borderline negative pairs, which do not satisfy the intended relationship but are nonetheless related in a similar way. after removing duplicates, this resulted in an average of 114 entity pairs for each relation, and 573 pairs in total. we augmented these entity pairs with the same number of randomly chosen entity pairs as the annotated pairs in each relation type. the entities for these random pairs were selected from the 50,000 most popular wikidata entities, in terms of the number of page views of the associated wikipedia article. second phase in the second phase, each annotator scored all the entity pairs that were provided in phase 1, using the 5-point scale shown in third phase the third and final phase was aimed at resolving disagreements between the annotations from the second phase. specifically, for each entity pair where there was a difference of 3 points between the highest and the lowest score, the annotator(s) with a diverging view were asked to check their previous annotation, and to either update their score or to provide a justification. a total of 255 unique entity pairs were checked in this way (310 scores were checked in total). we subsequently verified the justifications that were provided. in the final ranking for each relation was obtained by averaging the scores of the 7 annotators. ## results ## analysis we now aim to gain a better understanding of the behaviour of lms. first, we analyse the effect of model size (subsection 6.1). then, we experiment with different zero-shot and few-shot learning setups (subsection 6.2), and we present a qualitative analysis of the predictions (subsection 6.3). for the latter two analyses, we focus on the best performing models for each lm family from the main experiment, using their optimal prompts: flan-ul2, flan-t5 xxl , opt 13b , and gpt-3 davinci . foot_11