Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models (2024)

Abstract: In this work, we tested the Triplet Extraction (TE) capabilities of a variety of Large Language Models (LLMs) of different sizes in the Zero- and Few-Shots settings. In detail, we proposed a pipeline that dynamically gathers contextual information from a Knowledge Base (KB), both in the form of context triplets and of (sentence, triplets) pairs as examples, and provides it to the LLM through a prompt. The additional context allowed the LLMs to be competitive with all the older fully trained baselines based on the Bidirectional Long Short-Term Memory (BiLSTM) Network architecture. We further conducted a detailed analysis of the quality of the gathered KB context, finding it to be strongly correlated with the final TE performance of the model. In contrast, the size of the model appeared to only logarithmically improve the TE capabilities of the LLMs. We release the code on GitHub for reproducibility.

## introduction the task of triplet extraction (te) @xcite is of fundamental importance for natural language processing (nlp). this is because the core meaning of a sentence is usually carried by a set of (subject, predicate, object) triplets. therefore, the capability to identify such triplets is a key ingredient for being able to understand the sentence. currently, the state-of-the-art (sota) for te is achieved by models that approach the te task in an end-to-end fashion @xcite @xcite @xcite . that is, they are trained to perform all the te sub-tasks, namely, named entity recognition (ner @xcite ), entity linking (el @xcite ), and relation extraction @xcite ), together. these sota models follow the classic nlp paradigm, i.e., they are trained by supervision on specific te datasets. however, this dependence on labeled data restricts their generality and, therefore, limits the applicability of such models to the real world. while several labeled datasets for the te task are publicly available @xcite , these cover only part of the spectrum of possible entities and relations. this means that a supervised model trained on these public data will be restricted to the closed set of entities and relations seen during training, implying that it may lack generalization capabilities. producing a tailored dataset for training a model for particular applications, is, however, in general expensive @xcite . for this reason, the recent language understanding and reasoning capabilities demonstrated by large language models (llms), such as the generative pre-trained transformer 4 (gpt-4) (ope-nai, 2023), llm meta ai (llama) @xcite , and falcon @xcite to name a few, have led researchers @xcite @xcite @xcite to investigate whether they represent a viable option to overcome the limitations imposed by supervised models for te. in detail, the new approach being that at inference time the llms are prompted to extract the triplets contained in a sentence, while being provided with only a few labeled examples (or no example at all in the zero-shot setting). this llm approach largely limits the amount of data needed to perform the task, and, in particular, lifts the restriction of adhering to a predefined closed set of relations. however, the investigations so far indicated that the zero and few-shots performance of the llms appears to be often underwhelming compared to the classic fully trained nlp models @xcite @xcite . in order to enhance the abilities of llms in the te task, we propose in this work to aid them with the addition of a knowledge base (kb). we demonstrate that augmenting llms with kb information, i.e., dynamically gathering contextual information from the kb, largely improves their te capabilities, thereby making them more competitive with classic nlp baselines. in particular, we show that when the retrieved information is presented to the llms in the form of complete te examples relevant to the input sentence, their performance gets closer to the fully trained sota models. ## related work classical end-to-end fully supervised models currently hold the best performance in the te task. starting from the older baseline, @xcite , which also introduced the revised version of the webnlg dataset for natural language generation (nlg) that is commonly used, several other architectures based on the bidirectional recurrent neural networks (rnns) @xcite @xcite have steadily improved the sota over the years. more recently, transformer-based models achieved a big leap forward in performance, with the recent unirel model being the current sota @xcite in the datasets we consider. a further class of fully supervised models, such as huguet cabot and navigli (2021) and @xcite , treats the te problem as a sequence-to-sequence generation task, which is more similar to the llm approach adopted here, but still requires some training or finetuning. with the advent of llms, chia et al. (2022) and @xcite tested the use of such models for those te cases where the availability of examples to train on is low. the first work proposed to use a llm to generate training examples to finetune a relation extractor model to recognize relations for which labels were not available. the latter work, instead, suggested using relation templates of the form 〈x〉 relation 〈y〉 and finetune a llm to fill out 〈x〉 and 〈y〉 with the entities appearing in the sentence. @xcite , @xcite , and @xcite investigated the general te task in both zero-and few-shots settings. these studies proposed different approaches based on llm prompting. the first work tested the few-shots performance of @xcite ) and textto-text transfer transformer (t5) @xcite under the inclusion of manually-crafted and dataset-dependent contextual information in the prompt. the second work proposed to perform te by sequentially prompting chatgpt in two stages: asking to individuate the possible relation types first and then extracting the entities participating in each relation. the procedure demonstrated better results than a one-stage approach where the model is prompted to extract the triplet directly. finally, the third work, evaluated @xcite and gpt-4 (openai, 2023) on some standard benchmarks in the zero-and one-shot settings. however, classical fine-tuned models proved to be superior in the majority of the cases. in our study, we similarly test the zero-and few-shots capabilities of llms in two standard te datasets that have not been covered by these previous works. in contrast to @xcite that manually crafted static dataset-specific context to be fed to the llm, we propose here to dynamically gather contextual information useful for extracting the triplets from a kb. this makes our approach more flexible and less data-dependent, as the kb does not require any manual operation and can be easily switched depending on the need. also, in contrast to other works, we investigate a wide range of language models with varying sizes. this allows us to provide an in-depth analysis of the scaling of the performance, both, from the perspective of the model chosen, and the quality of the contextual kb information included in the prompt. ## the pipeline in this section, we provide a detailed illustration of the pipeline used to test the te capabilities of llms. ## experiments in this section, we first provide details about the datasets and models we tested. this is followed by ## conclusion in this work, a pipeline for zero-and few-shots te from sentences was presented and tested for various llms. we showed that the inclusion of kb information into the llms prompting can substantially improve the te performance. in particular, small models were often able to outperform their bigger siblings without access to the additional kb information. furthermore, with the information from the kb organized as sentence-triplets pair examples relevant to the input sentence, the accuracy of the llms improved further. in this setting, the larger llms were getting closer to the classical sota models and outperformed most of the older baselines. however, even for the largest models, te remains a challenging task without any finetuning. llms were still no match for sota classical finetuned models in the two standard benchmark datasets we tested as part of our work, in agreement with @xcite al. (2023); wei et al. (2023b); zhu et al. (2023). moreover, the performed investigation of the quality of the retrieved kb context showed that the solution to the te task was often contained inside it already. this first indicated that a large overlapping between the train, validation and tests sets exists for both webnlg and nyt, leading us to reconsider their generality for benchmarking te capabilities and suggesting that a revision with better test isolation might be helpful. secondly, it demonstrated that, while llms are capable of correctly individuating the relevant information in the context, they do not shine, yet, in re-elaborating such information, generalizing and making use of it for different examples. indeed, the investigation of the impact of the quality of the retrieved kb context, showed as the performance of the llama-65b model linearly decreased with the probability of finding the solution of the task within the context already, indicating that the intrinsic incompleteness of kbs might represent a big limiting factor of this approach. concurrently, we found that the te performance improved only approximately logarithmically with the size of the model. this suggests that improving the quality of the kb and the associated information retriever might be more effective than increasing the modeling power of the llm for te. tables 7 and 8 report the results obtained by the openai models on the webnlg and nyt datasets in all the different settings. the comparison with the other models, c.f. tables 4 and 5 , shows them to be comparable to the falcon 40b model in the majority of the cases. however, they recorded very underwhelming results on the nyt dataset both, in the 0.5-shots and few-shots settings. our manual inspection of the triplets they provided as an answer suggested that they were less keen to adhere to the entities and relations appearing in the provided kb context, often paraphrasing or reformulating them in a more prolix form that lowered the accuracy. this might be a consequence of the instructed training they had gone through, as discussed in section 4.1. in contrast, the results provided by the non-instructed text-davinci-002 model were more in line with all the other llms.