Exploring Medium-Sized LLM s for Knowledge Base Construction (2025)

Abstract: Knowledge base construction (KBC) is one of the great challenges in Natural Language Processing (NLP) and of fundamental importance to the growth of the Semantic Web. Large Language Models (LLMs) may be useful for extracting structured knowledge, including subject-predicate-object triples. We tackle the LM-KBC 2023 Challenge by leveraging LLMs for KBC, utilizing its dataset and benchmarking our results against challenge participants. Prompt engineering and ensemble strategies are tested for object prediction with pretrained LLMs in the 0.5-2B parameter range, which is between the limits of tracks 1 and 2 of the challenge.Selected models are assessed in zero-shot and few-shot learning approaches when predicting the objects of 21 relations. Results demonstrate that instruction-tuned LLMs outperform generative baselines by up to four times, with relation-adapted prompts playing a crucial role in performance. The ensemble approach further enhances triple extraction, with a relation-based selection strategy achieving the highest F1 score. These findings highlight the potential of medium-sized LLMs and prompt engineering methods for efficient KBC.

## introduction the creation of high-quality, machine-readable knowledge bases (kbs) is critical to advancements in natural language processing (nlp) and semantic web technologies @xcite . these technologies enable us to structure information in formats that enhance accessibility and interoperability for both humans and machines. large language models (llms) have recently emerged as powerful tools for a range of tasks, including the automation of knowledge extraction, particularly generating subject-predicate-object triples, that are fundamental components of knowledge graphs @xcite . by contributing to the construction of structured kbs, llms play a crucial role in enabling semantic reasoning, querying, and web-based applications. in this work, we investigate the potential of llms to automate knowledge base construction (kbc) by exploring the knowledge bases from pre-trained language models (lm-kbc) 2023 challenge @xcite . although the study does not involve direct participation in the challenge, it was selected due to its ongoing relevance and potential for further research. the dataset provided by the 2023 challenge includes 21 well-balanced relations, covering different topics such as geography, entertainment, or chemistry. these relations involve various categories, ensuring a wide range of diverse entities across different domains. the task is to predict an object given a subject-relation pair. for example, the subject "andorra" and the relation statebor-dersstate should yield the set of objects "spain, france". these subject-relation pairs are given to the llms to predict the corresponding set of objects. authors that participated in the lm-kbc 2023 challenge had to pick one of two different tracks. track 1 required the participants to use models with less than 1 billion parameters, whereas track 2 was open to models of any size, resulting in a choice of very powerful models, such as gpt-4 and llama 2 with 70b parameters @xcite . as a result, there was a gap in the exploration of models with sizes close to the threshold (i.e., medium-sized), which this study aims to address, offering a valuable balance between computational efficiency and predictive power. the 2023 challenge was chosen over the 2024 edition @xcite for several reasons. when we started this work, the proceedings of the 2024 edition had not been published yet, limiting the ability to draw insights from both the task and approaches. furthermore, the dataset in the 2024 version comprises only 5 relations, which restricts the scope of experimentation. our main research goal is to explore how llms can enhance the automation of kbc, particularly by accurately predicting objects, ultimately forming subject-predicate-object triples. to achieve this, we define a set of subgoals. we evaluate the performance of instructiontuned llms in the prediction task, including llama3.2-1b @xcite , gemma2-2b @xcite , and qwen2.5 @xcite , with 0.5b and 1.5b parameters. additionally, we extended our analysis beyond instructiontuned models to include deepseek-r1-distill-qwen-1.5b, as the deepseek family is revolutionizing the ai industry @xcite . this model falls within the same size range as our selected llms, making it a relevant addition for exploring its potential in the task. by analyzing these models, we aim to understand how medium-sized llms perform relative to both the smaller models from track 1 and the much larger models from track 2 used in the lm-kbc 2023 challenge, offering insights into the trade-offs between model size and accuracy. choosing not to rely on larger models offers advantages such as requiring fewer computational resources, enabling faster inference times, and potentially being run locally without the need for extensive infrastructure. to refine knowledge extraction, we explore the impact of different prompt engineering strategies. specifically, we investigate the impact of zero-shot and few-shot learning paradigms, designing tailored prompts for each relation type to optimize prediction accuracy. by structuring our prompts to align with the nature of each relation, we aim to improve object prediction while minimizing the need for computationally expensive fine-tuning. in addition, we assess the effectiveness of ensemble methods in improving triple generation accuracy. we compare two ensemble strategies: relation-based model selection, which assigns the best-performing model for each relation, and majority voting, which selects the most frequently predicted object across models. by leveraging the complementary strengths of different llms, we aim to determine whether ensemble techniques provide a significant advantage over individual model predictions. the main contributions of this work are summarized as follows: this paper is structured as follows: in section 2, we review findings in the field of kb construction using llms, focusing on the contributions from the 2023 lm-kbc challenge. section 3 outlines our methodology, including model selection, prompting techniques, and ensemble methods. in section 4, we present our results, followed by a discussion in section 5, where we discuss our findings, comparing them to prior work and highlighting key trends and limitations. finally, section 6 concludes the paper, summarizing the contributions and suggesting directions for future research. the code and experimental results of the study are available at https://github.com/ tomasccpinto/ldk25-medium-llms-kbc . ## related work in recent years, the construction of high-quality, machine-readable kbs has increasingly leveraged llms @xcite , marking a paradigm shift from traditional dependence on structured data sources like wikidata foot_0 @xcite to models such as gpt-4 @xcite , bert @xcite , and llama 3 @xcite . this shift has spurred significant progress in automating kbc, particularly in extracting structured subject-predicate-object triples directly from unstructured text. these efforts are exemplified by benchmarks like the lm-kbc challenges, which have provided a compre-hensive framework for evaluating these capabilities. ## methodology in this section, we present a comprehensive overview of the methodology employed to address our study. this includes a detailed description of each phase of the work, from the selection and preparation of the llms to the extraction of knowledge. ## results our experiments demonstrate notable progress in using instruction-tuned llms for kbc, achieving results far exceeding those of generative baselines like gpt-3. specifically, our best setting delivered up to four times better performance, highlighting the effectiveness of prompt engineering and contextual enhancements. based on these results, we make several observations. regarding the performance of individual models: ## discussion the results of this study provide valuable insights into the performance of various language models and prompting strategies for knowledge-based tasks. this section aims to provide a detailed interpretation of the findings, highlighting connections to existing research and discussing potential areas for improvement. ## conclusion this study explored the use of llms for kbc, focusing on their ability to predict object entities within the context of the lm-kbc 2023 challenge. through a systematic evaluation of multiple models, mostly instruction-tuned, and leveraging techniques such as prompt engineering and ensemble methods, we derived several key insights. our best f1 score, which stems from our ensemble configuration, is higher than that of the winner of track 1, proving the effectiveness of our approach. furthermore, we observe a strong correlation between parameter size and model performance, also within the qwen2.5 model itself. we also see performance differences based on the prompting method used: zero-shot prompting, tailored to each relation, achieved superior results compared to few-shot approaches. triple prompts consistently outperformed our question prompts. in addition, contextual enhancements, particularly through paragraph-level information, proved critical in improving f1 scores across all models, demonstrating the value of incorporating external knowledge. ensemble techniques, when one model is clearly dominant, marginally improved performance. the limited gains suggest further refinement is needed to enhance effectiveness. our work contributes to the growing body of research on natural language processing and semantic web technologies, demonstrating the viability of medium-sized llms for efficient kbc. by achieving results proportionally competitive with those of larger models under resource constraints, we underscore the value of methodological innovation over raw computational scale. however, despite our methods showing promising results, they still fall short of the standards required for robust kbc. this underscores that llms, in their current state, are not yet capable of replacing structured kbs, but rather complement them. to build on our work, future research could focus on incorporating contextual knowledge into 3-shot prompts and exploring their use within ensemble models. additionally, investigating advanced ensemble techniques, such as dynamic weighting or neural blending, as well as leveraging larger, more diverse datasets, could significantly enhance llm performance in kbc.