Fine-Tuning, Prompting and RAG for Knowledge Graph-to- R ussian Text Generation. How do these Methods generalise to Out-of-Distribution Data? (2025)

Abstract: Prior work on Knowledge Graph-to-Text generation has mostly evaluated models on in-domain test sets and/or with English as the target language. In contrast, we focus on Russian and we assess how various generation methods perform on out-of-domain, unseen data. Previous studies have shown that enriching the input with target-language verbalisations of entities and properties substantially improves the performance of fine-tuned models for Russian. We compare multiple variants of two contemporary paradigms — LLM prompting and Retrieval-Augmented Generation (RAG) — and investigate alternative ways to integrate such external knowledge into the generation process. Using automatic metrics and human evaluation, we find that on unseen data the fine-tuned model consistently underperforms, revealing limited generalisation capacity; that while it outperforms RAG by a small margin on most datasets, prompting generates less fluent text; and conversely, that RAG generates text that is less faithful to the input. Overall, both LLM prompting and RAG outperform Fine-Tuning across all unseen testsets. The code for this paper is available at https://github.com/Javanochka/KG-to-text-fine-tuning-prompting-rag

## introduction generating from structured data has a variety of applications such as verbalising tabular data @xcite ), question answering from databases @xcite and generating from meaning representations @xcite @xcite @xcite or from knowledge graphs @xcite . we consider knowledge graph-to-text (kg-to-text) generation where the task consists of verbalising the content of a kg, and a kg is a set of triples of the form (subject, property, object) where subject are entities and objects are values or entities. while kg-to-text generation has been extensively explored in the context of the webnlg shared tasks @xcite @xcite , in this work, we investigate the generalisation capacity of three main generation methods (fine-tuning, llm prompting and retrieval augmented generation) taking russian as a target language and evaluating on out-of-domain (ood) datasets. compared to generation into english, generating from knowledge graphs into russian faces the challenge of decoding into a language with a different script, a rich verb morphology and a complex nominal declension system. prior work shows that lexicalising entities into russian is a difficult task @xcite and that pre-trained models fine-tuned on the russian webnlg dataset show a strong performance decrease on non i.i.d data @xcite . starting with the state-of-the-art kg-to-russian models presented in @xcite , we begin by showing that a pre-trained model fine-tuned on the webnlg data under-performs on both seen and unseen data but that the same model achieves much better results when the input graph is enriched with russian lexicalisations of the entities and properties present in the input graph. next we explore various llm-based, few-shot prompting approaches @xcite . generating with an llm benefits from the parametric knowledge acquired from their training on large amounts of data. compared to @xcite 's model which is a fine-tuned version of t5 @xcite , and therefore trained on much smaller quantities of data than llms, the knowledge encapsulated in an llm might help generalise to unseen test data. as in the fine-tuning approach described above, we enrich the input prompt with russian lexicalisations (wikidata labels) of the entities and properties present in the input graph, and we experiment with various ways of selecting few-shots. intuitively, the selected few shots should help provide two types of information: (i) how the properties and entities present in the input graph are verbalised into russian and (ii) how a knowledge graph is verbalised into a text. accessing these two sources of knowledge can be viewed as learning both entity/property verbalisation and a generation template. sample templates can be identified from the associated text and property verbalisation by jointly accessing graph properties and their textual mention. we introduce a few-shot selection method which systematically improves generation results by selecting few shots whose graph size and properties resemble the size and properties present in the input graph. retrieval augmented generation @xcite , a method which extends the generation input with retrieved information, is another common way to integrate external knowledge into the generation process. different from the other two generation approaches we explored, we use retrieval, not to extract wikidata labels, but to retrieve textual data from wikipedia. this removes the need for a knowledge base that provides multilingual labels for entities and properties and enables an approach that can more easily adapt to other knowledge bases. we explore a variety of strategies for querying, chunking and few-shots selection and we show that the best performing rag variant performs almost on par with the llm-prompting approach while being less restrictive in terms of resources. in summary, we compare three ways of enriching the generation input to help the model generalise to unseen input: ## related work various approaches have been proposed to convert kgs into text including grammar-and templatebased approaches, custom encoder-decoders, finetuned pre-trained models and pipelines @xcite @xcite . more recently, llm prompting has also been explored. yuan and faerber (2023) evaluate chatgpt on the agenda and webnlg test sets -they find that both models perform relatively poorly on most measures. axelsson and skantze (2023) evaluate gpt-3.5-turbo on both the webnlg and on some non i.i.d. test sets they created, showing good results on english indomain data but low performance on russian and poor semantic adequacy on ood data. comparing zero-and few-shot variants of gpt3.5-turbo, llama-7b, vicuna-7b and llama-7b fine tuned on webnlg data, schneider et al. (2024) finds that fine-tuning yields the best results on webnlg test data. finally, he et al. (2025) show that selecting few-shots based on complexity and diversity enhance results for kg-to-english generation. other work has also explored retrieval augmented generation. in particular, jobanputra and demberg (2024) propose a few-shot rag system for english where few shots are selected to match the properties contained in the input graph. leveraging the webnlg training and test data, some work has explored generation into russian. agarwal et al. (2020) pre-train and fine-tune t5 on several parallel corpora in a multi-task (nlg/semantic parsing) setting, obtaining best results in the 2020 shared task. mille et al. (2024) use a rule-based approach to convert triples into english, a language model to paraphrase the resulting texts and a machine translation model to translate english into russian. on the webnlg testsets, the best results for generation into russian are obtained by fine-tuned models, with kumar et al. (2023) presenting a fine-tuned mt5 model @xcite and @xcite achieving the state of the art by fine-tuning the fred-t5 model @xcite on the webnlg training data for russian. other work has explored the behaviour of kg-to-text models on ood data. in particular, @xcite assess generalisation by constraining the training data and evaluating on the full webnlg testsets. they show that the sota pre-trained language models @xcite fail to generalize and that a compositional approach based on clustering helps improve generation. @xcite derive challenge testsets from the existing webnlg data to identify performance decrease that would associate with specific linguistic phenomena (e.g., large number of co-referring entities) or graph characteristics (e.g., high number of triples). @xcite show that, on unseen data which they created by collecting data records from public apis, more than 80% of the output contain at least one semantic error. we depart from previous work in that we focus on russian, evaluate on unseen testsets and compare the behaviour of fine-tuned models, llmprompting and rag on this unseen data. ## fine-tuning on labeled data the first method we explore builds on the stateof-the-art interno kg-to-text model for russian @xcite , which fine-tunes fred-t5 model on the webnlg training data foot_1 . to facilitate entity verbalisation, the authors modify the input in the training and test sets to include, in addition to the knowledge graph, the russian entity and property labels which are provided by the webnlg data. as our unseen test sets do not contain such labels, we extract russian entity and property labels from the wikidata knowledge base using the sparql queries shown in appendix b. retrieved labels are added to the generation input under the key "additional links" using the format "@xmath0. we then run the interno model on both seen and unseen data, comparing results when the input is ## llm prompting for llm-based generation, we use llama-3.1-8b-instruct @xcite and experimented with a few-shots prompting strategy that varies across three dimensions: the input, the prompting strategy and how few-shots are selected. details about each of these dimensions and example prompts, illustrating the various options for each dimension are given in appendix a and subsection a.4. briefly, the input is either the knowledge graph (g) or the kg enriched with wikidata property descriptions (+p), wikidata russian entity labels (+l), wikidata russian property descriptions and entity labels (+p+l) or wikidata property and entity descriptions (+all). the prompting strategy is to either prompt the llm for a direct verbalisation of the entire graph (d) or to first verbalise each fact in the input graph and then aggregate the cumulative verbalisations into a fluent text (cum). for few shot selection, we select 4 few shots foot_2 and explore three strategies: random (r); size-based, where we select few shots containing graphs whose size is the same as the size of the input graph (s) foot_3 ; and property based, which consists in selecting few shots whose graphs contain properties similar to the set of properties present in the input graph (p when the selected few-shots are from the english retrieval base; pru, when they are in russian). algorithm 1 in the appendix shows the algorithm used for p and pru few-shot selection. ## retrieval augmented generation for llm unlike the previous two approaches we explore, the rag method does not rely on russian labels from a knowledge base. instead, it retrieves semantically aligned wikipedia text to enrich the input kg, providing a flexible framework that can readily adapt to other knowledge bases. rag provides a natural way to enrich the input by retrieving additional data from a vector base containing task-relevant knowledge and adding the retrieved data to the generator input. we combine rag with few-shots prompting by including (i) four few-shots and (ii) the retrieved content in the prompt as context for decoding. typically, rag retrieves data based on a given query (here the generation input) and the vector base from which data is retrieved contains data chunks. we explore various options for the query, the vector base chunks and few-shots selection. we used llama-3.1-8b-instruct (grattafiori et al., 2024) as the base llm model. additionally, we ran experiments with qwen2.5-7b-instruct @xcite and mistral-7b-instruct-v0.3 @xcite . paragraph retrieval. we first experiment with a simple approach where the additional data is the first paragraph of the wikipedia article associated with the graph root entity. as previously observed @xcite , this paragraph contains much of the information contained in wikidata graphs. hence, we explore a simple approach where, instead of retrieving labels from the wikidata knowledge base or searching a vector base for relevant content, we include in the prompt the first paragraph of the wikipedia page corresponding to the root entity of the input graph. as shown in ## comparing the three approaches we compare the three approaches (fine-tuning, prompting, rag) using the best model we identified for each approach i.e., augmenting the input with wikidata labels for the interno model fine-tuned on the russian webnlg data (interno+labels in ## conclusion focusing on russian as a target language, we examine how well different models can convert englishcentric knowledge graphs into russian text on datasets where the input graphs contain unseen entities and/or properties.we compare three common methods for generating text: fine-tuning a pretrained model on task-specific data, prompting a large language model and retrieval augmented generation. for each method, we design and evaluate multiple variants, selecting the one that performs best. across datasets we find that the fine-tuning approach excels on in-domain data but systematically underperforms on unseen data, exposing limited generalisation capacity; that llm-prompting has a slight advantage over rag in terms of automatic metrics; and that, based on a human evaluation, rag produces more fluent, but less faithful text than llm-prompting. in future work, we plan to examine how these results extend to other languages.