Multilingual Verbalisation of Knowledge Graphs (2025)

Abstract: Most work on Knowledge Graph (KG) verbalisation is monolingual leaving open the question of how to scale KG-to-Text generation to languages with varying amounts of resources. In this work, we explore KG-to-Text generation on nine languages including five high-resource (HR) languages (English, Chinese, French, Spanish, Russian) and four low-resource (LR) languages (Breton, Irish, Maltese, Welsh). We first construct silver multilingual training data for all nine languages and new gold out-of-domain test data for the five HR languages. Using this data and already available in-domain test sets for 7 of our 9 languages, we then compare three strategies: (1) NLG+MT—a state-of-the-art KG-to-English model followed by Machine Translation (MT) into the target language; (2) FTMT—multilingual MT models fine-tuned end-to-end on the silver data; and (3) FewShot—few-shot LLM prompting comparing 4 LLMs. We explore different prompting strategies and show that our best prompting strategy performs the best on all 9 languages, discussing the relative performance of the three approaches on Low vs High Resource languages and on in- vs out-of-domain data.The models, the test set, and the silver training data are available at https://github.com/MeloS7/Multilingual-KG-Verbalisation.

## introduction knowledge graphs (kg) encode knowledge about the world by recording facts i.e., triples of the form (e1, p, e2) with e1, e2 some entities or values (dates, quantities etc.) and p a binary relation also often referred to as a property. kg-to-text generation is the task of converting knowledge graphs into text. figure 1 shows an example kg graph and its verbalisation. previous work on kg-to-text generation has mainly focused on monolingual approaches, most of them targeting english @xcite figure 1 : example kg/text pair with a knowledge graph from the dbpedia knowledge base and an english text illustrating a possible verbalisation of the kg. 2020; @xcite @xcite @xcite @xcite . two key reasons for this are (i) the lack of naturally occurring parallel kg/text data for testing and training and (ii) the fact that kg-to-text generation models are usually trained or fine-tuned on such data. in this work, we address both these issues. on the one hand, we create multilingual silver training data and gather or create gold test data for kg-to-text generation into 9 languages: english (en), spanish (es), french (fr), chinese (zh), russian (ru), breton (br), irish (ga), maltese (mt), and welsh (cy). on the other hand, we explore three alternative models for kg-to-text generation: a hybrid model which does not require multilingual training data as it combines kg-to-english generation with machine translation (mt) from english into the target languages; direct generation where selected mt models are fine-tuned on silver multilingual training data to generate from an input graph; and few-shot llm prompting, which does not require training data but uses a retrieval base for few shot extraction. our contributions can be summarised as follows. for english and russian, the best models are either trained or fine-tuned on webnlg data @xcite . more recent work has also started investigating llms. @xcite found that chatgpt and gpt3 perform poorly on the english webnlg 2017 test set when compared with the best performing models from webnlg 2017. @xcite shows that zero-shot generation with chatgpt achieves near state-of-the-art performance on some measures of the webnlg 2020 challenge, but falls behind on others. additionally, they compare factual, counter-factual and fictional statements showing that there is a significant connection between seen data (data that is available on the web) and the quality of the output text. for the lr languages, since no gold training data is available, approaches are either grammar-based @xcite , hybrid kg-to-english genera-tion+machine translation @xcite or llm-based approaches @xcite . the grammar-based approach obtains good results but is limited to irish. @xcite show that combining gpt-3.5 with google translate outperforms all other systems on all metrics in the three languages it was tested on (irish, welsh, maltese) . none of these approaches shows convincing results for llm prompting on languages other than english. we extend on previous work by comparing a hybrid nlg+mt approach, direct generation with fine-tuned mt models and a direct few-shot llmbased approach on 9 languages (5 hr + 4 lr). our results show that, in contrast to previous results which indicated an advantage for the hybrid nlg+mt approach, a direct few-shot llm-based approach performs best. we first explain how we created the datasets used for fine-tuning, testing and few-shot retrieval. we then describe the three kg-to-text generation models we investigated and discuss the results on both in-and out-of-domain data. ## data webnlg. we collect all webnlg test sets that are currently available. this includes the test sets from the webnlg 2020 shared task (english, russian); from the 2023 webnlg shared task (breton, welsh, irish and maltese) and a recently published test set for spanish @xcite foot_0 . these test sets share the same 1,779 kgs (1,102 for russian). the texts are human-written by crowdworkers, researchers working on the topic (spanish) or by professional translators. the webnlg test sets are available on the web and thus likely to be part of the llms training data. we use this dataset for an in-domain evaluation of kg-to-text generation. ## models we compare two supervised approaches with a few-shot llm-based approach. the supervised methods (nlg+mt and ftmt) employ mediumsized models (hundreds of millions of parameters), whereas the few-shot setting relies on large-scale llms with tens of billions of parameters. while this introduces a difference in model size, our comparison is designed to reflect realistic usage conditions: supervised methods typically depend on retraining with available backbones, whereas prompting-based approaches leverage off-the-shelf llms without additional training cost. ## conclusion we explored three ways of extending kg-to-text generation to multiple languages in the absence of training data: few-shot prompting, generate-and-translate, and mt models fine-tuned on silver kg/-text data. for few-shot prompting, we compared 9 different ways of prompting and found that a prompting strategy which enriches the input with target language wikidata labels and target language few-shots selected based on their property overlap with the input graph yields the best results. automatic metrics show that this approach out-performs nlg+mt and ftmt across the board by a good margin. we also note that, on lr languages (breton, welsh, maltese, irish), performance markedly decreases for all models . not handle problems related to knowledge bases other than wikidata, which are also important for out-of-domain generalisation. we obtain few-shot data for the prompting approach and training data for the kg-to-text nlg+mt approach using machine translation, which adds some noise to the generation process. finally, even though our proposed in-and out-of-domain test sets cover a large spectrum of wikidata entities and properties, they still do not cover all possible combinations. while this work is focused on presenting a methodology and analysis for multilingual kg-to-text generation, the methods need to be trained on larger datasets to ensure the method's generalisation capabilities to larger domains, other knowledge bases, and more languages. deploying these methods without proper training and assessment through gold benchmarks could lead to introducing harmful biases in real-world situations. elbayad, kenneth heafield, kevin heffernan, elahe kalbassi, janice lam, daniel licht, jean maillard, et al. 2022. no language left behind: scaling human-centered machine translation. arxiv preprint arxiv:2207.04672. liam cripwell, anya belz, claire gardent, albert gatt, claudia borg, marthese borg, john judge, michela lorandi, anna nikiforovskaya, and william soto martinez. 2023. the 2023 webnlg shared task on low resource languages. overview and evaluation results (webnlg 2023). in proceedings of the workshop on multimodal, multilingual natural language generation and multilingual webnlg challenge (mm-nlg 2023), pages 55-66, prague, czech republic. association for computational linguistics. daniel duma and ewan klein. 2013. generating natural language from linked data: unsupervised template extraction. in proceedings of the 10th international conference on computational semantics (iwcs 2013) -long papers, pages 83-94, potsdam, germany. association for computational linguistics. angela fan, shruti bhosale, holger schwenk, zhiyi ma, ahmed el-kishky, siddharth goyal, mandeep baines, onur celebi, guillaume wenzek, vishrav chaudhary, naman goyal, tom birch, vitaliy liptchinsky, sergey edunov, edouard grave, michael auli, and armand joulin. 2020. beyond english-centric multilingual machine translation. preprint, arxiv:2010.11125. claire gardent, anastasia shimorina, shashi narayan, and laura perez-beltrachini. 2017. the webnlg challenge: generating text from rdf data. in proceedings of the 10th international conference on natural language generation, pages 124-133, santiago de compostela, spain. association for computational linguistics. rik koncel-kedziorski, dhanush bekal, yi luan, mirella lapata, and hannaneh hajishirzi. 2019. text generation from knowledge graphs with graph transformers. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 2284-2293, minneapolis, minnesota. association for computational linguistics. nalin kumar, saad obaid ul islam, and ondrej dusek. 2023. better translation + split and generate for multilingual rdf-to-text (webnlg 2023). in proceedings of the workshop on multimodal, multilingual natural language generation and multilingual webnlg challenge (mm-nlg 2023), pages 73-79, prague, czech republic. association for computational linguistics. junyi li, tianyi tang, wayne xin zhao, zhicheng wei, nicholas jing yuan, and ji-rong wen. 2021. fewshot knowledge graph-to-text generation with pretrained language models. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 1558-1568, online. association for computational linguistics. yinhan liu, jiatao gu, naman goyal, xian li, sergey edunov, marjan ghazvininejad, mike lewis, and luke zettlemoyer. 2020. multilingual denoising pre-training for neural machine translation. corr, abs/2001.08210. michela lorandi and anya belz. 2023. data-to-text generation for severely under-resourced languages with gpt-3.5: a bit of help needed from google translate (webnlg 2023). in proceedings of the workshop on multimodal, multilingual natural language generation and multilingual webnlg challenge (mm-nlg 2023), pages 80-86, prague, czech republic. association for computational linguistics. michela lorandi and anya belz. 2024. high-quality data-to-text generation for severely under-resourced languages with out-of-the-box large language models. in findings of the association for computational linguistics: eacl 2024, pages 1451-1461, st. julian's, malta. association for computational linguistics. simon mille, elaine uí dhonnchadha, stamatia dasiopoulou, lauren cassidy, brian davis, and anya belz. 2023. dcu/tcd-forge at webnlg'23: irish rules! (wegnlg 2023). in proceedings of the workshop on multimodal, multilingual natural language generation and multilingual webnlg challenge (mm-nlg 2023), pages 87-92, prague, czech republic. association for computational linguistics. anna nikiforovskaya and claire gardent. 2024. evaluating rdf-to-text generation models for english and russian on out of domain data. in proceedings of the 17th international natural language generation conference, pages 134-144, tokyo, japan. association for computational linguistics. kishore papineni, salim roukos, todd ward, and wei-jing zhu. 2002. bleu: a method for automatic evaluation of machine translation. in proceedings of the 40th annual meeting of the association for computational linguistics, pages 311-318, philadelphia, pennsylvania, usa. association for computational linguistics. maja popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. in proceedings of the tenth workshop on statistical machine translation, pages 392-395, lisbon, portugal. association for computational linguistics. virginia ramón-ferrer, carlos badenes-olmedo, and oscar corcho. 2025. spanish triple-to-text benchmark on low-resource large language models. nils reimers and iryna gurevych. 2019. sentence-bert: sentence embeddings using siamese bertnetworks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3982-3992, hong kong, china. association for computational linguistics. martin schmitt, leonardo f. r. ribeiro, philipp dufter, iryna gurevych, and hinrich schütze. 2021. modeling graph structure via relative position for text generation from knowledge graphs. in proceedings of the fifteenth workshop on graph-based methods for natural language processing (textgraphs-15), pages 10-21, mexico city, mexico. association for computational linguistics. martin schmitt, sahand sharifzadeh, volker tresp, and hinrich schütze. 2020. an unsupervised joint system for text generation from knowledge graphs and semantic parsing. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 7117-7130, online. association for computational linguistics. thibault sellam, dipanjan das, and ankur parikh. 2020. bleurt: learning robust metrics for text generation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7881-7892, online. association for computational linguistics. jörg tiedemann and santhosh thottingal. 2020. opus-mt -building open translation services for the world. in proceedings of the 22nd annual conference of the european association for machine translation, pages 479-480, lisboa, portugal. european association for machine translation. jannis vamvas and rico sennrich. 2022. nmtscore: a multilingual analysis of translation-based text similarity measures. in findings of the association for computational linguistics: emnlp 2022, pages 198-213, abu dhabi, united arab emirates. association for computational linguistics. xunjian yin and xiaojun wan. 2022. how do seq2seq models perform on end-to-end data-to-text generation? in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 7701-7710, dublin, ireland. association for computational linguistics. shuzhou yuan and michael faerber. 2023. evaluating generative models for graph-to-text generation. in proceedings of the 14th international conference on recent advances in natural language processing, pages 1256-1264, varna, bulgaria. incoma ltd., shoumen, bulgaria. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. preprint, arxiv:1904.09675. tianyi zhang*, varsha kishore*, felix wu*, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in international conference on learning representations.