Evaluating RDF -to-text Generation Models for E nglish and R ussian on Out Of Domain Data (2024)

Abstract: While the WebNLG dataset has prompted much research on generation from knowledge graphs, little work has examined how well models trained on the WebNLG data generalise to unseen data and work has mostly been focused on English. In this paper, we introduce novel benchmarks for both English and Russian which contain various ratios of unseen entities and properties. These benchmarks also differ from WebNLG in that some of the graphs stem from Wikidata rather than DBpedia. Evaluating various models for English and Russian on these benchmarks shows a strong decrease in performance while a qualitative analysis highlights the various types of errors induced by non i.i.d data.

## introduction knowledge graphs (kgs) describe connections between entities (e.g., people, places or events) thereby representing knowledge about the world. the task of kg-to-text generation consists in verbalising the content of a kg. much research on kg-to-text generation focuses on the webnlg dataset @xcite often restricting evaluation to the webnlg test sets. while these include both seen (in domain) and unseen (out of domain, ood) data for english, no unseen test data is available for russian. furthermore, the input graphs all stem from dbpedia and the texts are often stilted as they are either crowd-sourced (english data) or machine translated from the crowdsourced texts and manually verified (russian data). to assess how well current nlg models perform on ood kg-to-text generation, we create several novel benchmarks for both english and russian which address these shortcomings and differ from the webnlg test sets in several ways. first, they include both english and russian -webnlg only has unseen test data for english. second, they include both dbpedia and wikidata foot_0 graphs -webnlg focuses on dbpedia graphs. third, they contain various ratios of unseen entities and properties -this allows for a detailed analysis of how the type and ratio of unseen data impact performance. using these benchmarks, we then assess and compare several kg-to-text models. the results show a strong decrease in performance for all models compared to results on in domain data. a qualitative analysis highlights the various types of errors induced by ood data suggesting directions for further research on kg-to-text. for russian, the two best performing models are @xcite and @xcite . both models fine-tune a pre-trained model on the webnlg data with @xcite finetuning the pre-trained fred (full-scale russian enhanced denoiser, 1.7m parameter) model and @xcite mt5 base . neither of these models were evaluated on unseen data. ## creating new benchmarks for english and russian we aim to create benchmarks which support a finegrained assessment of how various types of unseen items impact generation. terminology. an unseen element is a kg element (entity or property) not seen in the webnlg training/dev data. an unseen category is a dbpedia category which is not part of the 16 categories foot_2 used in webnlg to create the training data. we create separate benchmarks depending on whether the input graph contains unseen entities, unseen entities and properties or unseen category. while the latter two benchmarks permit assessing how well models perform on out of domain data, the former helps evaluating how much performance degrades with varying ratios of unseen entities. for english and russian, we derive these benchmarks from the kelm dataset @xcite , a large dataset of (graph,text) pairs created using distant supervision. for russian, we additionally derive benchmarks from the webnlg data following a methodology similar to that used to create the webnlg unseen test set for english. kelm. @xcite created the kelm dataset in several steps as follows. first, wikidata triples were heuristically aligned to wikipedia sentences yielding a dataset of approximately 6m noisily aligned (graph, sentence) pairs and covering 1,041 wikidata properties. second, 15m wikidata graphs where created based on relation co-occurrence counts and the corresponding text was generated from these graphs using a t5 model fine-tuned on the silver 6m (graph,sentence) pairs. the semantic adequacy (semantic match between graph and text) and the fluency of 200 randomly selected kelm (graph,text) pairs were annotated by human judges (8 annotators, 2 judgements per instance) on a 1-5 scale, yielding an average rate of 4.36 for semantic adequacy and 4.60 for fluency. examples of kelm instances are shown in webnlg. the webnlg dataset is a dataset of (graph,text) pairs where graphs were extracted from dbpedia and texts were crowdsourced to match the input graph. for english, the training data covers 16 dbpedia categories and the test set has three subsets: seen (490 instances), a test set where graphs include only entities and properties present in the training data; unseen entities (393 instances), where graphs include entities not present in the training data; and unseen categories (896 instances), a test set where graphs are rooted in entities whose category does not belong to the 16 categories present in the training data. 4 for russian, the training data only covers nine categories foot_4 and all instances in the test set (1,200 instances) are from the seen categories. ## creating kelm benchmarks to create the kelm unseen test sets (kelm-e, kelm-e+p), we first select subsets of kelm that contain unseen entities and properties. we then ask human annotators to verify the semantic adequacy of the (graph, text) pairs (does the text match ## creating webnlg benchmarks for russian we derive two webnlg russian benchmarks from the webnlg english test set by first selecting graphs with unseen categories or unseen entities 10 the levenshtein distance indicates the minimum number of insertion, deletion or substitution of individual characters that are required to transform one sentence into another and the levenshtein ratio normalises this distance by the length of the two sentences and inverts the score so that a perfect match will have a score of 1.0, and completely dissimilar strings will be assigned a value of 0.0 (ldistance: levenshtein distance, lratio: levenshtein ratio): lratio(a, b) = 1 -ldistance(a,b) len(a)+len(b) as explained above, english webnlg differs from russian webnlg in that it covers 16 categories (vs. 9 for russian) and the test set includes an unseen category and an unseen entities test set. to create an unseen category test set for russian (webnlg-c, 1,251 instances), we simply select from the english test set all instances which belong to the 7 categories not included in russian webnlg training and dev data. the second test set @xcite instances) consists of the instances that are from seen categories in russian webnlg train or dev set, but the entities are unseen. these two subsets were then translated from english to russian by 5 professional translators, who have russian as a native language. as for the validation of the kelm translations, the translators were provided with the english text, the nllb translation and the dbpedia russian labels of the graph entities and again we observe a high ratio of changes introduced by the translators ( comparing the english texts to the corresponding graphs, the translators spotted a few errors (165 instances were highlighted out of the whole test set). those errors include references to female scientists or politicians by he/him, subject and object interchanged in the text comparing to the kg data. we created a new version v3.1 of the webnlg test data which integrates these corrections in the english version of the data and will be uploaded to the webnlg website once this paper is published. ## assessing generalisation we evaluate current pre-trained encoder-decoders on our benchmarks. since the best approaches in the 2020 edition of the webnlg shared task were based on t5 or mt5 @xcite , we consider various versions of this model fine-tuned on the webnlg english/russian training data. we also include in our evaluation the control-prefixes @xcite model, a state-of-the-art model for kg-to-text generation as well as the models for russian submitted to the webnlg 2023 challenge @xcite . we evaluate the models using automatic metrics and run a qualitative analysis to identify the most common errors occurring when assessing current models on out of domain data. russian. we also evaluate mt5 base,f t and mt5 large,f t fine-tuned on webnlg russian training data on the russian benchmarks. in addition, we evaluate the mt0 pre-trained model (mt5 finetuned on crosslingual tasks, @xcite , 2023)) fine-tuned on the webnlg training data for russian (mt0 f t ) and two models for russian that participated in the webnlg 2023 challenge. the first model is interno, a model based on fred-t5 (full-scale russian enhanced denoiser, 1.7m parameters, (zmitrovich et al., 2023)) and fine-tuned on webnlg training data (kazakov et al., 2023). we used the final checkpoints submitted to the webnlg 2023 challenge. the second model is cuni, a mt5 base model which was fine-tuned on multilingual data created by machine translating (using nllb) webnlg training data into maltese, irish gaelic and welsh and including the original russian data (kumar et al., 2023). 13 ## qualitative analysis to get a better understanding of the type of errors made by generation models on ood data, we run a qualitative analysis on the models outputs. ## conclusion we created challenging benchmarks for kg-to-text generation into english and russian, quantitatively demonstrated the effects of applying models trained on one distribution (e.g., webnlg data) to a new distribution (e.g., unseen entities and/or properties) and identified nine error types which arise in this setting. the ability of existing generation models to generalise to ood data is underexplored and we hope the benchmarks and evaluations we provide inspire further research on this topic, for instance under alternate kg-to-text models.