The 2023 W eb NLG Shared Task on Low Resource Languages. Overview and Evaluation Results ( W eb NLG 2023) (2023)

Abstract: The WebNLG task consists of mapping a knowledge graph to a text verbalising the con- tent of that graph. The 2017 WebNLG edi- tion required participating systems to gener- ate English text from a set of DBpedia triples, while the 2020 WebNLG+ challenge addition- ally included generation into Russian and se- mantic parsing of English and Russian texts. In contrast, WebNLG 2023 focuses on four under-resourced languages which are severely under-represented in research on text genera- tion, namely Breton, Irish, Maltese and Welsh. In addition, WebNLG 2023 once again includes Russian. In this paper, we present the organi- sation of the shared task (data, timeline, eval- uation), briefly describe the participating sys- tems and summarise results for participating systems.

## introduction the webnlg challenges seeks to add to the research on knowledge graph verbalisation i.e., how to convert a knowledge graph into a text verbalising its content. this is illustrated in figure 1 where the text shown conveys the content of the input graph. alan bean graduated from ut austin in 1955 with a bachelor of science degree. he was hired by nasa in 1963 and served as a test pilot. apollo 12's backup pilot was alfred worden and was commanded by david scott. the first edition of the challenge, the 2017 webnlg shared task, required participating sys-tems to generate english text from a set of dbpedia triples @xcite . the 2020 webnlg+ challenge encompassed four tasks: rdf-to-english, rdf-to-russian, english-to-rdf and russian-to-rdf. with the development of large-scale pretrained models, research in automatic text generation has acquired new impetus. yet, the current state-ofthe-art is dominated by a handful of languages, for which training data is relatively easy to acquire. at the same time, the field has recently witnessed some encouraging developments which focus on generation for under-resourced and underrepresented languages @xcite . this trend is paralleled by a growing interest in multilingual models and applications in nlp more broadly. the webnlg 2023 challenge was organised in response to these trends and specifically addresses generation for four under-resourced languages: breton, irish, maltese and welsh. in addition, webnlg 2023 once again includes russian. timeline. noisy training data, gold development data and evaluation scripts were released on february 24, 2023. the test data was made available on june 8th and the deadline for submitting system results was june 15th. automatic evaluation results were announced on june 13th and the human evaluation results on august 18th. results were first released anonymously so that participants had the opportunity to withdraw their systems. in what follows, we summarise the main features of webnlg 2023. section 2 describes the datasets used for the challenge. section 3 briefly presents the participating systems (more detailed descriptions are provided by the participants in separate papers). section 4 introduces the evaluation methodology, section 5 discusses the automatic evaluation results for participating systems and section 6 the human evaluation results. finally, sec- ## data to obtain the development and test data for each of the low-resource languages (breton, irish, maltese, and welsh), we had professional translators manually translate the english text from the webnlg 2020 development and test sets, given both the english text and the input rdf graph. 1 we only consider the first reference of each test example in the original english dataset when translating, except in the case of breton which contains 2 translated references for some of the test items. for russian, we continue to use the webnlg + 2020 data. in addition to the professionally translated dev and test data, we provide optional 'noisy' training data for the low-resource target languages, which was obtained via machine translation of the texts in the english webnlg 2020 training set. for this, we used the 24-layer zero machine translation system @xcite ). 2 ## participating systems this section provides a brief overview of the participating systems and of the runs submitted by the participants to the shared task. further results obtained after the submission deadline and more details about each systems are available in the system descriptions provided by the participants @xcite @xcite @xcite . seven teams submitted a system run and two teams withdrew after the automatic evaluation results for test set were shared with participants, yielding a final total of five participating teams ( in terms of architecture, monolingual submissions were either rule-based @xcite or a monolingual llm fine-tuned on the webnlg data @xcite . submissions targeting multiple languages predominantly adopted an nlg+mt pipeline approach. ## results of automatic evaluation in this section, we present results of the evaluation using automatic metrics. these are summarised in as regards the monolingual systems, interno is ranked (joint) first for russian on all metric, while dcu/tcd-forge is ranked second for bleu and chrf++, fourth for ter, and third or fourth for bert p/r/f1. while this suggests that monolingual approaches (whether llm-based or rulebased) can provide competitive solutions for specific languages, the broader pattern across the four languages shows that combining llms with machine translation is more effective in the absence of large amounts of training data. dcu-nlg-pbn which combines gpt-3.5 with google translate and is not trained/finetuned on any webnlg data, outperforms all systems on all metrics in the three languages it was tested on. it is also instructive to consider the relative performance of irel and cuni-wue, the latter leveraging mt for improved training data, while the former uses it to translate outputs. irel narrowly outperforms cuni-wue on maltese, welsh and irish on all metrics except bleu on irish. in summary, these results show a continued role for mt to handle under-resourced languages, even with very large models, as in the case of dcu-nlg-pbn, who report worse performance with direct generation @xcite . the inclusion of russian in webnlg 2023 provides a point of comparison with the results for the same language in the 2020 edition @xcite . in 2020, the cuni-ufal system kasner and dušek (2020) ranked second on russian. this year, it is narrowly outperformed by interno and cuni-wue, at least on bleu and chrf++. on bertscore, the systems are very close. indeed, results on russian for the present edition provide very small improvements over the best results for 2020. ## results of human evaluation in this section, we present and discuss results from the human evaluations for irish, maltese and welsh. for russian, we were unable to recruit and train translators able to perform the task reliably, as discussed in the next section below, and in appendix a. ## correlation between automatic and human evaluation metrics tables 6 to 8 are pearson correlation matrices for all automatic and human-assessed output quality measures. the colour coding goes from vivid green for strong positive correlations to vivid red for strong negative correlations, with both colours growing paler toward 0 (no correlation). as can be seen very clearly, there are strong correlations between all automatic metrics (note that ter is the only metric where a lower score is better). otherwise, the strongest correlations are between omissions and additions for all languages. in other words, a system that omits information that is in the data from the output is also likely to add information to the output that is not in the data. in terms of how automatic metrics on the one hand correlate with human-assessed measures on the other, the overall picture is that there are weak to medium correlations between all automatic metrics and fluency, additions and omissions, but not repetition. the weakness of the correlations between human and automatic metrics points to an interesting side-effect of building systems for under-resourced languages, in particular when using llms. in the past, automatic metrics which tend to assess quality as similarity to a set of test set reference outputs were good predictors of human assessments of outputs of systems which tended to be trained on another part of the same data set. for underresourced languages, training on reference outputs takes a back seat, hence evaluation on the basis of similarity to reference outputs is a far less effective predictor of human-assessed quality. this effect is exacerbated when using an llm out-of-the-box as in the case of the dcu-nlg-pbn system. ## conclusion reflecting on the webnlg 2023 results overall, it seems remarkable, especially when compared to the two previous webnlg iterations, that dcu-nlg-pbn, a system that uses gpt-3.5 plus google translate in zero-shot mode, with no task-specific adaptation at all, should emerge as the overall winner: its performance is not significantly different from the human-authored reference texts on any of the human-assessed performance measures, and the (automatic metric) margins by which it outperforms systems that have been trained specifically for this task are in many cases substantial. what is even more surprising is that in the five language/measure combinations where dcu-nlg-pbn is actually ranked higher than the humanwritten reference texts, the measure in question is never fluency (which llms are supposed to be particularly good at), but additions, omissions and repetitions (which llms are supposed to be particularly bad at). overall the shared task illustrates the limitations of current neural models when dealing with low resource languages. while fine-tuning existing encoder-decoders or decoders under-perform the nlg+mt pipeline, for languages such as breton, where machine translation is not available or low quality, the nlg+mt approach similarly shows poor results. in both cases, the scarcity of training data restricts the quality of the generated texts. approaches which have not been adopted by any of the participants but which would be interesting to explore include data augmentation, parameter efficient fine tuning and the development of languages models that are better attuned to specific language families. the shared task also highlights the practical difficulties involved in training and testing models for low resource languages. as our human evaluation illustrates, recruiting reliable experts to evaluate system outputs is challenging. similarly, creating development and test data is both costly and complex. we hope that the webnlg 2023 data will encourage and foster further research on generation into low resource languages.