The 2020 Bilingual, Bi-Directional W eb NLG + Shared Task: Overview and Evaluation Results ( W eb NLG + 2020) (2020)

Abstract: WebNLG+ offers two challenges: (i) mapping sets of RDF triples to English or Russian text (generation) and (ii) converting English or Russian text to sets of RDF triples (semantic parsing). Compared to the eponymous WebNLG challenge, WebNLG+ provides an extended dataset that enable the training, evaluation, and comparison of microplanners and semantic parsers. In this paper, we present the results of the generation and semantic parsing task for both English and Russian and provide a brief description of the participating systems.

## introduction the motivation behind the webnlg challenges is twofold. on the one hand, we seek to provide a common benchmark on which to evaluate and compare "micro-planners", i.e., natural language generation (nlg) systems which can handle the full range of micro-planning tasks including document structuring, aggregation, regular expression generation, lexicalisation and surface realisation @xcite . on the other hand, we are interested in building connections with research from the semantic web community which explores the relationship between knowledge bases (kbs) and natural language. there is a clear parallel between open information extraction (open ie) and rdf-based semantic parsing, and between rdfto-text generation and kb verbalisation. yet the interaction between nlp and semantic web research remains limited. by highlighting the nlp tasks involved in mapping rdf triples and natural language, we aim to stimulate cross-fertilisation between nlp and semantic web research. webnlg datasets align sets of rdf triples with text. while the 2017 webnlg shared task required participating systems to generate english text from a set of dbpedia triples @xcite , the 2020 webnlg+ challenge additionally includes generation into russian and semantic parsing of english and russian texts. thus the webnlg+ challenge encompasses four tasks: rdf-to-english, rdf-to-russian, english-to-rdf and russian-to-rdf. timeline. the training and development data was released on april 15, 2020, preliminary evaluation scripts on april, 30th and final evaluation scripts on may, 30th. the test data was made available on september, 13th and the deadline for submitting system results was september, 27th. automatic evaluation results were announced on october, 9th and the first version of the human evaluation results on november, 20th. the final version of the human evaluation results were released on november, 26th. results were first released anonymously so that participants had the opportunity to withdraw their systems. in what follows, we summarise the main features of webnlg+ 2020. section 2 describes the datasets used for the challenge. section 3 presents the participating systems. section 4 introduces the evaluation methodology, section 5 discusses the participants results in the automatic evaluation and section 6 in the human evaluation. finally, section 7 depicts the correlations between automatic evaluation metrics and human ratings as well as section 8 concludes with pointers for further developments. ## participating systems the webnlg+ data was downloaded more than 100 times, 17 teams submitted 48 system runs. from this sample, two teams withdrew their results, which gave us 15 participating teams with 46 runs for automatic evaluation ( in what follows, we summarise the primary submissions of the 15 participating teams. ## results of automatic evaluation in this section, we present the automatic scores on english and russian datasets for both tasks, namely, rdf-to-text and text-to-rdf. for english, we discuss the automatic scores, and make a distinction between results on (i) the entire dataset, (ii) seen semantic categories, (iii) seen semantic categories but unseen entities and (iv) unseen semantic categories. for russian, the only reported results are for the entire dataset, as the test set only contained seen entities and categories. ## results of human evaluation in this section, we describe the results of the human evaluation conducted on a sample of the outputs of rdf-to-text system submissions for both english and russian data. for english we evaluate systems' performance for (i) the full sampled test subset, (ii) subsets of each of the triple categories (seen categories, unseen entities, unseen categories). for russian, we provide results for the full sampled test subset only. the final results for all test data types are shown in in english. regarding text structure and fluency, results of neural approaches as fbconvai, amazonai (shanghai) and osu neural nlg were rated surprisingly high, sharing the same ranking cluster with the groundtruth references (webnlg-2020-ref). as expected, in terms of data coverage and relevance, the rule-based participating systems (rali-université-montréal, dangnt-sgu) performed quite strongly, being in the same cluster as the references. in fact, the relation between fluency and data coverage is noticeably different: although systems based on fine-tuned t5 and bart language models ranked on the top for fluency (amazon ai, fbconvai, osu neural nlg), the ones based on the latter language model (bart, fbconvai) suffered a drop in performance in terms of data coverage, whereas the ones based on the former (t5) performed similarly to the rule-based approaches. note that the scores for unseen categories are generally lower for all systems compared to the scores for seen categories and unseen entities. overall, while rule-based systems (rali-université-montréal, dangnt-sgu) perform well for the criteria which evaluate connection between rdf triple and the text (data coverage, relevance, correctness), for the categories which evaluate naturalness and russian. ## correlation between automatic and human evaluation metrics tables 15a and 15b describe the sentence-level pearson correlations of the evaluation metrics for the rdf-to-text task in english and russian, respectively. novel learned metrics, such as bertscore and bleurt, seem to correlate more with the human evaluation ratings than traditional token-and character-overlapping metrics, such as bleu, meteor, chrf++ and ter. for the english evaluation, bleurt, the newest metric, was the one that best correlated with the human evaluation ratings, especially with correctness and fluency. for russian, bertscore was the automatic metric that best correlated with the human ratings, except for data coverage, with which chrf++ correlated the most. this latter was one of the character-and n-gram metrics that correlates more with human ratings. this report described the data, participating systems, results and findings of the 2020 bilingual, bi-directional webnlg+ shared task. the shared task of this year involved two tasks: rdf-to-text and text-to-rdf. in the following sections, we describe the main findings for each task conducted on this version of the shared task.