A General Benchmarking Framework for Text Generation (2020)

Abstract: The RDF-to-text task has recently gained substantial attention due to the continuous growth of RDF knowledge graphs in number and size. Recent studies have focused on systematically comparing RDF-to-text approaches on benchmarking datasets such as WebNLG. Although some evaluation tools have already been proposed for text generation, none of the existing solutions abides by the Findability, Accessibility, Interoperability, and Reusability (FAIR) principles and involves RDF data for the knowledge extraction task. In this paper, we present BENG, a FAIR benchmarking platform for Natural Language Generation (NLG) and Knowledge Extraction systems with focus on RDF data. BENG builds upon the successful benchmarking platform GERBIL, is opensource and is publicly available along with the data it contains.

## introduction nlg is the process of generating coherent natural language text from non-linguistic data @xcite . a large number of approaches with distinct inputs have been employed for nlg systems over the last years @xcite . after having been addressed in only a few papers at the beginning of the last decade @xcite , the generation of natural language from resource description framework (rdf) data has gained substantial attention @xcite . the rdf-totext task has hence been proposed to investigate the quality of automatically generated texts from rdf knowledge graphs (kgs) @xcite . recent studies have focused on comparing systematically neural pipeline and end-to-end data-to-text approaches for the generation of text from rdf triples @xcite . however, a transparent comparison of rdfbased nlg systems is costly and prone to failure when relying only on benchmarking datasets such as webnlg without a proper benchmarking platform. recent works have hence proposed evaluation tools for text generation such as vizseq @xcite and mt-compareval @xcite . however, none of these tools abides by the fair principles @xcite , which are now widely regarded as a key first step to ensure reproducible research in scientific experiments. moreover, none of the tools aforementioned involves rdf data for the knowledge (relations and entities) extraction task (ke). in this paper, we address this gap by presenting beng, a fair benchmarking platform for nlg and knowledge extraction systems. beng is available as an online instance with a user-friendly interface that can be freely used by researchers to benchmark their systems without the need to set up the benchmarking platform by themselves. moreover, beng is an open-source project which can be extended (w.r.t. metrics and systems) and executed locally. 1,2 our benchmarking results show that beng can foster the development of nlg approaches by providing an easy way to access, compare, and reuse results among nlg systems. moreover, beng can support the investigation of multilingual approaches as it contains variations of the webnlg datasets in languages other than english (german and russian). ## related work a significant body of research has been devoted to providing evaluation tools for text generation approaches. recently, @xcite proposed vizseq as a visual analysis toolkit for independent text generation tasks, for example, machine translation (mt) or nlg. vizseq supports multimodal (images, videos, texts) sources and multiple text references to provide a detailed picture of system evaluations. in mt, compare-mt @xcite and mt-compareval @xcite are related tools for comparative analysis with automatic measures that provide a high-level view of major differences between mt outputs. in turn, vis-eval metric viewer @xcite and ibleu @xcite present metric scores as a visual interface. other tools focus on the interpretability of the text generation process and language model parameters such as the opennmt visualisation tool @xcite , lm @xcite , and seq2seq @xcite . although mt and nlg tasks rely on the same metrics for evaluating their outputs, none of the aforementioned tools rely on fair principles for the sake of reproducible research. therefore, beng is the first evaluation tool that abides by the fair principles for the text generation task. ## framework beng addresses the problem of comparing different nlg systems using automatic metric results while relying on fair principles. it is based on a service-oriented architecture that reuses components from the fair benchmarking platform ger-bil @xcite , a benchmarking platform for named entity recognition and entity linking systems. we chose gerbil because it has already been used successfully in more than 80,000 experiments. we reuse the mechanisms implemented by gerbil to handle large experiments, generate experiment uniform resource identifiers (uris) and store the results of experiments in a database. some of the main components of gerbil (e.g., annotation systems, datasets, matching process, and performance metrics) are replaced by components which abide by the requirements of the evaluation of nlg systems. beng follows the fair guiding principles for scientific data management and stewardship @xcite as detailed in to cater for the needs of nlg experiments, beng provides different experiment configurations, which are explained in the following along with their parameters. ## conclusion in this paper, we introduced beng, a general benchmark framework for text generation based on gerbil's service-oriented architecture. we integrated new experiment types, datasets, and measures. the main advantages of beng are that it follows the fair guiding principles and provides a web-based frontend that allows for several use cases enabling lay people and expert users to perform informed comparisons of annotation tools. in future work, we plan to include other popular nlg benchmarks such as e2e and sr @xcite @xcite and extend the experiment types as well as include the web-services for models instead of uploading the hypotheses.