enunlg: a Python library for reproducible neural data-to-text experimentation (2023)

Abstract: Over the past decade, a variety of neural architectures for data-to-text generation (NLG) have been proposed. However, each system typically has its own approach to pre- and post-processing and other implementation details. Diversity in implementations is desirable, but it also confounds attempts to compare model performance: are the differences due to the proposed architectures or are they a byproduct of the libraries used or a result of pre- and post-processing decisions made? To improve reproducibility, we re-implement several pre-Transformer neural models for data-to-text NLG within a single framework to facilitate direct comparisons of the models themselves and better understand the contributions of other design choices. We release our library at https://github.com/NapierNLP/enunlg to serve as a baseline for ongoing work in this area including research on NLG for low-resource languages where transformers might not be optimal.

## introduction dozens of different models for neural data-to-text generation have been proposed in the last decade, before we even consider recent efforts to repurpose large language models for data-to-text natural language generation (nlg). however, these models vary greatly with respect to both low-level and highlevel design choices, requiring different kinds of delexicalisation and normalisation processes, different ways of encoding and tracking meaning, and using a variety of neural network libraries, among other differences. while we can use the outputs of individual models released by their authors to assess the relative performance of these implementations, there is little work aiming to explore which performance differences are due to the proposed architectures themselves as opposed to other implementation details. in order to explore these differences, encourage reproduction experiments, and datasets models wen sclstm (described) e2e challenge sclstm (released) cleaned e2e tgen webnlg checklist * nmethodius charsclstm * our enunlg library is developed for python 3.9 with pytorch 1.9.1. in addition to implementing the models themselves, we provide a variety of file readers & writers to consume different corpora and convert them into appropriate representations for each model. at present, we have tools in place to work with the wen datasets (dialog system responses for restaurant, hotel, laptop, and tv descriptions: @xcite , cleaned data from the e2e challenge (restaurant descriptions @xcite , the nmethod-ius corpus (museum exhibits: stevens-guille et al., 2020), and webnlg @xcite . meaning representation (mr) parsers are included for cued dialogue acts, e2e slot-value pairs, and rdf triples. supported neural representations for these mr types include bit-vectors, flattened trees, and unbracketed sequences of triples. word embeddings can be randomly initialised or loaded from existing vectors. we reimplement the sclstm model proposed by @xcite , originally implemented using theano and python 2. during reimplementation, we found that the codebase released with the paper implemented a different architecture from what was described in the paper, so we provide both versions in our library. we also provide a reimplementation of tgen @xcite , originally implemented using tensorflow 0.6. @xcite implemented their checklist model in lua with torch and deriu and cieliebak (2018) used tensorflow 1.10.0 for their charsclstm. ## planned uses our goals in developing enunlg fall into three broad categories: reproducibility, pedagogy, and easy experimentation. by enabling the use of a single framework with consistent reference implementations of multiple models, the library promotes reproducibility and facilitates fair comparisons, controlling for differences in, e.g., delexicalisation, tokenisation, neural network libraries, etc. a small, consistent codebase that addresses the different elements of implementing neural data-to-text systems also serves a pedagogical function, providing a starting point for student projects. finally, our design choices aim to make engineering experiments trivial (e.g. hyperparameter search, changing tokenisation, etc) and scientific experiments easy (e.g. developing new end-to-end and pipeline systems for neural nlg) and promote work in low-resource nlg @xcite . ## conclusions we present enunlg, a library for reproducible experimentation in neural data-to-text generation. the code is available from https://github. com/napiernlp/enunlg . we hope that the availability of an extensible library for neural nlg will improve reproducibility in our research community and provide a new set of reference implementations for baseline models.