How Do S eq2 S eq Models Perform on End-to-End Data-to-Text Generation? (2022)

Abstract: With the rapid development of deep learning, Seq2Seq paradigm has become prevalent for end-to-end data-to-text generation, and the BLEU scores have been increasing in recent years. However, it is widely recognized that there is still a gap between the quality of the texts generated by models and the texts written by human. In order to better understand the ability of Seq2Seq models, evaluate their performance and analyze the results, we choose to use Multidimensional Quality Metric(MQM) to evaluate several representative Seq2Seq models on end-to-end data-to-text generation. We annotate the outputs of five models on four datasets with eight error types and find that 1) copy mechanism is helpful for the improvement in Omission and Inaccuracy Extrinsic errors but it increases other types of errors such as Addition; 2) pre-training techniques are highly effective, and pre-training strategy and model size are very significant; 3) the structure of the dataset also influences the model’s performance greatly; 4) some specific types of errors are generally challenging for seq2seq models.

## introduction data-to-text generation is a task of automatically producing text from non-linguistic input @xcite . the input can be in various forms such as databases of records, spreadsheets, knowledge bases, simulations of physical systems. traditional methods for data-to-text generation @xcite @xcite implement a pipeline of modules including content planning, sentence planning and surface realization. recent neural generation systems @xcite are trained in an end-to-end fashion using the very successful encoder-decoder architecture @xcite as their backbone. @xcite introduce a systematic and comprehensive comparison between pipeline and end-to-end architectures for this task and conclude that the pipeline models can generate better texts and generalize better to unseen inputs than end-to-end models. however, with the rapid development of the seq2seq models especially pre-trained models, more and more end-to-end architectures based on seq2seq paradigm get state-of-the-art results on data-to-text benchmarks nowadays. although bleu score @xcite , which is based on precision, has been improved dramatically on standard data-to-text benchmarks such as webnlg @xcite , @xcite and rotowire @xcite over the recent years, it is commonly accepted that, compared with human evaluation, bleu score can not evaluate the models very well. it is too coarse-grained to reflect the different dimensions of the models' performance and not always consistent with human judgment @xcite @xcite . moreover, existing human evaluations on data-to-text generation are usually limited in size of samples, numbers of datasets and models, or dimensions of evaluation. in this study, we aim to conduct a thorough and reliable manual evaluation on seq2seq-based endto-end data-to-text generation based on multiple datasets and evaluation dimensions. we want to know the pros and cons of different seq2seq models on this task, and the factors influencing the generation performance. particularly, following multidimensional quality metric(mqm) @xcite , similar to the job on summarization evaluation @xcite , we use 8 metrics on the accuracy and fluency aspects to count errors, respectively. therefore, compared with existing manual evaluation reports, it is more informative and objective. using this method, we manually evaluate several representative models, including transformer @xcite , transformer with pointer generator @xcite , t5(small&base) @xcite , bart(base) @xcite foot_0 . we test these models on four common datasets, including e2e @xcite , webnlg @xcite , wikibio @xcite , totto @xcite . thus we can discuss the effectiveness of the pre-training method, some essential techniques and the number of parameters. we can also compare the differences between datasets and how they influence the models' performance. empirically, we find that: there is some work evaluating and analyzing the data-to-text generation task. perez-beltrachini and @xcite propose a methodology to analyze the data-to-text benchmarks and apply their method to wikibio, rnnlg @xcite and im-agedesc @xcite datasets. ferreira et al. (2019) introduce a systematic comparison between pipeline and end-to-end architectures for neural data-to-text generation. @xcite propose a methodology for human to evaluate the accuracy of the generated texts. sequence-to-sequence seq2seq paradigm is a general and flexible paradigm that is typically implemented by an encoder-decoder framework. @xcite discuss sequence to sequence learning with neural networks. furthermore, there are some representative architectures that have been proposed such as recurrent neural network @xcite and transformer @xcite . seq2seq paradigm can be naturally applied to any task, as long as their input and output can be represented as sequences. therefore, there have been many attempts to apply seq2seq to different tasks. more recently, pretrained models based on seq2seq paradigm @xcite have proved their power on lots of tasks @xcite . there has been much work analyzing seq2seq models which is always task-specific and based on automatic or human evaluation. for example, @xcite analyze the common models' performance on summarization. to our knowledge, little work has been done to comprehensively evaluate the performance of seq2seq models on data-to-text generation. and much work is based on automatic metrics such as rouge or bleu which can be different from human evaluation as some work @xcite @xcite shows. therefore it is meaningful to manually evaluate representative seq2seq models on the data-to-text task. ## models and datasets we conduct experiments using five representative seq2seq models on four commonly used data-totext datasets and evaluate the generated texts accordingly foot_1 . note that we do not use models that are designed for specific data sets or data structures @xcite @xcite , but adopt models that allow inputs of different formats and structures, which brings convenience to comparison on different data sets. besides, most specific models for data-to-text generation are actually based on these typical seq2seq models @xcite , which also proves the rationality of our selection of these models. ## evaluation method we first evaluate models' performance using automatic metric bleu @xcite , and the bleu scores are comparable to the mainstream research. then, we use human evaluation similar to polytope @xcite to further analyze and evaluate the performance of the models on different datasets. bleu is a precision-based metric for evaluating the quality of generated text and it is widely used by work on data-to-text generation. multidimensional quality metric (mqm) (mariana, 2014) is a framework for describing and defining custom translation quality metrics. it defines flexible issue types and a method to generate quality scores. based on mqm, @xcite introduce an error-oriented fine-grained human evaluation method polytope. it defines five issue types about accuracy, three issue types about fluency, syntactic labels and three error severity rules. note that we do not use the syntactic labels in polytope, as they are not the focus of our evaluation in this study. the definitions of our evaluation dimensions are very similar to @xcite , but for the sake of the integrity of the paper and more specifically to the task of data2text, we still explain them below. after annotating every generated sentence with these error types and severity, we finally calculate an overall score to evaluate the model's performance. ## result analysis we evaluate the five models mentioned above on four datasets using the above metrics. the overall human evaluation score and bleu score of each model on each dataset are shown in ## bleu or human evaluation? we can see that the overall trend of the bleu score is consistent with human evaluation, which can basically reflect the overall performance of the model. and many conclusions we made above can also be proved by the bleu score. for example, the biggest pre-training model t5-base achieves the highest bleu score too among the selected models, pointer generator with copy mechanism still performs better than transformer and totto is still the most difficult dataset. although our primary goal is not to promote a human evaluation metric, our dataset with human annotations gives us a testbed to analyze the correlations and differences between automatic and human metrics. there have been a lot of discussions in the community about the unreliability of bleu metric. @xcite recommend not using bleu on text simplification. they found that bleu scores can neither reflect grammar nor the meaning of preservation. @xcite show that bleu and some other commonly used indicators are not well consistent with human judgment when evaluating nlg tasks. we compute the pearson correlation coefficients between bleu score and manual evaluation in terms of accuracy and fluency. we categorize the error types into accuracy and fluency aspects according to the definition in section 4.1, and use equation 2 to calculate accuracy score and fluency score respectively. the pearson correlation coefficient between bleu score and accuracy is 0.61 and in fluency aspect is 0.08. there is a huge gap between them and we can see that bleu can evaluate accuracy to a certain extent and it is poor at fluency. moreover, the bleu metric is too coarse-grained to reveal the model's specific problems, which enlighten us on how to improve the model. our result is consistent with views of other work. ## conclusion we empirically compared five representative seq2seq models on the data-to-text task using a fine-grained set of human evaluation metrics based on mqm. we aim to make a systematic and comprehensive evaluation and analysis on end-to-end seq2seq models for the data-to-text task. we analyze the effect of milestone techniques such as copy and pre-training, the influence of the dataset and model size and the models' performance in terms of different types of errors. our evaluation shows that pre-trained models can generate quite good texts. but there is still much room for improvement in this task. furthermore, the improvement of specific errors such as omission error and inaccuracy intrinsic error is also worth exploring in the future.