Error Analysis of T o TT o Table-to-Text Neural NLG Models (2022)

Abstract: We report error analysis of outputs from four Table-to-Text generation models fine-tuned on ToTTo, an open-domain English language dataset. We carried out a manual error annotation of a subset of outputs (a total of 3,016 sentences) belonging to the topic of Politics generated by these four models. Our error annotation focused on eight categories of errors. The error analysis shows that more than 46% of sentences from each of the four models have been error-free. It uncovered some of the specific classes of errors; for example, WORD errors (mostly verbs and prepositions) are the dominant errors in all four models and are the most complex ones among other errors. NAME (mostly nouns) and NUMBER errors are slightly higher in two of the GeM benchmark models, whereas DATE-DIMENSION and OTHER categories of errors are more common in our Table-to-Text model. This in-depth error analysis is currently guiding us in improving our Table-to-Text model.

## introduction end-to-end neural table -to-text models produce outputs that suffer from hallucination (output texts contain parts that are not supported by input data). this may be because these models learn the noise from complex examples during the training process and produce more errors than rule-based systems @xcite . the automatic metrics such as bleu and rouge do not uncover common classes of errors and are therefore less helpful to improve the models @xcite . the accuracy evaluation shared task by @xcite using the gold standard methodology proposed by @xcite was successful in identifying errors that are difficult to detect using automatic metrics @xcite . in this paper, we performed a detailed error analysis, adopting the @xcite methodology on four table -to-text model outputs (trained on the totto dataset) to identify and group the errors these models make in the output text. we created one of these table -to-text models by fine-tuning a t5-base text-to-text model with the totto dataset using bleu as a validation metric with the standard cross-entropy objective function, and we will be applying error corrections to this model in our future work. the other three model outputs came from gem benchmark table -to-text models fine-tuned from t5-small, t5-base, and t5large text-to-text models @xcite . previous research studies for error analysis predominantly focused on machine translation (mt) systems using a simple framework by @xcite and extensively using multidimensional quality metrics (mqm) framework by @xcite , and on other nlg tasks @xcite . other annotation methods have been used to check for errors in the totto dataset. @xcite use a method based on mqm, assigning multiple labels to individual sentences. we take a different approach, annotating at the more granular level of token spans, i.e., words or phrases. ## related work in the field of machine translation, error analysis has been carried out for a long time @xcite . more recently, the multidimensional quality metrics (mqm) framework based on a hierarchy of errors was applied to carry out error analysis of wmt data @xcite . this analysis identified error types (error classes) responsible for the difference in output quality between human and machine-generated translations. within the nlg context, @xcite performed error analysis for the topic-to-essay nlg task and proposed a human annotation framework for evaluating sub-sentence grammar, sentence logic, repetition, semantic coherence and contextual consistency. their experiment results show that the neural models produced relatively high semantic errors compared to the grammatical and repetition errors. within the table -to-text context, thomson and reiter (2020) designed a gold-standard error analysis methodology with a taxonomy of simple errors for the annotators to evaluate the factual accuracy of table -to-text nlg models. they apply this methodology to system-generated basketball summaries from the rotowire dataset @xcite . @xcite described a shared task where different evaluation techniques for basketball summaries from the rotowire dataset were submitted and their results show that metricbased techniques struggled to detect factual errors. ## conclusion we fine-tuned our neural table -to-text model (m1: bleu) with the known configuration details and compared its outputs with the gem benchmark model outputs. this analysis provided additional insights of error classes (such as incorrect verb predictions for word errors, name and num-ber swap when two or more of these details are in the input table, hallucinations for number and date_dimension), which is not possible to determine from evaluation metric scores. our analysis shows that these four transformer based models can perform textual reasoning to some extent but lack a deeper level reasoning capabilities (for example, mathematical reasoning and for more complex table structure when multiple inputs are present). this level of insights from the manual error analysis will provide opportunities to overcome this reasoning capabilities in our model in the future work.