Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in T o TT o (2024)

Abstract: Neural Table-to-Text models tend to hallucinate, producing texts that contain factual errors. We investigate whether such errors in the output can be traced back to problems with the input. We manually annotated 1,837 texts generated by multiple models in the politics domain of the ToTTo dataset. We identify the input problems that are responsible for many output errors and show that fixing these inputs reduces factual errors by between 52% and 76% (depending on the model). In addition, we observe that models struggle in processing tabular inputs that are structured in a non-standard way, particularly when the input lacks distinct row and column values or when the column headers are not correctly mapped to corresponding values.

## introduction table-to-text generation refers to the task of generating natural language descriptions from tabular data @xcite and is widely used in several application domains such as medical diagnosis @xcite , financial @xcite and weather reporting @xcite @xcite and sports summaries @xcite . neural language models are known to generate fluent texts @xcite but may generate outputs that are factually incorrect or unrelated to the provided input data. such undesirable generation is called 'hallucination' @xcite @xcite . previous studies on table -to-text tasks adopt traditional seq2seq methods to generate table descriptions @xcite @xcite . recently, transformer based models @xcite @xcite have shown remarkable progress in language generation from textual input @xcite , however tabular data still needs more improvement to control hallucinations @xcite . neural models struggle with tabular data, especially when the inputs do not have distinct cell values from rows and columns mapped along with their respective headers. these input problems lead the model to generate more factual errors @xcite . using the totto tabular dataset @xcite , we identify and address input problems that are responsible for factual errors. some common tabular input problems in the totto are i. 'nonatomic' cell values, where a column contains multiple values such as leader name, party name and % of votes in one cell rather than a single indivisible value, ii. missing important cell values in the input (see in this paper, we score the quality of output texts by manually annotating output errors instead of using automatic evaluation metric scores such as bleu @xcite , rouge @xcite , parent @xcite and bleurt @xcite . we conducted a pilot study, where we fine-tuned t5-base and t5-large models @xcite , analysing 1,677 politics domain texts from the totto dataset through manual error analysis adopted from @xcite . these manual error annotations allowed us to identify patterns of errors in the generated text which were then traced back to input problems. our approach is summarised as follows: i. we systematically correct the tabular inputs for the politics domain in totto to adhere to a standard form to ensure the generation of factual texts from neural models. the correction procedure is elaborated in sec. 5.2 and is supplemented by pseudocode in app. d. tailoring zero-shot prompts for specific input and error annotation on 160 texts showed that correcting input reduces factual errors by 52% in llama 2-7b and 76% in llama 2-13b (sec. 6.2). ii. the manual error annotation methodology adopted from @xcite is detailed in app. b; this builds on the work of @xcite for totto politics domain outputs 2 . the inter-annotation agreement on the error annotation was good, with a fleiss' kappa of 0.622 (sec. 7). ## input problems due to the practical challenges involved in improving the tabular input for the entire dataset, which includes unique headers and tabular structures for each input, our focus is on analyzing a subset of samples containing errors. we examined 124 error samples from the t5-b and 86 error samples from t5-l models within the totto politics domain, as identified in ## results and discussion we manually annotated the outputs from all models working on corrected input data, following the same procedure outlined in sec. 3.1 and defined in app. b. in this section, we summarize the results of the total input data corrections and discuss them. the previous section also described and discussed error analysis locally while presenting individual corrections. ## inter-annotation experiment one of the authors manually annotated errors in 1,508 outputs before input correction and 169 problematic outputs after correction (a total of 1,677 from both t5 models). similarly, the annotation for both llama 2 models covered 160 outputs before and after input correction. two additional annotators annotated 60 outputs each, generated by four different models both before and after input correction. we provided them with a detailed document that included definitions of error categories, guidelines, tabular inputs and output texts for error annotation 5 . annotators fol5 we release our error annotations from our human lowed the guidelines and marked the errors. each annotator spent approximately 3 hours to complete this experiment. the annotated errors are shown in a confusion matrix in ## conclusion this paper presented a study that quantitatively demonstrates that fixing input problems such as insufficient data and data records containing nonatomic content improves the factual accuracy of output text by as high as 76% for one of the study models. correcting inputs also seems to improve the number of entirely error-free output texts. however, we still need to investigate why errors categorised as 'omissions' increase after input corrections. in our future work, we aim to explore other tabular datasets for problems with input data and study the generalization of the fixes explored in the current work.