Differences in Semantic Errors Made by Different Types of Data-to-text Systems (2024)

Abstract: In this paper, we investigate how different semantic, or content-related, errors made by different types of data-to-text systems differ in terms of number and type. In total, we examine 15 systems: three rule-based and 12 neural systems including two large language models without training or fine-tuning. All systems were tested on the English WebNLG dataset version 3.0. We use a semantic error taxonomy and the brat annotation tool to obtain word-span error annotations on a sample of system outputs. The annotations enable us to establish how many semantic errors different (types of) systems make and what specific types of errors they make, and thus to get an overall understanding of semantic strengths and weaknesses among various types of NLG systems. Among our main findings, we observe that symbolic (rule and template-based) systems make fewer semantic errors overall, non-LLM neural systems have better fluency and data coverage, but make more semantic errors, while LLM-based systems require improvement particularly in addressing superfluous.

## introduction human evaluation remains the gold standard to determine the quality of texts generated by natural language generation (nlg) systems @xcite . one aspect of human evaluation is error analysis, where researchers identify and categorise errors in system outputs. ideally, it is achieved by manually annotating output text in a multiple-annotators setting @xcite . although labour intensive, error analysis can provide a healthy dose of skepticism and help to ensure systems have the functionality intended @xcite . semantic errors, including missing, added or repeated content, are common in current language generation outputs, particularly for neural methods @xcite . documenting and analysing these errors in different types of systems helps in understanding specific faults within system output that we can look to address with improved models in a way that per-system scores do not. in the work reported in this paper, we start by obtaining word-span error annotations of semantic errors in a variety of data-to-text system input/output pairs. we then analyse the annotations to determine how many semantic errors different (types of) systems make, and what specific types of errors they make, and thus to get an overall understanding of semantic strengths and weaknesses among various types of nlg systems. our specific contributions are as follows: the paper is organised as follows. section 2 presents related work. section 3 describes the experimental design in detail. section 4 outlines the overall experiment set-up. section 5 presents results and analysis. section 6 offers a discussion. section 7 concludes with a summary and future directions. the appendices include the participant recruitment email, feedback from pilot participants, annotation steps, and additional results tables and analyses. data and resources are on github. foot_0 ## related work many human evaluations of data-to-text systems only score or label outputs at the sentence or paragraph level. if this is all that is known about output quality, finer errors and nuances often go undetected and therefore unaddressed. reporting wordspan level semantic errors found in nlg system outputs is necessary for in-depth error analysis and understanding of the factors contributing to such errors, so that solutions can be tailored to specific error types. @xcite propose to measure semantic accuracy of data-to-text generation using a neural model pre-trained for natural language inference (nli). human annotators used a threepoint likert scale to compare their results to the crowd-sourced human ratings @xcite . @xcite propose an omission and hallucination detector for texts generated by neural data-to-text systems in the meteorology domain, and performed expert analysis with the aim of classifying these errors by severity, taking domain knowledge into account. @xcite introduce the hallucination evaluation benchmark (halueval) to assess hallucination errors in llms using human-annotated samples, aiming to improve the models' accuracy in recognising hallucinations. human annotators used yes/no labels to annotate whether chatgpt responses contained hallucinated content. thomson et al. report different error types in nlg system outputs @xcite . the shared task on evaluating accuracy @xcite focuses on both manual and automatic techniques to evaluate the factual accuracy of texts generated by neural nlg systems. popovic et al. report error analyses by taking word span into account to evaluate inter-annotator agreement in mt outputs @xcite . @xcite focus on detecting semantic errors in model outputs by comparing the generated text to the input data. errors are annotated at word-level, with every word in the output text being considered a potential source of error. this is the most comparable work to ours, although they do not annotate the input since they do not address omission. to the best of our knowledge, none of the other publications report performing word span annotations of semantic errors in input and system output pairs from different data-to-text systems. ## results and analysis in this section, we present our results and analysis. we report the raw error counts ( ## discussion correlation and dependency insights. we observe a strong positive correlation between omissions and additions across different types of systems, perhaps indicating a common underlying cause for these errors where they do occur. notably, neural systems (both llm and non-llm) exhibit this trend, perhaps suggesting that when these systems fail to include expected elements, they overcompensate by adding unexpected ones. distinct system type behaviours. rule-based systems show a strong negative correlation between repetitions and omissions, and have a higher tendency towards addition errors than the other two, possibly because they struggle with precision in filtering out unnecessary items despite their intended factual accuracy @xcite . on the other hand, the neural systems all have strong positive correlations between omissions and additions. non-llm neural systems show the highest such correlation, emphasising the need for robust training and error-mitigation strategies. impact of seen vs. unseen data. all error rates are higher in data containing unseen properties than in data containing only seen. we observe a clear trend where systems perform better on familiar (seen) data across all error types. this is consistent with the expectation that models or systems are generally more accurate when dealing with data they have previously encountered. the considerably higher error rates for the unseen category indicate that systems' cannot transfer all learning to unseen data. this is particularly evident in the substantial increase in omission and repetition errors, suggesting that the underlying model may require further training or fine-tuning to improve its generalisation capabilities. in contrast, addition errors show a smaller increase. ## conclusion and future work in the work presented in this paper, we conducted a manual word-span annotation experiment with the aim of investigating the different types and numbers of semantic errors observed in the texts generated by 15 table-to-text generation systems, namely 13 webnlg 2020 systems and two more recent llm-based systems. we have described the error types, instructions for the evaluation and set up of experiments we used for this purpose. we have presented an analysis of the absolute numbers of errors made by different systems, and the error rates observed relative to input size, system type and unseen vs. seen properties. among our findings, we observed high correlation between omission and addition errors, higher correlations between omission and addition errors in neural systems, and higher error rates in the unseen category compared to seen for for all error types. overall, we found that the symbolic (rule and template-based) systems are more semantically consistent with the input. non-llm neural systems achieve higher fluency and data coverage but need careful management of semantic errors, while llm-based systems require improvement particularly in addressing over-generation (additions) and missing content (omissions). among these results the particularly high addition error rate of llm systems (0.305) stands out. these observations pinpoint future directions for what to focus on in improving output quality in different types of systems. • the evaluator should be careful while selecting the word span when marking an error. the evaluator should select complete tokens, i.e., words in the text, that are delimited by whitespace. for example, the selection for 'president' in 'joe biden is the president of the united states.' is correct, but selecting just 'pres' is not correct, as in 'joe biden is the president of the united states.' similarly, 'joe_biden | president | united_states' is correct, but 'joe_biden | president | united_states' is not. for example, consider the input triple "alessio_romagnoli | youthclub | a.s._roma" and the corresponding verbalisation "alessio romagnoli plays for the a . s . roma youth team." here 'plays for' can be inferred from the presence of 'youthclub' in the input triple. this is considered valid/correct and should not be marked as an error.