Revisiting Automated Evaluation for Long-form Table Question Answering (2024)

Abstract: In the era of data-driven decision-making, Long-Form Table Question Answering (LFTQA) is essential for integrating structured data with complex reasoning. Despite recent advancements in Large Language Models (LLMs) for LFTQA, evaluating their effectiveness remains a significant challenge. We introduce LFTQA-Eval, a meta-evaluation dataset comprising 2,988 human-annotated examples, to rigorously assess the efficacy of current automated metrics in assessing LLM-based LFTQA systems, with a focus on faithfulness and comprehensiveness. Our findings reveal that existing automatic metrics poorly correlate with human judgments and fail to consistently differentiate between factually accurate responses and those that are coherent but factually incorrect. Additionally, our in-depth examination of the limitations associated with automated evaluation methods provides essential insights for the improvement of LFTQA automated evaluation.

## introduction in the current landscape where decisions are increasingly driven by data, the utility of tabular data provides a well-organized and efficient means of presenting data, which is essential for informed decision-making processes @xcite @xcite . within this context, long-form table question answering (lftqa) has emerged as a vibrant area of research, bridging the gap between structured data and the comprehensive insights required in real-world scenarios @xcite . as illustrated in figure 1 , given the complex question and numerous data points in a table, an lftqa system must understand the relationships within the data and perform human-like table question answering (lftqa) task investigated in our work. recent studies highlight the exceptional performance of large language models (llms) in lftqa tasks @xcite @xcite . however, the reliable evaluation of llm-based systems in this domain remains a relatively unexplored area. unlike conventional text generation tasks, where automatic metrics such as bleu and rouge can somewhat effectively gauge the fluency and surface-level coherence of the generated text, lftqa demands a more nuanced assessment approach. these traditional metrics, primarily designed for shorter texts, often fall short in lftqa where the answers not only need to be contextually rich and structurally complex but also deeply rooted in logical reasoning derived from the underlying tabular data. they struggle to evaluate the logical structure and reasoning accuracy essential for long-form responses, as they do not account for the correctness of data interpretation or the ability to maintain a coherent argument over extended narratives. this limitation significantly impacts their utility in scenarios where the decision-making process relies heavily on the ac-curate and logical processing of structured data, necessitating the development of new metrics that can more effectively measure these critical aspects. our research demonstrates that existing automatic metrics are inadequate in distinguishing between high-quality, factually accurate answers and those that are merely coherent. this discrepancy is alarming because developers might choose suboptimal systems for real-world applications if they rely solely on automatic metrics to compare and rank different lftqa systems. to better investigate the automated evaluation methods for lftqa tasks, we have constructed a meta-evaluation dataset named lftqa-eval, consisting of 2,988 humanannotated examples. specifically, we gathered outputs from leading llm-based systems on the fe-taqa @xcite and qtsumm @xcite datasets. we then benchmarked existing automatic evaluation metrics for these tasks, leveraging our collected human annotations across two distinct dimensions: faithfulness and comprehensiveness. our experimental results demonstrate that all the examined automated metrics exhibit low correlations with human judgments, revealing their unreliability in determining the quality of llm-generated answers and comparing different llm-based systems. moreover, we conducted an in-depth analysis of the failures associated with automated evaluation methods, supplemented by illustrative examples that provide valuable insights into potential areas for enhancement. ## lftqa-eval construction to better investigate the automated evaluation methods for lftqa tasks, we have constructed a meta-evaluation dataset named lftqa-eval. the following subsections discuss the data collection methodology and annotation process. ## conclusion our exploration into the evaluation of llms for lftqa tasks reveals a significant gap between current automatic metrics and human judgment, particularly in assessing answer faithfulness and comprehensiveness. the insights from the lftqa-eval dataset highlight the need for more nuanced evaluation methods that align more closely with human evaluative standards. addressing this discrepancy is essential for advancing the reliability of lftqa systems and ensuring their practical utility in real-world scenarios.