D oc M ath-Eval: Evaluating Math Reasoning Capabilities of LLM s in Understanding Long and Specialized Documents (2024)

Abstract: Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DocMath-Eval, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing specialized documents containing both text and tables. We conduct an extensive evaluation of 48 LLMs with Chain-of-Thought and Program-of-Thought prompting methods, aiming to comprehensively assess the capabilities and limitations of existing LLMs in DocMath-Eval. We found that even the current best-performing system (i.e., GPT-4o) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe that DocMath-Eval can serve as a valuable benchmark for evaluating LLMs' capabilities in solving challenging numerical reasoning problems within expert domains.

## introduction recent advancements in large language models (llms) have attracted significant attention due to their capabilities in solving a broad range of tasks (openai, 2023; ai@meta, 2024), including math word problems (mwps) commonly found in academic exams @xcite @xcite @xcite @xcite . these mwps vary from basic arithmetic to advanced algebra, showcasing llms' proficiency in numerical reasoning -a crucial skill for interpreting and manipulating numerical data across various contexts. despite first, we know from the table that the values of debt in 2021 and 2022 are xxx and xxx, respectively. we then calculate the (...abbreviate…) therefore, the final answer is 17.3% ## docmath-eval in this section, we first offer a formal definition of the docmath-eval task. we then explain the rationale and methodology for adopting python program as the standardized solution format for docmath-eval. subsequently, we detail the data annotation process used to construct the challenging dm complong evaluation set, as well as the data re-annotation process for compiling the other three evaluation sets. ## experiment setup this section discusses the experiment setup, including the evaluated llms, prompting methods, and our implementation details. ## results and analysis we next discuss our main findings from the experiments and our analysis of the dm complong subset. ## conclusion this paper introduces docmath-eval, a comprehensive benchmark designed to evaluate the capabilities of llms in numerical reasoning over long and specialized documents. our experiments show that even the best-performing current models still fall short of human expert performance on problems requiring complex reasoning over extended contexts. this highlights the need for future research to improve llms' proficiency in complex numerical reasoning tasks within expert domains. we present the percentage of samples that received an average score of 4 or higher, as an indicator of the annotation quality of docmath-eval.