E con L ogic QA : A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning (2024)

Abstract: In this paper, we introduce EconLogicQA, a rigorous benchmark designed to assess the sequential reasoning capabilities of large language models (LLMs) within the intricate realms of economics, business, and supply chain management. Diverging from traditional benchmarks that predict subsequent events individually, EconLogicQA poses a more challenging task: it requires models to discern and sequence multiple interconnected events, capturing the complexity of economic logics. EconLogicQA comprises an array of multi-event scenarios derived from economic articles, which necessitate an insightful understanding of both temporal and logical event relationships. Through comprehensive evaluations, we exhibit that EconLogicQA effectively gauges a LLM’s proficiency in navigating the sequential complexities inherent in economic contexts. We provide a detailed description of EconLogicQA dataset and shows the outcomes from evaluating the benchmark across various leading-edge LLMs, thereby offering a thorough perspective on their sequential reasoning potential in economic contexts. Our benchmark dataset is available at https://huggingface.co/datasets/yinzhu-quan/econ_logic_qa.

## introduction logical reasoning is a pivotal skill in many professional and academic domains, enabling individuals to make informed decisions by understanding relationships between sequential events or pieces of information. in practice, the reasoning capabilities of large language models (llms) are frequently utilized across various applications, yet their effectiveness in logical reasoning remains underexplored. despite its importance, there is an evident gap in the literature regarding the capability of llms to perform logical reasoning at a high level. this paper addresses this gap by introducing econlogicqa, a new benchmark designed to rigorously assess the logical reasoning capabilities of llms specifically within the contexts of economics, business, and supply chain management. econlogicqa distinguishes itself from existing benchmarks by challenging llms to not only identify but also logically sequence multiple interconnected events from realistic economic scenarios. this approach aims to reflect the intricate decisionmaking processes required in these fields, going beyond mere fact recall or simple event prediction. by focusing on the sequencing of events based on logical rather than simply chronological order, econlogicqa inspects the llms' ability to engage with and understand the underlying mechanics of economic phenomena. the benchmark utilizes a curated dataset derived from a wide range of business news articles, guiding gpt-4 to generate multi-choice questions that demand an intelligent understanding of logical connections. a rigorous human review process ensures the accuracy and appropriateness of the content, refining the dataset to enhance its practical value. through comprehensive testing across various state-of-the-art llms, this paper not only demonstrates econlogicqa's effectiveness in evaluating logical reasoning but also provides insights into the potential improvements and applications of llms in complex reasoning tasks. our contributions of this paper are as follows: ## econlogicqa in this section, we detail the dataset generation and human review processes for creating the econ-logicqa benchmark and provide illustrative examples from it. ## experiments this section outlines experiments with the econ-logicqa dataset, assessing the sequential reasoning of multiple open and proprietary large language models (llms) in economic scenarios.