G round C ocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models (2025)

Abstract: The rapid progress of large language models (LLMs) has seen them excel and frequently surpass human performance on standard benchmarks. This has enabled many downstream applications, such as LLM agents, to rely on their reasoning to address complex task requirements. However, LLMs are known to unexpectedly falter in simple tasks and under seemingly straightforward circumstances - underscoring the need for better and more diverse evaluation setups to measure their true capabilities. To this end, we choose to study compositional and conditional reasoning, two aspects that are central to human cognition, and introduce GroundCocoa - a lexically diverse benchmark connecting these reasoning skills to the real-world problem of flight booking. Our task involves aligning detailed user preferences with available flight options presented in a multiple-choice format. Results indicate a significant disparity in performance among current state-of-the-art LLMs with even the best performing model, GPT-4 Turbo, not exceeding 67% accuracy despite advanced prompting techniques.

## introduction conditional and compositional reasoning are central to navigating and interacting with complex systems through decision-making processes @xcite . conditional reasoning refers to the understanding and application of logical rules, often structured in "ifthen" forms, which are fundamental to evaluating potential scenarios and anticipating outcomes in daily decision-making. compositional reasoning involves solving complex problems by integrating solutions to simpler sub-problems in a structured manner. this cognitive process is crucial for understanding the relationships between different components of a task. we evaluate how effectively current llms exhibit these cognitive abilities, which are essential for both human and artificial intelligence. to that end, we introduce groundcocoa foot_0 , a benchmark designed to assess compositional and conditional reasoning within a grounding task. set within a real-world inspired flight reservation scenario, groundcocoa comprises questions framed as user needs. finding and booking flights is a complex task where user requirements might be many and highly convoluted. while we use flight booking to illustrate our idea, the compositional primitives forming our user requirements test for general skills such as temporal reasoning (e.g., "i want a flight departing after 5 pm") and mathematical reasoning (e.g., "ticket price should be under @xmath05000?"), greater world knowledge (e.g., "i'd like to avoid layovers in europe") etc., which we leave for future work.