ACCORD : Closing the Commonsense Measurability Gap (2025)

Abstract: We present ACCORD, a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. ACCORD introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. Uniquely, ACCORD can automatically generate benchmarks of arbitrary reasoning complexity, so it scales with future LLM improvements. Indeed, our experiments on state-of-the-art LLMs show performance degrading to below random chance with only moderate scaling, leaving substantial headroom for improvement. We release a leaderboard of the benchmark suite tested in this work, as well as code for automatically generating more complex benchmarks.

## introduction llms perform remarkably well on diverse reasoning tasks @xcite . however, more detailed analysis reveals their reasoning remains unreliable @xcite @xcite . llm reasoning often lacks robustness to simple lexical triggers @xcite or irrelevant context @xcite . in chain-of-thought (cot) approaches, llms can decompose problems into illogical or irrelevant reasoning chains @xcite @xcite and can systematically rationalize incorrect conclusions based on a simple reordering of answer choices @xcite . furthermore, these limitations appear to worsen as the complexity of a reasoning task increases @xcite . separately, llms are increasingly being used as knowledge bases (kbs) @xcite given their impressive ability to store knowledge in model parameters. however, such parametric knowledge incurs limitations @xcite @xcite . for example, @xcite report that, given the context "elon musk [...] is the owner and ceo of twitter", gpt-3.5 answered the question "who is the ceo of twitter?" with "jack dorsey" (the previous ceo), since musk became ceo only after gpt-3.5's september 2021 knowledge cutoff. grounding of llms with contextual knowledge during inference has become a leading paradigm (e.g., retrieval-augmented generation). however, as the above example illustrates, although their parametric knowledge can become outdated @xcite @xcite @xcite , llms tend to integrate contextual knowledge unfaithfully @xcite @xcite , especially when parametric and contextual knowledge directly conflict @xcite @xcite @xcite @xcite . unfortunately, this so-called 'context unfaithfulness' confounds empirical measurements of reasoning, leading to construct validity concerns @xcite @xcite . indeed, llms can circumvent the intended objective of various reasoning tasks using inductive biases from parametric knowledge @xcite , dataset artifacts @xcite , and other spurious shortcuts @xcite @xcite . more formally, borrowing from @xcite , we conceptualize a default world model, w def , as the set of conditions and assumptions underpinning the vast majority of existing llm training data. under this view, ## accord framework accord generates an anti-factual benchmark from an existing factual commonsense dataset. in this work, we apply accord to commonsenseqa (csqa) @xcite to create the accord csqa benchmark suite. we chose csqa due to its popularity. however, accord is equally applicable to other commonsense reasoning datasets (see appendix a for examples). accord's algorithm is complex. figure 2 serves as a concrete running example throughout this section. significant additional detail and explanation is given in appendices b, c and d. in particular, figure 4 in appendix b illustrates an abstract view of the accord framework using tree data structures. these two figures are highly complementary and we encourage the reader to refer to both. consider a question-answer (qa) instance from csqa with question q and answer choices @xmath0, a 2 , . . . , a q } (see figure 2a ). as discussed in §1, because csqa is w def -aligned, llms can spuriously circumvent the reasoning required to arrive at the factual answer a @xmath1. to overcome this hurdle, we introduce an anti-factual context c af , which consists of statements aligned to some antifactual world, w af ̸ = w def (see figure 2f ). specifically, c af is constructed to ensure some alternative answer choice a af ̸ = a def can also be implied via negation (see figure 2g ). the crux of accord lies in the careful design of variants of c af to systematically examine llm performance (see appendix b.1 for motivating design details). ## experiments we apply accord to csqa to create accord n csqa , a benchmark suite where @xmath2. for each instance in the csqa development set, we infer the conceptnet relation that generated the instance by computing the majority vote among all conceptnet assertions matching one of its answer choices to its source concept (see @xcite for details). the resulting distribution is highly skewed, with 607 instances of conceptnet's atlocation relation (the most popular) compared with 13 instances of isa (the least popular). to balance accord csqa , we randomly subsample each relation type, keeping only 13. of the 6 × 13 = 78 instances, we reject two due to inciting violence, and another two where we cannot craft a valid pairing template. the baseline, accord 0 csqa , consists of the remaining 74 instances. we hand-craft 1-3 pairing templates for these 74 instances, resulting in 93 pairing templates. from these 93 pairings, we generate rea-soning trees of sizes 1-5, producing a suite of benchmarks, accord 1-5 csqa , with 245,514 unique total trees and a guaranteed minimum of 143 unique trees per pairing. to reduce computational costs while ensuring wide coverage, we subsample the trees to keep precisely one tree per unique combination of reasoning hops and distractors per pairing, which produced a smaller benchmark with 2,864 unique trees. this step also balances the dataset, which is otherwise exponentially weighted towards the larger tree sizes (since there are exponentially more unique trees as the size increases). we present results from this smaller benchmark in figure 3 , but release both versions (see appendix e for additional details). figure 3 shows the performance of gpt-4o (2024-05-13) (openai, 2024a), llama-3-70b-instruct (ai@meta, 2024b), and mixtral-8x22b-instruct-v0.1 (the mistral @xcite on accord csqa in a zero-shot setting. the "no context" baseline represents accord 0 csqa . ## results and discussion recall that, unlike prior commonsense benchmarks, accord quantifies performance as a function of problem size, t , reasoning hops, n, and distractors, d, where @xmath3.4). simultaneously, accord disentangles (contextual) grounding and (parametric) reasoning effects through ceteris paribus comparisons of factual and anti-factual reasoning task variants (see §2.7). are llms good reasoners or merely good parroters? for any given n (columns 1 and 3) or d (columns 2 and 3) in figure 3 , factual reasoning significantly outperforms anti-factual reasoning. since we are carefully controlling for all other factors, this performance gap is directly indicative of context unfaithfulness: llms are inductively biased towards the factual answer choice. in fact, anti-factual performance quickly degrades to below random chance with only very moderate scaling of reasoning complexity. this suggests that as soon as their multi-hop reasoning capacity is exceeded, llms almost exclusively resort to spurious shortcuts. importantly, this performance gap is evidence of low construct validity in many factual commonsense reasoning datasets: the intended measurement of llm reasoning ability is overwhelmingly confounded by llm parroting ability. what is the effect of reasoning hops vs distractors? for @xmath4, both factual and anti-factual performance degrade rapidly with increasing n. this trend has been understood in formal reasoning @xcite , but we demonstrate it here for the first time with commonsense. interestingly, the trend for d (column 2) is reversed, which goes against expectations @xcite . column 3 explains this artifact, which occurs because reasoning hops and distractors are complementary for a given problem size (@xmath5, and, as such, interact very strongly. in column 3, when controlling for distractors, reasoning hops have a dominant effect on the trend. on the other hand, increasing distractors only slightly shifts the entire trend line downward. in other words, given a fixed t (i.e., a fixed context budget), llms tend to prefer a smaller n and larger d than the reverse. that is, the ability of llms to filter out distractors significantly outclasses their multi-hop reasoning capacity. marginalizing over reasoning hops (column 2) obfuscates this nuance, resulting in a reversed trend. being able to disentangle these effects is one of our key contributions. ## conclusion we presented accord, a framework for generating anti-factual commonsense reasoning benchmarks. accord introduces formal elements to commonsense reasoning, and thus takes a significant step towards closing the commonsense measurability gap with respect to formal reasoning. in particular, accord disentangles commonsense grounding and reasoning abilities in llms, while controlling for both reasoning complexity, reasoning skills, and distractors. experiments on our accord csqa benchmark suite, an application of the accord framework to csqa, demonstrate that the performance of state-of-the-art llms degrades to random chance with only moderate scaling, leaving substantial room for improvement. moreover, we demonstrate a significant gap between factual and anti-factual performance. this highlights the construct validity concerns of typical (factual) benchmarks, which unfortunately allow llms to circumvent the intended reasoning task with parametric parroting of the factual answer. accord is uniquely designed to automatically scale its difficulty level in tandem with future llm improvements by leveraging compositional scalability to generate future benchmarks of arbitrary reasoning complexity with minimal additional human effort.