T urn B ench- MS : A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models (2025)

Abstract: Despite impressive advances in large language models (LLMs), existing benchmarks often focus on single-turn or single-step tasks, failing to capture the kind of iterative reasoning required in real-world settings. To address this limitation, we introduce **TurnBench**, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the “Turing Machine Board Game.” In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds. This dynamic setup requires models to reason over time, adapt based on past information, and maintain consistency across steps—capabilities underexplored in current benchmarks. TurnBench includes two modes: *Classic*, which tests standard reasoning, and *Nightmare*, which introduces increased complexity and requires robust inferential chains. To support fine-grained analysis, we provide ground-truth annotations for intermediate reasoning steps. Our evaluation of state-of-the-art LLMs reveals significant gaps: the best model achieves 84% accuracy in Classic mode, but performance drops to 18% in Nightmare mode. In contrast, human participants achieve 100% in both, underscoring the challenge TurnBench poses to current models. By incorporating feedback loops and hiding task rules, TurnBench reduces contamination risks and provides a rigorous testbed for diagnosing and advancing multi-step, multi-turn reasoning in LLMs.

## introduction reasoning is central to human cognition and a key benchmark for evaluating the capabilities of artificial intelligence (ai) systems @xcite . in the context of large language models (llms), assessing reasoning ability is especially critical as these models are increasingly deployed in complex, realworld tasks. while a growing body of work has proposed datasets and evaluation methods for probing llm reasoning @xcite @xcite , such as first, most existing evaluations focus on singleturn or single-step reasoning tasks, overlooking the iterative and interactive nature of real-world problem-solving. human reasoning often involves cycles of information gathering, hypothesis testing, and adaptation to feedback. this is especially true in scenarios where information is incomplete or distributed across multiple interactions. while recent benchmarks attempt to assess multi-step reasoning @xcite , they rarely simulate settings that require reasoning across multiple turns. second, current evaluation metrics typically emphasize final-answer correctness, with little insight into the model's intermediate reasoning process @xcite . as complex reasoning often admits multiple valid paths, simply scoring final outputs fails to distinguish between genuine inference and lucky guesses. though some methods attempt process-level evaluation via manual annotation or automated proxies @xcite , these are limited by subjectivity and the absence of reliable ground truth for intermediate reasoning. third, data contamination poses a serious concern. static benchmarks-often sourced from public datasets or templated questions-can overlap with pretraining corpora, making it difficult to disentangle memorization from actual reasoning @xcite @xcite . this undermines the reliability of benchmark results and inflates perceived model performance. to address these gaps, we introduce turnbench, a novel benchmark designed to evaluate multi-turn, multi-step reasoning through an interactive codebreaking task inspired by the turing machine board game. in this game, a model must uncover a hidden three-digit code by engaging in multiple rounds of interaction with logical verifiers. each verifier is governed by a hidden rule; only one rule per verifier is active in a given instance. to succeed, the model must iteratively guess codes, select verifiers, analyze feedback, and gradually infer the underlying logical or arithmetic constraints-mirroring how humans perform exploratory reasoning. turnbench explicitly addresses key shortcom-ings in existing benchmarks. first, it evaluates multi-turn, multi-step reasoning by requiring llms to adapt dynamically to feedback across multiple rounds and integrate partial clues to formulate and revise hypotheses over time. second, it enables process-level evaluation through a rulebased mechanism that compares models' intermediate inferences-i.e., their identification of active rules in each verifier-against ground truth, allowing structured analysis of reasoning steps beyond final answer correctness. finally, turnbench offers strong contamination resistance due to its dynamic rule configurations: even under fixed game setups, varying rule activations lead to distinct reasoning trajectories, minimizing the risk of data leakage from llm pretraining corpora. our work makes the following key contributions: llms in interactive game environments: recent work has explored the use of llms as agents in interactive games to assess their planning, reasoning, and decision-making capabilities across diverse domains such as board games, card games, and social deduction settings @xcite @xcite @xcite @xcite . these benchmarks typically present the game state in textual or structured formats and prompt llms to make the next move using natural language generation. for instance, pokerbench @xcite adopts classification-based decision scenarios, while @xcite and balrog @xcite evaluate agents through multi-turn, interactive gameplay. more recently, lmact @xcite and @xcite extend this line of work by framing game-based reasoning as multistep, multi-turn interaction tasks. unlike benchmarks focused on win rate or action legality, these settings emphasize the process of rule discovery and iterative hypothesis testing in constrained domains. common evaluation metrics include win rate, legality of actions, strategy optimality, and task completion. ## experiment 4.1 experiment setup to thoroughly test the reasoning abilities of the state-of-the-art models, all "thinking" models had their reasoning effort set to "high." additionally, to compare the reasoning gap between the most advanced llms and humans, we invited five human participants with no prior experience with the game to take part in the experiment. we evaluated two game modes: classic and nightmare. each mode's scenarios were divided equally into three difficulty levels: easy, standard, and hard. for classic mode, we constructed 270 benchmark scenarios (90 per difficulty). for nightmare mode, 45 scenarios were selected (15 per difficulty). human participants played 45 classic mode games (15 per difficulty), with the nightmare mode evaluation set matching the models'. all models and human participants were tested under identical conditions, with the same task prompts and problem setups (appendix f). to ensure parity in information access, we developed a user interface for humans that displayed exactly the same text as the models saw at each step. humans were also asked to record their reasoning and thought processes throughout. accuracy is calculated using scikit-learn @xcite . to specifically analyze the impact of model size on performance, we conducted a targeted follow-up experiment. we sampled 45 game setups from the classic mode (15 from each difficulty level) and evaluated two additional large-scale open-source models: llama-3.3-70b-instruct @xcite and mistral-3.2-small (mistralai, 2025). this allows for a more direct comparison across a range of model parameters. ## conclusion in this paper, our investigation using turnbench has clarified the capabilities and limitations of large language models (llms) in multi-turn, multi-step reasoning. turnbench addresses several key limitations of current benchmarks and offers an effective method for automatically analyzing the reasoning processes of llms. using this framework, we evaluated multiple standard chat models and thinking models, uncovering key findings that highlight the limitations of existing models. in summary, turnbench fills a gap in the evaluation of llms' multi-turn, multi-step reasoning capabilities and provides a novel solution for assessing model reasoning processes. we hope that our work will inspire further research into multi-turn reasoning.