MARS -Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation (2025)

Abstract: Large Language Models ( LLMs ), e.g. ChatGPT, have been widely adopted in real-world dialogue applications. However, LLMs’ robustness, especially in handling long complex dialogue sessions, including frequent motivation transfer, sophisticated cross-turn dependency, is criticized all along. Nevertheless, no existing benchmarks can fully reflect these weaknesses. We present MARS-Bench , a M ulti-turn A thletic R eal-world S cenario Dialogue Bench mark, designed to remedy the gap. MARS-Bench is constructed from play-by-play text commentary so to feature realistic dialogues specifically designed to evaluate three critical aspects of multi-turn conversations: ultra multi-turn, interactive multi-turn, and cross-turn tasks. Extensive experiments on MARS-Bench also reveal that closed-source LLMs significantly outperform open-source alternatives, explicit reasoning significantly boosts LLMs’ robustness on handling long complex dialogue sessions, and LLMs indeed face significant challenge when handling motivation transfer and sophisticated cross-turn dependency. Moreover, we provide mechanistic interpretability on how attention sinks due to special tokens lead to LLMs’ performance degradation when handling long complex dialogue sessions based on attention visualization experiment in Qwen2.5-7B-Instruction.

## introduction large language models (llms) have made remarkable advances, enabling fluent interactions with users, even on sessions with more than 30 turns, messy information and unnatural motivation transfer. however, llms' robustness has long been criticized when handling users' shifts between information-seeking questions, reasoning tasks, and creative content generation in same dialogue sessions without clear task boundaries, while no existing evaluations fully reflect the weakness. we point out that llms must (i) retrieve evidence dispersed across distant, sometimes dozens-of-turnold, utterances, and (ii) reason jointly over these fragments while adapting to frequent task switches to handle the aforementioned scenarios. as these conversational scenarios grow increasingly complex, there is a clear need for robust evaluation protocols that can systematically assess llms' ability to understand, reason, and respond coherently across conversational turns. however, many benchmarks focus on short conversations, provide the full dialogue history upfront rather than revealing it turn by turn, and rarely test reasoning over information scattered across distant turns @xcite @xcite @xcite ## mars-bench: design and construction this section is organized as follows: data collection and processing (3.1), task categories (3.2), and benchmark statistics (3.3). ## discussion we organize our discussion around the following research questions: (1) whether increasing the number of dialogue turns harms model performance; ## conclusion we introduce mars-bench, a benchmark constructed from real-world play-by-play sports data to evaluate llms in complex multi-turn dialogue settings. it defines four task types: instruction following, context retrieval, information reasoning, and task switching, enabling systematic and finegrained assessment of long-context dialogue capabilities. experimental results suggest that models employing explicit reasoning strategies tend to perform more consistently, although all models exhibit persistent challenges in instruction alignment and context retention. further analysis indicates that dialogue depth, input structure, and the accumulation of errors across turns can significantly affect model ge2.5p c3.7th db1.5th o3-mini 4o dsv3 0 20 40 60 80 100 accuracy(%) 1 turn (reasoning) 1 turn (non-reasoning) 10 turns (reasoning) 10 turns (non-reasoning) 20 turns (reasoning) 20 turns (non-reasoning) figure 9: ablation on interaction turns: effect on overall accuracy. model performance on score updates under different interaction turns (1, 10, 20). most models exhibit significant improvement with increasing turns.