T ool H aystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactions (2025)

Abstract: Large language models (LLMs) have demonstrated strong capabilities in using external tools to address user inquiries. However, most existing evaluations assume tool use in short contexts, offering limited insight into model behavior during realistic long-term interactions. To fill this gap, we introduce ToolHaystack, a benchmark for testing the tool use capabilities in long-term interactions. Each test instance in ToolHaystack includes multiple tasks execution contexts and realistic noise within a continuous conversation, enabling assessment of how well models maintain context and handle various disruptions. By applying this benchmark to 14 state-of-the-art LLMs, we find that while current models perform well in standard multi-turn settings, they often significantly struggle in ToolHaystack, highlighting critical gaps in their long-term robustness not revealed by previous tool benchmarks.

## introduction recent breakthroughs in large language models (llms) have transformed their role toward llm agents; they can execute real-world tasks such as managing financial transactions and scheduling appointments @xcite . tool-augmented language model (talm) benchmarks @xcite @xcite have played a critical role in evaluating tooluse capabilities based on simplified instructionfollowing setups where they autonomously execute external tools to address user goals. despite these advancements, the scope of most existing work is still limited short interaction, which fail to capture the behavior of llm agents in long-term interactions. the queries of most existing benchmarks are simple (e.g., "book a restaurant named ....") which can be resolved within one long-term interactions short-term interactions oh, and cancel the restaurant ... ... book_flight(...) cancel_restaurant(...) book_restaurant(...) john's birthday is 4 days after mine , any ideas for birthday present? plans changed. we're going to hawaii for my birthday instead. can you book the restaurant abc for my birthday? it's may 15th. semantic noises fragmented context evolving goals feb, 25' mar, 25' apr, 25' may, 25' ( ) ( ) do you think it'll arrive before john's birthday? can you check the reviews and delivery times? + paper 1: ..., paper 2: ..., pa... search_papers(...) can you find me some recent ai papers? search_wikipedia(...) search for quotes existing interactions are simple and resolved within few exchanges whose quotes? figure 1 : toolhaystack addresses long-term interactions that include evolving goals, semantic noise, and fragmented context. or two exchanges. in contrast, many real-world tasks, such as travel planning or project coordination, cannot be resolved in short interactions @xcite or steps @xcite . for example, the user may want to perform tasks based on previous requests without restating all the details (e.g., "cancel the reservation you made for my birthday.") or request to collaborate with the agents across long time frame (e.g., "help me writing a research paper."). these tasks inherently require interactions that unfold and might evolve over time. while recent efforts have extended test scenarios through multi-turn interactions @xcite and api response simulations @xcite , these approaches fall short in capturing the full complexity of long-term interactions. in real-world contexts, continuous exchanges between the user, the agent, and the external environment introduce various contextual noises-such as interleaved task flows @xcite , in this paper, we aim to evaluate talms in realistic long-term interaction scenarios. to this end, we introduce toolhaystack, a realistic and composable test suite for talms. specifically, we define long-term agent context as a highly noisy context where multiple user goals and partial task execution histories are entangled in a single context. test instances are structured to simulate natural task flows by interleaving task-relevant ("needles") and irrelevant contexts ("haystack") 2 , posing unique challenges of identifying key information among distractions accumulated over time. based on the structure, toolhaystack simulates six realistic scenarios derived from three core challenges (context recall, information shift, and missing context), each with two difficulty levels. we systematically evaluate 14 highly capable llms with tool-use capabilities, covering both open-source and proprietary models on tool-haystack. our findings reveal that even highly capable llms with advanced long-context modeling @xcite still struggle to maintain consistency and robustness under long-term scenarios-suggesting that contemporary benchmarks may overestimate their readiness and reliability for real-world deployment. beyond general performance, we also conduct detailed ablation studies to uncover what factors influence 2 we draw the analogy from the classic needle in a haystack problem @xcite to emphasize the noisy nature of long-term interaction. our main contributions are as follows: ## related work realistic evaluation of talms. to better approximate real-world conditions, benchmarks such as tooldial @xcite , hammerbench @xcite , and τ -bench @xcite introduce more naturalistic, multi-turn dialogues. recent efforts like rotbench @xcite and fail-talm @xcite examine robustness under adversarial or erroneous conditions. however, these benchmarks remain largely constrained to single-session or short-horizon tasks, offering limited insight into talm behavior over extended interactions. some recent benchmarks such as multichallenge @xcite have attempted to evaluate language models in multiturn scenarios systematically. compared to mul-tichallenge, toolhaystack differs significantly in problem space, failure types, and evaluation design. multichallenge focuses on instruction-following in relatively short and clean conversations ( 5 turns), with limited extent and control over noise structure. in contrast, toolhaystack targets tool-use scenarios with an average of over 32 turns per instance, explicit argument chaining over long-horizon, and ## rethinking evaluation of talms most llm benchmarks are rapidly saturated by frontier models @xcite . likewise, previous talm evaluation frameworks might have limited lifespans. this underscores a constant need for more realistic, challenging and adaptive evaluation frameworks that can keep pace with model advancements. we highlight that long-term interactions go beyond simple extension of standard multi-turn evaluation. ## toolhaystack: a realistic and composable test suite for long-term tool utilization we introduce toolhaystack, a benchmark to stress-test talm's abilities in long-horizon tasks. ## building toolhaystack to ensure high-quality data while maintaining costefficiency, toolhaystack employs a multi-stage generation pipeline with three main phases: (1) tool set collection; (2) scenario-grounded tool sequence construction (3) dialogue instantiation. detailed process of generation process is presented in appendix c. ## experiments 6.1 experimental setup models. we evaluate a comprehensive set of highly capable talms. closed-source models include gpt family (4o, 4.5-preview, and 4omini), claude family (3.7-sonnet, 3-opus), gemini family (2.5-pro, 2.0-flash-001), amazon-nova-pro, mistral-large and grok-3. open-source models are qwen2.5-72b-instruct, llama-3.3-70b-instruct, deepseek-v3, toolace-2-llama-3.1-8b, watt-tool-8b and xlam-2-32b-fc-r. note that toolace-2-llama-3.1-8b, watt-tool-8b and xlam-2-32b-fc-r are finetuned for function calling tasks. models are selected to cover a diverse and competitive set of top-performing talms, particularly those demonstrating strong capabilities in recent multi-turn tool benchmarks (e.g., bfcl). retrieval as providing all the api documents is infeasible, a subset of feasible apis is typically given to talm through semantic matching. in toolhaystack, the model is tested in oracle setup with the 5 api documents, where a groundtruth api document and 4 apis are given to the model based on their semantic relevance to the target session. evaluation metric. we use call accuracy which calculates the proportion of correct function calls: ## conclusion in this work, we introduced toolhaystack, a novel benchmark designed to rigorously assess the long-term robustness of tool-augmented language models (talms) in complex, noisy, and realistic interaction scenarios. our extensive evaluations across 10 state-of-the-art models demonstrate that while existing talms excel in short, clean multiturn dialogues, their performance significantly degrades in long-term, distraction-rich contexts. this performance gap highlights the critical need for dedicated long-term evaluation frameworks and reveals current limitations in models' ability to retain context, handle goal shifts, and avoid hallucination over time. toolhaystack offers a composable, scalable, and diagnostically rich test suite to drive progress toward more reliable, agentic llm systems. we hope our benchmark facilitates future research aimed at closing the gap between controlled evaluations and the complex realities of real-world tool use.