ThinkingBox

Introduction

ThinkingBox is an agentic harness for defining isolated MCP tool environments, creating stateful scenarios and test cases, running an LLM agent with simulated-user interaction, and evaluating the resulting backend state, side effects, and response requirements. It can support offline evaluation or conversation generation and agentic training workflows.

This OpenEnv adapter exposes the public ThinkingBox-Bench through the standard WebSocket reset/step interface. The adapter is currently designed for evaluation only. Canonical coverage uses the pinned public benchmark release; user-supplied ThinkingBox scenarios can also be evaluated when their executable data and external services are provided, but those runs are not canonical ThinkingBox-Bench results.

Quick Start

Start the externally managed ThinkingBox services described under Deployment Boundary, then run the OpenEnv server:

OPENENV_TB_CONFIG=/path/to/thinkingbox.yaml \
uv run --project envs/thinkingbox_env --frozen server

Connect from a trusted harness:

from thinkingbox_env import ThinkingBoxEnv

async with ThinkingBoxEnv(base_url="http://127.0.0.1:8000") as env:
    result = await env.reset("file.py:test_name")
    tools = await env.list_tools()
    result = await env.call_tool("tool_name", {"argument": "value"})
    result = await env.submit_message("Final response")

Examples

• Public client example: reset a server-visible task and inspect its public task and tool observations without loading benchmark implementation data on the client.
• Evaluation wrapper: invoke the packaged thinkingbox-eval CLI for native model orchestration, repetition IDs, privacy-reviewed canonical JSONL, and strict aggregation.

Run either script from the repository root:

uv run --project envs/thinkingbox_env \
  python examples/thinkingbox/example_usage.py file.py:test_name

uv run --project envs/thinkingbox_env \
  python examples/thinkingbox/eval_testlist.py \
  --config /path/to/thinkingbox.yaml \
  --output results.jsonl

The evaluator’s --config and --dataset paths are local to the trusted evaluator. For a remote OpenEnv server, pass the corresponding server-visible paths with --env-config and --env-dataset. Strict canary and full coverage profiles accept only the pinned public benchmark; custom scenario bundles use ordinary non-profiled evaluation.

Environment Details

The adapter preserves native ThinkingBox hydration, history replay, tool overrides, parallel tool batches, direct responses, simulated users, effects, fixtures, TestContext, TestScript, Judge, error latching, and teardown. Final rewards are binary pass/fail; infrastructure failures are reported separately and never converted into benchmark failures.

The model-facing action surface contains tool discovery, tool calls, and assistant messages. A trusted harness may call finish(), but finish is not advertised in the public action schema. Private user context, initialization state, assertion source, credentials, proxy session identifiers, effects, and grading internals remain inside the environment.

seed is accepted for OpenEnv API compatibility but intentionally unused because tasks are selected deterministically by UID. Concurrent sessions are stacked independent environment instances, one trajectory per instance; they are not multiplexed trajectories.

Executable assets come from the pinned thinkingbox-bench-v1.0 release in microsoft/thinkingbox-data. The Hugging Face microsoft/ThinkingBox-Bench dataset is a viewer-friendly representation, while GitHub remains the executable source.

Configuration

The server accepts an ordinary native ThinkingBox YAML configuration through OPENENV_TB_CONFIG or the reset config= argument. The file supplies mcp_proxy, orchestrator.agent_model, user_model, judge_model, judge_type, and user-completion behavior; see the ThinkingBox LLM configuration guide.

The client message timeout remains longer than server-side model and episode operations so a valid long-running turn is not abandoned prematurely.

Deployment Boundary

The image starts only the OpenEnv API. It currently needs an externally managed ThinkingBox Session Proxy, benchmark MCP servers, Typesense, and agent, user-simulator, and judge model endpoints. GET /health reports process liveness; GET /ready reports the observable external dependencies and marks scenario-specific Typesense readiness as unobservable from this process.

Citation

Please cite the paper if ThinkingBox is useful in your experiments.

• Paper: One Success Isn’t Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
• Framework: microsoft/thinkingbox
• Executable benchmark: microsoft/thinkingbox-data
• Viewer dataset: microsoft/ThinkingBox-Bench

@misc{li2026successisntreliabilitythinkingbox,
  title={One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},
  author={Zhuochun Li and Youngmin Ko and Ali Keramati and Nicola Ferri and Susana Palmaz Lopez Pelaez and Liang-Chun Tsai and Calvin Wang and Mirco Milletari and Tuhin Kundu and Vadim Smolyakov and Kjartan Olafsson and Tommy Guy},
  year={2026},
  eprint={2608.19741},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2608.19741},
}