A sandbox and 507-task benchmark that grades tool-using agents on final system state.
Agent benchmarks tend to credit a single successful run and to score what the agent says or which tool it calls. Real work beyond code is stateful: an agent has to gather missing information over several turns, follow domain policies, coordinate dependent tools, and leave the backend in exactly the right state with no collateral effects. During my 2026 applied-scientist internship at Microsoft (Copilot Studio), I contributed to ThinkingBox, an open-source sandbox for tool–agent–user interaction, and co-developed ThinkingBox-Bench, its reliability benchmark (Li* et al., 2026).
ThinkingBox runs each task in an isolated, MCP-compatible tool session with a stateful backend, records the complete execution trace, and grades the outcome with executable checks on the terminal backend state and side effects rather than on the final message. ThinkingBox-Bench contains 507 policy-conditioned workflows across five business domains (retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support); 477 tasks are graded on state alone and 30 add a response rubric. I built the simulated user that plays the customer or employee and the LLM-judge evaluators, and I am a co-first author of the paper. The core framework was initiated and built by colleagues at Microsoft before my internship.
The headline finding is a large gap between discovering a solution and reliably producing one. With 20 independent attempts per task, Claude Opus 5 succeeds on 66.5% of single attempts but on all 20 attempts for only 47.5% of tasks; Kimi-K3, the strongest open-weight model, falls from 57.4% to 17.6% by the paper’s estimate. Many failed trials end cleanly after valid state-changing actions, so response- or tool-call-level signals are poor proxies for end-to-end completion. The benchmark (v1.0, August 2026) is released with MIT-licensed code and CDLA-Permissive-2.0 data, and the sandbox’s executable verdicts also serve as rewards for reinforcement-learning post-training.
Top: the sandbox loop, from simulated user and agent through the isolated tool session to executable judging. Bottom: for every model, the share of tasks solved on every one of 20 attempts (green) is far below first-try success (blue) and success on at least one attempt (orange). Figure from the arXiv paper.
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce ThinkingBox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, ThinkingBox-Bench contains 507 policy-conditioned workflows across business scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Our experiments reveal that even the strongest proprietary and open-weight models show steep reliability drops: Claude Opus 5 falls from 66.50% pass@1 to 47.53% when all 20 attempts must succeed, and Kimi-K3 from 57.37% to 17.60%. Moreover, many failed trials terminate cleanly after valid state-changing actions, so response- or tool-call-level signals poorly proxy end-to-end completion. ThinkingBox-Bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both ThinkingBox and ThinkingBox-Bench.
@article{li2026thinkingbox,title={One Success Isn't Reliability: {ThinkingBox}, a Sandbox and Benchmark for Agents in Stateful Business Workflows},author={Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and Palmaz Lopez Pelaez, Susana and Wang, Yuepeng and Smolyakov, Vadim and Jiang, Xiang and {\'O}lafsson, Kjartan and Guy, Tommy},journal={arXiv preprint arXiv:2608.19741},year={2026},month=aug,archiveprefix={arXiv},}
