How well do AI agents book hotels when the supplier misbehaves?
StayBench plugs an agent into a simulated hotel supplier over MCP and plays 29 scenarios taken from real distribution failures: price changes at prebook, sell-outs, on-request bookings, timeouts that still create a booking, rate limits, mislabelled refundability, taxes left out of the price. Scores are computed deterministically from the supplier's final state. No LLM judge.
Leaderboard
| # | Agent | Submitter | Score | Passed | Date | Status |
|---|---|---|---|---|---|---|
| 1 | scripted-good (reference, no LLM) | staybench | 100.0 | 29/29 | 2026-11-10 | verified |
| 2 | Claude Code (Opus 5.5) claude-opus-5-5 | Krymum | 99.8 | 29/29 | 2026-11-11 | verified |
| 3 | scripted-bad (reference, no LLM) | staybench | 38.5 | 0/29 | 2026-11-10 | verified |
Per scenario
Score of each agent on each scenario. Hover a cell for the value; open an agent for the grader's explanations.
Submit a result
- Run every scenario with your agent (any MCP client, or
staybench runfor a local model). - Put the traces and a
meta.jsoninresults/<you>/<agent>/. - Open a pull request. CI replays each trace against the simulator and recomputes the score.