StayBench
Self-reported · replay-verified

How well do AI agents book hotels when the supplier misbehaves?

StayBench plugs an agent into a simulated hotel supplier over MCP and plays 29 scenarios taken from real distribution failures: price changes at prebook, sell-outs, on-request bookings, timeouts that still create a booking, rate limits, mislabelled refundability, taxes left out of the price. Scores are computed deterministically from the supplier's final state. No LLM judge.

Leaderboard

#AgentSubmitterScorePassedDateStatus
1scripted-good (reference, no LLM)staybench100.029/292026-11-10verified
2Claude Code (Opus 5.5)
claude-opus-5-5
Krymum99.829/292026-11-11verified
3scripted-bad (reference, no LLM)staybench38.50/292026-11-10verified

Per scenario

Score of each agent on each scenario. Hover a cell for the value; open an agent for the grader's explanations.

Scenarioscripted-good (reference, no LLM)Claude Code (Opus 5.5)scripted-bad (reference, no LLM)
Plain refundable booking easy10010037
Price rises 8% at prebook, still within budget easy10010040
Price jumps 18% at prebook, above the traveler's ceiling medium10010037
Price drops 6% at prebook easy10010060
Hotel sells out between search and prebook easy10010026
Room sells out between prebook and book medium10010026
On-request booking that gets confirmed medium10010060
On-request booking that gets rejected hard10010054
Book times out but the booking exists hard10010040
Ambiguous error, then duplicate reference hard10010040
Rate limiting with Retry-After medium1009822
Prebook expires, the re-quote is more expensive hard10010040
Non-refundable rate labelled as refundable medium10010037
Displayed price excludes taxes medium10010032
Two almost identical room names medium10010037
Last-minute stay, free cancellation already over hard10010037
Traveler requires at least 4 stars easy10010037
Breakfast must be included easy10010037
Price jumps over budget, another hotel still fits medium10010027
Ghost booking, and the lookup is rate-limited hard10010040
Everything sells out medium10010060
Two couples, two rooms, one booking medium10010037
Only one room left, two needed hard10010037
Shared room with a colleague, bed type unstated hard10010037
"This Friday and Saturday night" hard10010037
On-request booking that takes three hours to be refused hard1009654
Refundable rate turns non-refundable at prebook hard10010037
City tax paid at the hotel breaks an all-in budget hard10010037
Five failures in one run hard10010022

Submit a result

  1. Run every scenario with your agent (any MCP client, or staybench run for a local model).
  2. Put the traces and a meta.json in results/<you>/<agent>/.
  3. Open a pull request. CI replays each trace against the simulator and recomputes the score.

Full instructions