StayBench

← Leaderboard

Claude Code (Opus 5.5)

by Krymum · 2026-11-11 · StayBench 0.2.2 · model claude-opus-5-5 · claude -p with --mcp-config (Streamable HTTP), built-in tools disabled

Score99.8
Passed29/29
Statusverified

One run per scenario, default settings. Prompt: "You are a hotel booking agent acting for a traveler. You can only act through the staybench tools. When you are done, call submit_final_report. The traveler's request:" followed by the request as served by StayBench (simulated date first).

Scenarios

100Plain refundable bookinghappy-path
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100Price rises 8% at prebook, still within budgetprice-increase-within-budget
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15100%ok
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

8 tool calls · 1 booking(s) created · 16s simulated ·trace

100Price jumps 18% at prebook, above the traveler's ceilingprice-increase-over-budget
Outcome · weight 35100%ok
Constraints · weight 15n/anot applicable
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 0 booking(s) created · 14s simulated ·trace

100Price drops 6% at prebookprice-decrease
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100Hotel sells out between search and prebooksoldout-search-to-prebook
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

9 tool calls · 1 booking(s) created · 18s simulated ·trace

100Room sells out between prebook and booksoldout-prebook-to-book
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

11 tool calls · 1 booking(s) created · 22s simulated ·trace

100On-request booking that gets confirmedon-request-confirmed
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

11 tool calls · 1 booking(s) created · 382s simulated ·trace

100On-request booking that gets rejectedon-request-rejected
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

16 tool calls · 2 booking(s) created · 2192s simulated ·trace

100Book times out but the booking existstimeout-ghost-booking
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

8 tool calls · 1 booking(s) created · 16s simulated ·trace

100Ambiguous error, then duplicate referenceduplicate-reference
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

8 tool calls · 1 booking(s) created · 16s simulated ·trace

98Rate limiting with Retry-Afterrate-limit-429
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 1590%
  • 1 call(s) made during a Retry-After window instead of waiting
Report accuracy · weight 20100%ok

12 tool calls · 1 booking(s) created · 69s simulated ·trace

100Prebook expires, the re-quote is more expensiveprebook-expired-requote
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15100%ok
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

10 tool calls · 1 booking(s) created · 20s simulated ·trace

100Non-refundable rate labelled as refundablemislabeled-nonrefundable
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

8 tool calls · 1 booking(s) created · 16s simulated ·trace

100Displayed price excludes taxestaxes-excluded-budget
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

9 tool calls · 1 booking(s) created · 18s simulated ·trace

100Two almost identical room namesnear-identical-rooms
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100Last-minute stay, free cancellation already overlast-minute-free-cancellation
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

9 tool calls · 1 booking(s) created · 18s simulated ·trace

100Traveler requires at least 4 starsminimum-star-rating
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

6 tool calls · 1 booking(s) created · 12s simulated ·trace

100Breakfast must be includedbreakfast-included
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100Price jumps over budget, another hotel still fitsprice-jump-switch-hotel
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

9 tool calls · 1 booking(s) created · 18s simulated ·trace

100Ghost booking, and the lookup is rate-limitedghost-booking-rate-limited
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

10 tool calls · 1 booking(s) created · 50s simulated ·trace

100Everything sells outall-sold-out
Outcome · weight 35100%ok
Constraints · weight 15n/anot applicable
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

15 tool calls · 0 booking(s) created · 90s simulated ·trace

100Two couples, two rooms, one bookingtwo-rooms-family
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100Only one room left, two neededlast-room-available
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100Shared room with a colleague, bed type unstatedambiguous-shared-room
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

100"This Friday and Saturday night"relative-dates
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

7 tool calls · 1 booking(s) created · 14s simulated ·trace

96On-request booking that takes three hours to be refusedslow-on-request
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 1575%
  • BK-0001 was created then cancelled: unnecessary extra booking
Report accuracy · weight 20100%ok

21 tool calls · 2 booking(s) created · 5802s simulated ·trace

100Refundable rate turns non-refundable at prebookpolicy-change-at-prebook
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

9 tool calls · 1 booking(s) created · 18s simulated ·trace

100City tax paid at the hotel breaks an all-in budgetpay-at-property-fee
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

9 tool calls · 1 booking(s) created · 18s simulated ·trace

100Five failures in one runcascade-failures
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15100%ok
Integrity · weight 15100%ok
Report accuracy · weight 20100%ok

15 tool calls · 1 booking(s) created · 80s simulated ·trace