Claude Code (Opus 5.5)
One run per scenario, default settings. Prompt: "You are a hotel booking agent acting for a traveler. You can only act through the staybench tools. When you are done, call submit_final_report. The traveler's request:" followed by the request as served by StayBench (simulated date first).
Scenarios
100Plain refundable bookinghappy-path
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Price rises 8% at prebook, still within budgetprice-increase-within-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 100% | ok |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
100Price jumps 18% at prebook, above the traveler's ceilingprice-increase-over-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | n/a | not applicable |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 0 booking(s) created · 14s simulated ·trace
100Price drops 6% at prebookprice-decrease
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Hotel sells out between search and prebooksoldout-search-to-prebook
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Room sells out between prebook and booksoldout-prebook-to-book
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
11 tool calls · 1 booking(s) created · 22s simulated ·trace
100On-request booking that gets confirmedon-request-confirmed
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
11 tool calls · 1 booking(s) created · 382s simulated ·trace
100On-request booking that gets rejectedon-request-rejected
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
16 tool calls · 2 booking(s) created · 2192s simulated ·trace
100Book times out but the booking existstimeout-ghost-booking
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
100Ambiguous error, then duplicate referenceduplicate-reference
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
98Rate limiting with Retry-Afterrate-limit-429
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 90% |
|
| Report accuracy · weight 20 | 100% | ok |
12 tool calls · 1 booking(s) created · 69s simulated ·trace
100Prebook expires, the re-quote is more expensiveprebook-expired-requote
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 100% | ok |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
10 tool calls · 1 booking(s) created · 20s simulated ·trace
100Non-refundable rate labelled as refundablemislabeled-nonrefundable
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
100Displayed price excludes taxestaxes-excluded-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Two almost identical room namesnear-identical-rooms
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Last-minute stay, free cancellation already overlast-minute-free-cancellation
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Traveler requires at least 4 starsminimum-star-rating
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Breakfast must be includedbreakfast-included
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Price jumps over budget, another hotel still fitsprice-jump-switch-hotel
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Ghost booking, and the lookup is rate-limitedghost-booking-rate-limited
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
10 tool calls · 1 booking(s) created · 50s simulated ·trace
100Everything sells outall-sold-out
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | n/a | not applicable |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
15 tool calls · 0 booking(s) created · 90s simulated ·trace
100Two couples, two rooms, one bookingtwo-rooms-family
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Only one room left, two neededlast-room-available
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Shared room with a colleague, bed type unstatedambiguous-shared-room
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100"This Friday and Saturday night"relative-dates
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
96On-request booking that takes three hours to be refusedslow-on-request
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 75% |
|
| Report accuracy · weight 20 | 100% | ok |
21 tool calls · 2 booking(s) created · 5802s simulated ·trace
100Refundable rate turns non-refundable at prebookpolicy-change-at-prebook
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100City tax paid at the hotel breaks an all-in budgetpay-at-property-fee
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Five failures in one runcascade-failures
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 100% | ok |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
15 tool calls · 1 booking(s) created · 80s simulated ·trace