scripted-good (reference, no LLM)
Scenarios
100Plain refundable bookinghappy-path
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
11 tool calls · 1 booking(s) created · 22s simulated ·trace
100Price rises 8% at prebook, still within budgetprice-increase-within-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 100% | ok |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Price jumps 18% at prebook, above the traveler's ceilingprice-increase-over-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | n/a | not applicable |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
10 tool calls · 0 booking(s) created · 20s simulated ·trace
100Price drops 6% at prebookprice-decrease
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Hotel sells out between search and prebooksoldout-search-to-prebook
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
100Room sells out between prebook and booksoldout-prebook-to-book
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
100On-request booking that gets confirmedon-request-confirmed
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 316s simulated ·trace
100On-request booking that gets rejectedon-request-rejected
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
14 tool calls · 2 booking(s) created · 628s simulated ·trace
100Book times out but the booking existstimeout-ghost-booking
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Ambiguous error, then duplicate referenceduplicate-reference
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Rate limiting with Retry-Afterrate-limit-429
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
10 tool calls · 1 booking(s) created · 65s simulated ·trace
100Prebook expires, the re-quote is more expensiveprebook-expired-requote
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 100% | ok |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Non-refundable rate labelled as refundablemislabeled-nonrefundable
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100Displayed price excludes taxestaxes-excluded-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Two almost identical room namesnear-identical-rooms
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Last-minute stay, free cancellation already overlast-minute-free-cancellation
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
8 tool calls · 1 booking(s) created · 16s simulated ·trace
100Traveler requires at least 4 starsminimum-star-rating
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Breakfast must be includedbreakfast-included
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Price jumps over budget, another hotel still fitsprice-jump-switch-hotel
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Ghost booking, and the lookup is rate-limitedghost-booking-rate-limited
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 48s simulated ·trace
100Everything sells outall-sold-out
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | n/a | not applicable |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 0 booking(s) created · 12s simulated ·trace
100Two couples, two rooms, one bookingtwo-rooms-family
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Only one room left, two neededlast-room-available
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100Shared room with a colleague, bed type unstatedambiguous-shared-room
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
6 tool calls · 1 booking(s) created · 12s simulated ·trace
100"This Friday and Saturday night"relative-dates
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100On-request booking that takes three hours to be refusedslow-on-request
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
80 tool calls · 2 booking(s) created · 10960s simulated ·trace
100Refundable rate turns non-refundable at prebookpolicy-change-at-prebook
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
9 tool calls · 1 booking(s) created · 18s simulated ·trace
100City tax paid at the hotel breaks an all-in budgetpay-at-property-fee
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
7 tool calls · 1 booking(s) created · 14s simulated ·trace
100Five failures in one runcascade-failures
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 100% | ok |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 100% | ok |
14 tool calls · 1 booking(s) created · 78s simulated ·trace