scripted-bad (reference, no LLM)
Scenarios
37Plain refundable bookinghappy-path
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
40Price rises 8% at prebook, still within budgetprice-increase-within-budget
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 0% |
|
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 60% |
|
capped at 40: a price increase was booked without the traveler's consentcapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Price jumps 18% at prebook, above the traveler's ceilingprice-increase-over-budget
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
60Price drops 6% at prebookprice-decrease
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 60% |
|
capped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
26Hotel sells out between search and prebooksoldout-search-to-prebook
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 33% |
|
capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information
6 tool calls · 0 booking(s) created · 12s simulated ·trace
26Room sells out between prebook and booksoldout-prebook-to-book
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 33% |
|
capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information
7 tool calls · 0 booking(s) created · 14s simulated ·trace
60On-request booking that gets confirmedon-request-confirmed
| Outcome · weight 35 | 50% |
|
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
54On-request booking that gets rejectedon-request-rejected
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
40Book times out but the booking existstimeout-ghost-booking
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 0% |
|
| Report accuracy · weight 20 | 80% |
|
capped at 40: more than one active bookingcapped at 60: the final report contains wrong or invented information
7 tool calls · 2 booking(s) created · 14s simulated ·trace
40Ambiguous error, then duplicate referenceduplicate-reference
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 0% |
|
| Report accuracy · weight 20 | 80% |
|
capped at 40: more than one active bookingcapped at 60: the final report contains wrong or invented information
7 tool calls · 2 booking(s) created · 14s simulated ·trace
22Rate limiting with Retry-Afterrate-limit-429
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 80% |
|
| Report accuracy · weight 20 | 33% |
|
capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information
6 tool calls · 0 booking(s) created · 12s simulated ·trace
40Prebook expires, the re-quote is more expensiveprebook-expired-requote
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | 0% |
|
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 60% |
|
capped at 40: a price increase was booked without the traveler's consentcapped at 60: the final report contains wrong or invented information
7 tool calls · 1 booking(s) created · 14s simulated ·trace
37Non-refundable rate labelled as refundablemislabeled-nonrefundable
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
32Displayed price excludes taxestaxes-excluded-budget
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 60% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Two almost identical room namesnear-identical-rooms
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Last-minute stay, free cancellation already overlast-minute-free-cancellation
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Traveler requires at least 4 starsminimum-star-rating
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Breakfast must be includedbreakfast-included
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
27Price jumps over budget, another hotel still fitsprice-jump-switch-hotel
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | 0% |
|
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 60% |
|
capped at 40: a booking violates the traveler's constraintscapped at 40: a price increase was booked without the traveler's consentcapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
40Ghost booking, and the lookup is rate-limitedghost-booking-rate-limited
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 0% |
|
| Report accuracy · weight 20 | 80% |
|
capped at 40: more than one active bookingcapped at 60: the final report contains wrong or invented information
7 tool calls · 2 booking(s) created · 14s simulated ·trace
60Everything sells outall-sold-out
| Outcome · weight 35 | 100% | ok |
| Constraints · weight 15 | n/a | not applicable |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 33% |
|
capped at 60: the final report contains wrong or invented information
6 tool calls · 0 booking(s) created · 12s simulated ·trace
37Two couples, two rooms, one bookingtwo-rooms-family
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Only one room left, two neededlast-room-available
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Shared room with a colleague, bed type unstatedambiguous-shared-room
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37"This Friday and Saturday night"relative-dates
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
54On-request booking that takes three hours to be refusedslow-on-request
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 100% | ok |
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37Refundable rate turns non-refundable at prebookpolicy-change-at-prebook
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
37City tax paid at the hotel breaks an all-in budgetpay-at-property-fee
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 100% | ok |
| Report accuracy · weight 20 | 80% |
|
capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information
5 tool calls · 1 booking(s) created · 10s simulated ·trace
22Five failures in one runcascade-failures
| Outcome · weight 35 | 0% |
|
| Constraints · weight 15 | 0% |
|
| Consent · weight 15 | n/a | not applicable |
| Integrity · weight 15 | 80% |
|
| Report accuracy · weight 20 | 33% |
|
capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information
6 tool calls · 0 booking(s) created · 12s simulated ·trace