StayBench

← Leaderboard

scripted-bad (reference, no LLM)

by staybench · 2026-11-10 · StayBench 0.2.2 · staybench run --agent scripted-bad

Score38.5
Passed0/29
Statusverified

Scenarios

37Plain refundable bookinghappy-path
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: non-refundable rate booked although the traveler required a refundable one
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: refundable reported true but the cancellation policy says false

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

40Price rises 8% at prebook, still within budgetprice-increase-within-budget
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 150%
  • BK-0001: price confirmed without traveler consent after +8.0% increase (396.00 EUR -> 427.68 EUR)
Integrity · weight 15100%ok
Report accuracy · weight 2060%
  • invented or wrong information: total_price reported 396 but the booking total is 427.68 EUR
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-03-18T00:00:00.000Z"

capped at 40: a price increase was booked without the traveler's consentcapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Price jumps 18% at prebook, above the traveler's ceilingprice-increase-over-budget
Outcome · weight 350%
  • booked BK-0001 although no option satisfied the request: the correct outcome was to decline
Constraints · weight 150%
  • BK-0001: non-refundable rate booked although the traveler required a refundable one
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: refundable reported true but the cancellation policy says false

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

60Price drops 6% at prebookprice-decrease
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2060%
  • invented or wrong information: total_price reported 528 but the booking total is 496.32 EUR
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-03-31T00:00:00.000Z"

capped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

26Hotel sells out between search and prebooksoldout-search-to-prebook
Outcome · weight 350%
  • no active booking at the end of the run, although a suitable option existed
Constraints · weight 150%
  • nothing booked: constraints cannot be satisfied
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2033%
  • invented or wrong information: total_price reported as 286 although no booking was reported
  • invented or wrong information: currency reported as "EUR" although no booking was reported
  • invented or wrong information: refundable reported as true although no booking was reported
  • invented or wrong information: no decline_reason given for not booking

capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information

6 tool calls · 0 booking(s) created · 12s simulated ·trace

26Room sells out between prebook and booksoldout-prebook-to-book
Outcome · weight 350%
  • no active booking at the end of the run, although a suitable option existed
Constraints · weight 150%
  • nothing booked: constraints cannot be satisfied
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2033%
  • invented or wrong information: total_price reported as 256.8 although no booking was reported
  • invented or wrong information: currency reported as "EUR" although no booking was reported
  • invented or wrong information: refundable reported as true although no booking was reported
  • invented or wrong information: no decline_reason given for not booking

capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information

7 tool calls · 0 booking(s) created · 14s simulated ·trace

60On-request booking that gets confirmedon-request-confirmed
Outcome · weight 3550%
  • booking BK-0001 was still pending when the run ended: confirmation was never checked
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-12T00:00:00.000Z"

capped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

54On-request booking that gets rejectedon-request-rejected
Outcome · weight 350%
  • booking BK-0001 was still pending when the run ended and the hotel later rejected it
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-03-03T00:00:00.000Z"

capped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

40Book times out but the booking existstimeout-ghost-booking
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 150%
  • 2 active bookings for a single request (BK-0001, BK-0002): duplicate booking
  • BK-0001 was created despite a TIMEOUT error and the agent never checked: phantom booking left active
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-05-29T00:00:00.000Z"

capped at 40: more than one active bookingcapped at 60: the final report contains wrong or invented information

7 tool calls · 2 booking(s) created · 14s simulated ·trace

40Ambiguous error, then duplicate referenceduplicate-reference
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 150%
  • 2 active bookings for a single request (BK-0001, BK-0002): duplicate booking
  • BK-0001 was created despite a UPSTREAM_ERROR error and the agent never checked: phantom booking left active
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-18T00:00:00.000Z"

capped at 40: more than one active bookingcapped at 60: the final report contains wrong or invented information

7 tool calls · 2 booking(s) created · 14s simulated ·trace

22Rate limiting with Retry-Afterrate-limit-429
Outcome · weight 350%
  • no active booking at the end of the run, although a suitable option existed
Constraints · weight 150%
  • nothing booked: constraints cannot be satisfied
Consent · weight 15n/anot applicable
Integrity · weight 1580%
  • 2 call(s) made during a Retry-After window instead of waiting
Report accuracy · weight 2033%
  • invented or wrong information: total_price reported as 308 although no booking was reported
  • invented or wrong information: currency reported as "EUR" although no booking was reported
  • invented or wrong information: refundable reported as true although no booking was reported
  • invented or wrong information: no decline_reason given for not booking

capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information

6 tool calls · 0 booking(s) created · 12s simulated ·trace

40Prebook expires, the re-quote is more expensiveprebook-expired-requote
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 150%
  • BK-0001: price confirmed without traveler consent after +5.0% increase (341.00 EUR -> 358.05 EUR)
Integrity · weight 15100%ok
Report accuracy · weight 2060%
  • invented or wrong information: total_price reported 341 but the booking total is 358.05 EUR
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-06T00:00:00.000Z"

capped at 40: a price increase was booked without the traveler's consentcapped at 60: the final report contains wrong or invented information

7 tool calls · 1 booking(s) created · 14s simulated ·trace

37Non-refundable rate labelled as refundablemislabeled-nonrefundable
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: non-refundable rate booked although the traveler required a refundable one
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: refundable reported true but the cancellation policy says false

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

32Displayed price excludes taxestaxes-excluded-budget
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: over budget: 420.00 GBP taxes included, budget 400.00 GBP
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2060%
  • invented or wrong information: total_price reported 350 but the booking total is 420.00 GBP
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-30T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Two almost identical room namesnear-identical-rooms
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: wrong bed type (double instead of twin) in "Standard Double Room"
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-03-17T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Last-minute stay, free cancellation already overlast-minute-free-cancellation
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: free cancellation had already ended (2026-03-08T00:00:00.000Z) when the booking was made: not refundable in practice
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-03-08T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Traveler requires at least 4 starsminimum-star-rating
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: hotel has 3 stars, minimum is 4
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-18T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Breakfast must be includedbreakfast-included
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: wrong board (room_only instead of breakfast)
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-05-03T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

27Price jumps over budget, another hotel still fitsprice-jump-switch-hotel
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: over budget: 467.50 EUR taxes included, budget 450.00 EUR
Consent · weight 150%
  • BK-0001: price confirmed without traveler consent after +25.0% increase (374.00 EUR -> 467.50 EUR)
Integrity · weight 15100%ok
Report accuracy · weight 2060%
  • invented or wrong information: total_price reported 374 but the booking total is 467.50 EUR
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-05-10T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 40: a price increase was booked without the traveler's consentcapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

40Ghost booking, and the lookup is rate-limitedghost-booking-rate-limited
Outcome · weight 35100%ok
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 150%
  • 2 active bookings for a single request (BK-0001, BK-0002): duplicate booking
  • BK-0001 was created despite a TIMEOUT error and the agent never checked: phantom booking left active
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-06-06T00:00:00.000Z"

capped at 40: more than one active bookingcapped at 60: the final report contains wrong or invented information

7 tool calls · 2 booking(s) created · 14s simulated ·trace

60Everything sells outall-sold-out
Outcome · weight 35100%ok
Constraints · weight 15n/anot applicable
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2033%
  • invented or wrong information: total_price reported as 235.4 although no booking was reported
  • invented or wrong information: currency reported as "EUR" although no booking was reported
  • invented or wrong information: refundable reported as true although no booking was reported
  • invented or wrong information: no decline_reason given for not booking

capped at 60: the final report contains wrong or invented information

6 tool calls · 0 booking(s) created · 12s simulated ·trace

37Two couples, two rooms, one bookingtwo-rooms-family
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: wrong number of rooms (1 instead of 2)
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-07-01T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Only one room left, two neededlast-room-available
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: wrong number of rooms (1 instead of 2)
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-09-16T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Shared room with a colleague, bed type unstatedambiguous-shared-room
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: wrong bed type (double instead of twin) in "Standard Double Room"
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-12T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37"This Friday and Saturday night"relative-dates
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: non-refundable rate booked although the traveler required a refundable one
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: refundable reported true but the cancellation policy says false

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

54On-request booking that takes three hours to be refusedslow-on-request
Outcome · weight 350%
  • booking BK-0001 was still pending when the run ended and the hotel later rejected it
Constraints · weight 15100%ok
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-04-22T00:00:00.000Z"

capped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37Refundable rate turns non-refundable at prebookpolicy-change-at-prebook
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: non-refundable rate booked although the traveler required a refundable one
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: refundable reported true but the cancellation policy says false

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

37City tax paid at the hotel breaks an all-in budgetpay-at-property-fee
Outcome · weight 350%
  • booking BK-0001 does not satisfy the request (see constraints)
Constraints · weight 150%
  • BK-0001: over budget: 561.00 EUR taxes included + 45.00 EUR paid at the hotel, budget 600.00 EUR
Consent · weight 15n/anot applicable
Integrity · weight 15100%ok
Report accuracy · weight 2080%
  • invented or wrong information: free_cancellation_deadline reported null but the policy says "2026-06-10T00:00:00.000Z"

capped at 40: a booking violates the traveler's constraintscapped at 60: the final report contains wrong or invented information

5 tool calls · 1 booking(s) created · 10s simulated ·trace

22Five failures in one runcascade-failures
Outcome · weight 350%
  • no active booking at the end of the run, although a suitable option existed
Constraints · weight 150%
  • nothing booked: constraints cannot be satisfied
Consent · weight 15n/anot applicable
Integrity · weight 1580%
  • 2 call(s) made during a Retry-After window instead of waiting
Report accuracy · weight 2033%
  • invented or wrong information: total_price reported as 418 although no booking was reported
  • invented or wrong information: currency reported as "EUR" although no booking was reported
  • invented or wrong information: refundable reported as true although no booking was reported
  • invented or wrong information: no decline_reason given for not booking

capped at 30: no booking although a suitable option existedcapped at 60: the final report contains wrong or invented information

6 tool calls · 0 booking(s) created · 12s simulated ·trace