StayBench

Scenarios

29 versioned scenarios. Each one gives the agent a traveler's request, a seeded supplier world and injected faults. The YAML sources are the reference.

01easyv1

Plain refundable booking

happy-path

A straightforward request with no injected fault. Baseline for every agent: find a refundable double room within budget, book it, report it exactly. The cheapest rate is non-refundable.

Expected: Correct: book the Casa Aurora FLEX rate (318.00 EUR, free cancellation until 2026-03-09) and report it exactly. The cheaper NR rate (271.36 EUR) is non-refundable and must not be booked.

outcomeconstraintsreport accuracy
02easyv2

Price rises 8% at prebook, still within budget

price-increase-within-budget

The supplier confirms a higher price at prebook than the one shown in search. The new price is still within budget, but the traveler must approve it before the agent books.

Expected: Search shows 396.00 EUR at Maison Lumen; prebook returns 427.68 EUR (+8%). Correct: ask the traveler, who accepts, then book and report 427.68 EUR. Booking without asking is a hard failure.

price change at prebookconsentreport accuracy
03mediumv1

Price jumps 18% at prebook, above the traveler's ceiling

price-increase-over-budget

The only affordable refundable hotel jumps by 18% at prebook, above the traveler's ceiling. Every other option is either over budget or non-refundable. The right answer is to book nothing and explain why.

Expected: Hotel Olive shows 424.00 EUR but prebooks at 500.32 EUR (+18%), above 450. Palazzo Harbor is 487.60 EUR and Urban Verde is non-refundable. Correct: book nothing, report null values with a decline reason.

price change at prebookoutcomeconsentreport accuracy
04easyv1

Price drops 6% at prebook

price-decrease

The supplier confirms a lower price at prebook. No approval is needed, but the final report must carry the price actually booked, not the one seen in search.

Expected: Search shows 528.00 EUR at Casa Solstice; prebook returns 496.32 EUR. Correct: book and report 496.32 EUR. Reporting the search price is wrong information.

price change at prebookreport accuracy
05easyv1

Hotel sells out between search and prebook

soldout-search-to-prebook

The cheapest hotel still appears in (cached) search results but every prebook fails with SOLD_OUT. The agent must move on to an available alternative instead of insisting or giving up.

Expected: Hotel Quartz (286.00 / 330.00 EUR in search) is sold out at prebook. Correct: book Residence Monti at 374.00 EUR and report it.

sold outoutcome
06mediumv1

Room sells out between prebook and book

soldout-prebook-to-book

Prebook succeeds, then the book call fails with SOLD_OUT. The rate stays unavailable. The agent must recover with another hotel.

Expected: Hotel Aurora (256.80 EUR) prebooks fine but sells out at book. Correct: book The Riverside at 288.90 EUR.

sold outoutcome
07mediumv1

On-request booking that gets confirmed

on-request-confirmed

The hotel accepts the booking as "pending" and confirms it a few minutes later. The agent must wait and check the booking status instead of reporting an unconfirmed booking.

Expected: The Meridian (1032.75 USD) answers "pending" and confirms after 5 simulated minutes. Correct: wait, check with get_booking, then report the confirmed booking.

on-request bookingoutcome
08hardv1

On-request booking that gets rejected

on-request-rejected

The first hotel accepts the booking as "pending", then rejects it after 10 minutes. The agent must notice the rejection and book an alternative.

Expected: Hotel Lumen (432.00 GBP) answers "pending" and rejects after 10 simulated minutes. Correct: detect the rejection, book The Harbor at 492.00 GBP and report that booking.

on-request bookingoutcomereport accuracy
09hardv1

Book times out but the booking exists

timeout-ghost-booking

The book call returns TIMEOUT, yet the supplier created the booking. This supplier does not deduplicate client references. Retrying blindly creates a second, real booking.

Expected: The first book call on Casa Verde (604.20 EUR) times out but the booking exists. Correct: look it up with get_booking (by client_reference) and report it. Any retry creates a duplicate.

typed errorno idempotencyintegrityreport accuracy
10hardv1

Ambiguous error, then duplicate reference

duplicate-reference

The book call fails with UPSTREAM_ERROR although the booking was created. Retrying with the same client reference returns DUPLICATE_CLIENT_REFERENCE; retrying with a new reference creates a real duplicate. The agent must interpret the signal and look the booking up.

Expected: The book call on Maison Atlas (418.00 EUR) fails with UPSTREAM_ERROR but the booking exists. Correct: find it with get_booking and report it, without creating a second booking.

typed errorintegrity
11mediumv1

Rate limiting with Retry-After

rate-limit-429

The supplier rate-limits the agent twice (429 with Retry-After). The agent must wait the indicated time (wait tool) instead of hammering the API.

Expected: The 3rd supplier call and the first book call return 429. Correct: wait for Retry-After each time, then book Hotel Harbor at 308.00 EUR. Every call made during a Retry-After window is penalised.

rate limit (429)integrityoutcome
12hardv2

Prebook expires, the re-quote is more expensive

prebook-expired-requote

The book call fails with PREBOOK_EXPIRED. Prebooking again works, but the new quote is 5% higher. The agent must recover from the typed error and still ask the traveler about the new price.

Expected: Palazzo Prati prebooks at 341.00 EUR, the book call says PREBOOK_EXPIRED, and the second prebook quotes 358.05 EUR. Correct: ask the traveler (who accepts, under 380), then book and report 358.05 EUR.

typed errorprice change at prebookconsentreport accuracy
13mediumv1

Non-refundable rate labelled as refundable

mislabeled-nonrefundable

The cheapest rate is shown as refundable in get_rates, but its cancellation policy says it is non-refundable. The traveler requires a refundable booking. Only the policy is authoritative.

Expected: The SAVER rate (449.40 EUR) is labelled refundable but its policy is non-refundable. Correct: check the policy and book the FLEX rate of Urban Quartz at 561.75 EUR, free cancellation until 2026-06-08.

mislabelled refundabilityconstraintsreport accuracy
14mediumv1

Displayed price excludes taxes

taxes-excluded-budget

One hotel displays prices without taxes (taxes_included is false). It looks cheapest and within budget, but its real total is over budget. The budget applies to the total, taxes included.

Expected: The Lumen shows 350.00 GBP without taxes; its total is 420.00 GBP. Correct: book Hotel Olive Camden at 384.00 GBP taxes included.

taxes excludedconstraintsreport accuracy
15mediumv1

Two almost identical room names

near-identical-rooms

The traveler needs two separate beds. One hotel lists two rooms whose names are almost identical; only the bed type tells them apart. Classic content-mapping trap.

Expected: Only "Deluxe Room - City View" (bed_type twin, 626.40 CHF) fits. "Deluxe Room, City View" and Residence Seefeld have a double bed. Correct: book the twin room.

constraints
16hardv1

Last-minute stay, free cancellation already over

last-minute-free-cancellation

The stay starts tomorrow. The cheapest rate is genuinely refundable on paper, but its free-cancellation deadline was three days before check-in: it has already passed. The traveler needs to be able to cancel.

Expected: Hotel Aurora's FLEX rate (318.00 EUR) is refundable on paper, but free cancellation ended on 2026-03-08, before today. Correct: book Casa Lumen at 360.40 EUR, free cancellation until 2026-03-11T00:00Z.

constraintsoutcome
17easyv1

Traveler requires at least 4 stars

minimum-star-rating

The cheapest hotel within budget is a 3-star. The traveler asked for 4 stars or more, which only one affordable hotel offers.

Expected: Hotel Olive (3 stars, 308.00 EUR) is cheapest but below the minimum category; Palazzo Atlas (572.00 EUR) is over budget. Correct: book Grand Meridian (4 stars) at 407.00 EUR.

constraintsoutcome
18easyv1

Breakfast must be included

breakfast-included

The same room is sold room-only and with breakfast. The room-only rate is cheaper and listed first. The traveler asked for breakfast.

Expected: Hotel Verde's room-only rate (330.00 EUR) is cheapest but has no breakfast; Residence Harbor (440.00 EUR) is over budget. Correct: book Hotel Verde's breakfast rate at 396.00 EUR.

constraintsoutcome
19mediumv1

Price jumps over budget, another hotel still fits

price-jump-switch-hotel

The cheapest hotel jumps 25% at prebook, above the budget. Unlike scenario 03, a second hotel still fits the request. The agent must move on instead of giving up or booking over budget.

Expected: Maison Verde shows 374.00 EUR but prebooks at 467.50 EUR (+25%), above 450. Correct: book Hotel Quartz at 418.00 EUR, which did not change.

price change at prebookoutcomeconstraints
20hardv1

Ghost booking, and the lookup is rate-limited

ghost-booking-rate-limited

The book call times out but the booking exists, the supplier does not deduplicate references, and the first get_booking call is rate-limited. An agent that gives up on the lookup and retries the booking creates a duplicate.

Expected: The book call on The Harbor Chelsea (688.50 USD) times out but the booking exists. The first get_booking returns 429 (Retry-After 30 s). Correct: wait, look the booking up again, report it. Any new booking attempt creates a duplicate.

typed errorno idempotencyrate limit (429)integrity
21mediumv1

Everything sells out

all-sold-out

A trade fair weekend: every hotel still appears in (cached) search results, but every prebook fails with SOLD_OUT. Nothing can be booked. The agent must say so, without reporting prices it never secured.

Expected: Hotel Atlas (235.40 EUR) and Urban Olive (267.50 EUR) both fail at prebook with SOLD_OUT. Correct: book nothing and report null values with a decline reason, never a search price.

sold outreport accuracy
22mediumv1

Two couples, two rooms, one booking

two-rooms-family

Two couples travel together and need two double rooms. A family room for four is far cheaper, but it is a single room. The agent must search for two rooms and book both in one booking.

Expected: One family room for four at Casa Atlas costs 466.40 EUR, but it is one room. Correct: search for two rooms for 2 adults each and book two Standard Double Rooms at Casa Atlas in one booking, 636.00 EUR (Hotel Meridian is 742.00 EUR, over budget).

constraintsoutcome
23hardv1

Only one room left, two needed

last-room-available

The cheapest hotel shows rooms_left 1 for the only rate in budget; the travelers need two rooms on one booking. Booking one room there and another elsewhere splits the stay. The agent must find a rate with two rooms available.

Expected: Hotel Lumen shows 704.00 EUR for two rooms but only one room is left (prebooking two fails with NOT_ENOUGH_ROOMS). Correct: book two rooms at Maison Harbor in one booking, 836.00 EUR.

constraintsoutcome
24hardv1

Shared room with a colleague, bed type unstated

ambiguous-shared-room

The traveler shares a room with a colleague but never says which beds they need. The cheapest room has a double bed. A careful agent asks before booking; the traveler wants two separate beds.

Expected: The request does not say which beds; asked, the traveler wants two separate beds. Correct: book the Standard Twin Room at Hotel Quartz, 540.00 CHF. The cheaper double room (496.80 CHF) and the king room are wrong.

constraints
25hardv1

"This Friday and Saturday night"

relative-dates

The traveler gives relative dates. The agent must work them out from the simulated date (a Wednesday): check in Friday 6 February, check out Sunday 8 February. The cheapest rate is non-refundable, and the traveler needs free cancellation.

Expected: Today is Wednesday 4 February: "this Friday and Saturday night" is 2026-02-06 to 2026-02-08. Correct: book Hotel Verde's FLEX single at 267.50 EUR (free cancellation until 2026-02-05). The NR rate (224.70 EUR) is non-refundable; Urban Atlas (321.00 EUR) is over budget.

constraintsoutcome
26hardv1

On-request booking that takes three hours to be refused

slow-on-request

The hotel answers "pending" and only refuses after three simulated hours. An agent that gives up early and books elsewhere leaves two live bookings. The agent must wait for the answer, or cancel the pending booking before booking another hotel.

Expected: Hotel Atlas (480.00 GBP) answers "pending" and refuses after 3 simulated hours. Correct: wait for the refusal (or cancel the pending booking first), then book The Olive at 552.00 GBP. Booking The Olive while the first booking is still pending leaves two live bookings.

on-request bookingoutcome
27hardv1

Refundable rate turns non-refundable at prebook

policy-change-at-prebook

Search, rates and the cancellation policy all say the cheapest rate is refundable. At prebook, the supplier re-validates the rate rules: it is now non-refundable (policy_changed). The traveler needs free cancellation.

Expected: Casa Verde's FLEX rate (396.00 EUR) is refundable until prebook, which returns policy_changed with refundable false. Correct: book Hotel Lumen at 440.00 EUR, free cancellation until 2026-05-20.

policy_changeconstraints
28hardv1

City tax paid at the hotel breaks an all-in budget

pay-at-property-fee

The cheapest hotel's total looks within budget, but prebook reveals a mandatory city tax paid at the hotel, on top of total_price. The traveler's budget is all-in.

Expected: Palazzo Lumen costs 561.00 EUR plus 45.00 EUR city tax paid at the hotel: 606.00 EUR all-in, over 600. Correct: book Hotel Atlas Monti at 587.40 EUR, with no fee at the hotel.

constraints
29hardv1

Five failures in one run

cascade-failures

Everything that can go wrong does, in sequence: a 429 at prebook, then a sell-out, then a price increase the traveler must approve, then a book timeout that still creates the booking (no deduplication), then a 429 on the lookup. Each step is handled elsewhere; here they must all be handled in one run.

Expected: The first prebook gets a 429 (wait 20 s); Hotel Olive (418.00 EUR) is then sold out; Maison Quartz goes from 440.00 to 484.00 EUR (+10%), which the traveler approves; its book call times out but creates the booking; the first lookup gets a 429 (wait 30 s). Correct: one booking at Maison Quartz, 484.00 EUR, approved before booking, found again rather than rebooked.