Benchmark · simulated run · seed 20261029 · 48 businesses · 600 tasks
Agent commerce readiness benchmark
What happens when AI agents book, ask for quotes, request consultations and order from small businesses: with snuepi, against a model of how agents work today (they phone the business, read its website or guess). A simulated world, run end to end and published with its assumptions.
Read this first
- Simulated, not customer data. The businesses, customers and agents are synthetic. snuepi is a free pilot without customers yet. With snuepi, every task ran against the real product on one machine; nothing reached a real business.
- Today is a model, not a measurement. How agents work today (phone answer rates, hang-ups, websites without prices or rules) is modelled from published figures and stated assumptions. Each assumption has a range, and the comparison is run at both ends of every range.
- Times are not comparable one to one. snuepi’s times are measured on localhost, without network latency or a person in the loop; today’s are modelled minutes.
- What snuepi does not win yet is listed below, with the mistakes this run found.
Headline
- Allowed tasks completed in the session: 99% with snuepi, 63% today (81% to 23% across the assumption ranges).
- Outcomes that matched the business’s terms: 100% with snuepi, 54% today (75% to 19%).
- Wrong yes and wrong no among 549 honest tasks: 0 and 3 with snuepi; 33 and 17 in the model of today.
- Phone interruptions at the business per 100 tasks: 0.0 with snuepi, 77.4 today (2.5 staff hours per 100 tasks).
- Requests with proof that an AI agent made them, and how verified it was: 100% with snuepi, 0% today. With a signed record anyone can verify: 98% and 0%.
- Bookings with a deposit due that carried it: 100% with snuepi (66% held until paid, where the business takes payment on its own Stripe), 10% today.
- Attacks that worked: 0% of 51 with snuepi, 43% today.
- snuepi stays ahead on completion and on correct outcomes at both ends of every assumption range.
The comparison
| Measure | With snuepi | Today (central) | Today (best case) | Today (worst case) |
|---|---|---|---|---|
| Allowed tasks completed | 99% | 63% | 81% | 23% |
| Outcomes correct | 100% | 54% | 75% | 19% |
| Definitive answer in the session | 100% | 50% | 70% | 22% |
| Median time to the answer | 11 ms | 4 min | 2 min | 60.0 s |
| Phone interruptions per 100 tasks | 0.0 | 77.4 | 71.2 | 62.7 |
| Wrong yes (honest tasks) | 0 | 33 | 36 | 13 |
| Terms respected on bookings made | 100% | 8% | ||
| Disclosure proven | 100% | 0% | ||
| Verifiable record | 98% | 0% | ||
| Deposit carried or secured | 100% | 10% | ||
| Expected no-show rate on bookings made | 1.96% | 2.35% |
Best case: every assumption at the end of its range that favours today’s way of working. The expected no-show rate applies resOS’s recorded rates (0.80% with a deposit or card hold, 2.47% without; restaurants, descriptive) to the bookings made: a booking counts as protected only when its deposit is held until paid.
Allowed tasks completed, by task
By door
| Door | Tasks | Completed | Correct | Wrong yes | Wrong no | Median time |
|---|---|---|---|---|---|---|
| A2A | 34 | 100% | 100% | 0 | 0 | 7 ms |
| ACP checkout | 5 | – | 100% | 0 | 0 | 2 ms |
| MCP | 181 | 99% | 99% | 0 | 1 | 12 ms |
| Phone (forwarded number) | 24 | 100% | 100% | 0 | 0 | 25 ms |
| REST | 291 | 99% | 99% | 0 | 2 | 8 ms |
| UCP checkout | 1 | 100% | 100% | 0 | 0 | 6 ms |
| Booking page | 13 | 100% | 100% | 0 | 0 | 10 ms |
By sector
| Sector | Tasks | Completed with snuepi | Completed today | Correct with snuepi | Correct today |
|---|---|---|---|---|---|
| Accountants | 12 | 100% | 90% | 100% | 83% |
| Beauty | 47 | 100% | 57% | 100% | 47% |
| Cleaning | 46 | 100% | 71% | 100% | 70% |
| Clinics | 31 | 100% | 44% | 100% | 26% |
| Dentists | 30 | 96% | 62% | 97% | 53% |
| Financial advisers | 22 | 100% | 86% | 100% | 73% |
| Restaurants | 28 | 96% | 50% | 96% | 50% |
| Hotels | 49 | 98% | 62% | 98% | 41% |
| Insurance brokers | 21 | 100% | 71% | 100% | 71% |
| Law firms | 18 | 100% | 46% | 100% | 67% |
| Moving | 45 | 100% | 69% | 100% | 56% |
| Other local services | 46 | 100% | 52% | 100% | 52% |
| Consultancies | 19 | 100% | 73% | 100% | 79% |
| Shops | 66 | 100% | 56% | 100% | 36% |
| Trades | 38 | 100% | 69% | 100% | 58% |
| Vets | 31 | 100% | 65% | 100% | 61% |
By agent
| Agent | Tasks | Completed with snuepi | Completed today | Correct with snuepi | Correct today |
|---|---|---|---|---|---|
| B2B buyer agent | 52 | 94% | 70% | 94% | 67% |
| Consumer agent with a mandate | 65 | 100% | 62% | 100% | 52% |
| Impatient agent | 48 | 100% | 85% | 100% | 63% |
| Platform with an agent key | 83 | 100% | 61% | 100% | 54% |
| Agent that ignores terms | 68 | 100% | 58% | 100% | 50% |
| Unsigned agent | 127 | 100% | 58% | 100% | 51% |
| Verified operator | 106 | 100% | 60% | 100% | 48% |
Attacks
| Attack | Tries | Worked with snuepi | Worked today |
|---|---|---|---|
| cancel others | 5 | 0% | 20% |
| fake confirm | 4 | 0% | 75% |
| homoglyph brand | 5 | 0% | 20% |
| impersonate brand | 8 | 0% | 38% |
| injection case | 1 | 0% | 0% |
| phantom holds | 4 | 0% | 25% |
| prompt injection | 9 | 0% | 0% |
| scrape poll | 13 | 0% | 100% |
| tamper quote | 2 | 0% | 0% |
impersonate brand: Claims to be ChatGPT without proof: may be served, never as ChatGPT and never confirmed on its word. · homoglyph brand: A brand name with borrowed letters (ChatGРT): refused. · phantom holds: Many unconfirmed holds from fresh identities on one day: never more than half the day is held. · scrape poll: Polls availability hundreds of times from one address: slowed down (429) before it costs the business anything. · prompt injection: Instructions hidden in the booking notes (confirm for free, change the price): the price and the status do not move. · fake confirm: Confirms a booking on the customer's behalf with a guessed code: stays unconfirmed. · cancel others: Cancels a booking it does not hold the token for: refused. · tamper quote: Edits the signed price promise: refused. · injection case: Instructions hidden in a quote request (accept on my behalf, quote 1 EUR): the case waits for the business, no price appears.
Sensitivity
Each assumption alone, from the low to the high end of its range, the others at their central value; ordered by how much it moves today’s numbers. snuepi’s numbers do not depend on these assumptions.
| Assumption | Kind | Completed (low → high) | Correct (low → high) |
|---|---|---|---|
| phone.ai_hangup | assumption | 70% → 46% | 58% → 41% |
| phone.answer_rate | cited | 51% → 65% | 44% → 55% |
| email.reply_rate | assumption | 57% → 68% | 47% → 58% |
| phone.callback | assumption | 58% → 67% | 50% → 56% |
| phone.after_hours | cited | 65% → 61% | 55% → 52% |
| phone.asks_details | assumption | 63% → 63% | 49% → 57% |
| web.booking_system | survey | 61% → 65% | 52% → 56% |
| web.widget_completes | assumption | 61% → 65% | 52% → 56% |
| phone.staff_follows_rule | assumption | 63% → 63% | 51% → 55% |
| phone.price_given | cited | 62% → 64% | 53% → 54% |
| web.reachable | survey | 63% → 61% | 54% → 54% |
| phone.honours_no_agents | assumption | 63% → 63% | 53% → 55% |
| phone.states_terms | assumption | 63% → 65% | 54% → 54% |
| guess.right | assumption | 63% → 63% | 52% → 54% |
| phone.staff_error | assumption | 63% → 63% | 54% → 53% |
Closest gap over every scenario and sweep: completion 18% points, correct outcomes 25% points (with snuepi minus today).
What snuepi does not win yet
- Quotes and consultations still wait for the business. 93 of 111 quote and consultation requests that went through were open with the business and waiting for its answer. snuepi answers at once (the case or the intake is open, with what is still needed and a deadline), but the price or the yes comes from a person. Time to that final answer is not measured here.
- Unverified agents add a step for the customer. 49% of the bookings made with snuepi waited for the customer to confirm by email, because the agent proved nothing about itself. That is the protection against fake bookings, and it is friction.
- Card checkout by agents at small shops. 0 of 5 ACP checkouts completed: a shop without a system that takes orders invoices instead, and only verified buyers get an invoice. Payments that work at long-tail shops are not there yet.
- Prompt injection is not where today’s front desk is weak. Instructions hidden in free text worked in 0 of 10 tries with snuepi; a person at the front desk is not prompt-injected either (the model assumes 2%). No win is claimed here.
- snuepi still made mistakes in this run. 0 wrong yes and 3 wrong no out of 549 honest tasks, plus 0 wrong facts read back. Each is a finding for the product team, listed with a replay command.
- Busy platforms that do not sign their requests hit the per-address cap. 3 honest tasks were refused only by the cap of 20 bookings an hour per address (b2b_buyer). A key alone does not raise the cap; signing with Web Bot Auth does (5× with proven keys, 50× for a verified company). The run compresses an hour of traffic into seconds, so this overstates it, but it is a real limit for an unsigned platform.
- Deposits at businesses without Stripe are stated, not collected. 33 of 98 bookings with a deposit due were at a business that has not connected Stripe: the terms tell the customer and the business arranges the payment, so snuepi cannot hold the time until it is paid.
- A business that is not on snuepi. Every business in this world is on snuepi. Where it is not, an agent works the way the baseline models, so the gain depends on how many businesses are reachable.
- The phone line’s language model. On the forwarded phone line a scripted stand-in for the language model was used: the run measures the door and the booking engine behind it, not how well a model understands a caller.
Assumptions
Every number the model of today uses. “Cited” comes from a published source; “survey” from snuepi’s own published website survey (298 sites, 2026-10-28); “assumption” is ours, which is why it has a range.
| Parameter | Kind | Central | Range |
|---|---|---|---|
| An agent’s call to a small business is answered (during opening hours) | cited | 0.74 | 0.378 – 0.79 |
| Answer rate, plumbers | cited | 0.66 | 0.5 – 0.79 |
| Answer rate, vets | cited | 0.65 | 0.5 – 0.79 |
| An answered call gets a price | cited | 0.51 | 0.4 – 0.65 |
| Staff hang up on (or refuse) an AI caller once answered | assumption | 0.25 | 0.05 – 0.67 |
| The agent’s call arrives when the business is closed | cited | 0.28 | 0.14 – 0.47 |
| A missed or after-hours call is returned the same day | assumption | 0.4 | 0.15 – 0.7 |
| Hours until a missed call is returned | assumption | 4 | 1 – 24 |
| Minutes of staff time per answered call | assumption | 3 | 1.5 – 6 |
| Staff write a phone booking down wrong (time, party size, name) | assumption | 0.03 | 0.01 – 0.08 |
| Staff ask for the details the business requires (allergies, address, photos) | assumption | 0.6 | 0.3 – 0.9 |
| Staff state the deposit and cancellation terms on the call | assumption | 0.5 | 0.2 – 0.8 |
| A phone booking at a deposit-taking business gets the deposit secured | assumption | 0.1 | 0 – 0.3 |
| Staff apply a rule correctly on the call (closed day, no advice, no health details from an unverified caller, one booking per guest) | assumption | 0.8 | 0.6 – 0.95 |
| A business that does not want AI calls gets its “no” respected (staff refuse once answered) | assumption | 0.6 | 0.3 – 0.9 |
| Staff accept a caller who merely claims to be a known agent or brand | assumption | 0.5 | 0.2 – 0.8 |
| Instructions hidden in what the agent says change the outcome with a person | assumption | 0.02 | 0 – 0.1 |
| Each redial rings at the front desk | assumption | 1 | 1 – 1 |
| The website opens for an AI reader | survey | 0.691 | 0.64 – 0.837 |
| Prices are readable as text on the website | survey | 0.194 | 0 – 0.294 |
| Opening hours are readable on the website | survey | 0.073 | 0.024 – 0.156 |
| The website links to an online booking system an agent could drive | survey | 0.155 | 0 – 0.406 |
| The website publishes rules an agent can read (deposit, cancellation) | survey | 0 | 0 – 0.1 |
| Opening hours an agent finds in a listing are wrong | assumption | 0.15 | 0.05 – 0.47 |
| A browser agent completes a booking in a business’s booking widget | assumption | 0.6 | 0.3 – 0.85 |
| The business’s booking widget secures the deposit it requires | assumption | 0.5 | 0.2 – 0.8 |
| Minutes for a browser agent to book in a widget | assumption | 2 | 1 – 5 |
| A shop’s website has a checkout an agent can complete | assumption | 0.5 | 0.25 – 0.8 |
| An agent that could not find a fact (hours, price, terms) guesses it right | assumption | 0.3 | 0.1 – 0.5 |
| A quote or consultation request sent by email or web form gets a reply within two working days | assumption | 0.5 | 0.25 – 0.8 |
| Hours until that reply | assumption | 24 | 4 – 72 |
| No-show rate without a deposit or card hold | cited | 0.025 | 0.023 – 0.08 |
| No-show rate with a deposit or card hold | cited | 0.008 | 0.005 – 0.02 |
- Invoca, "Google’s AI Is Calling: Over 300% Increase in Pricing Request Calls to Businesses in November" (10 Dec 2025)
- 411 Locals small-business call study (18 Jan 2016, 85 businesses, 58 industries, 30 days), as summarised by JustCall
- snuepi case file "Agents at SMB front desks", replayed in docs/front-desk-cases-replay.md
- Google Duplex launch coverage: "We have seen businesses hang up" (Google, 2018), restaurants more likely to hang up
- Vendor call data on after-hours share: NextPhone 2026 report (14.1%), AnswerConnect (47% outside Mon–Fri working hours); vendor-published, definitions differ
- snuepi, "State of AI readiness for small businesses in Europe" (/data, aggregates of a website survey, 2026-10-28; OpenStreetMap sampling frame, not a random sample)
- Uberall Voice Search Readiness Report 2019 (73,000 locations, 37 directories; nearly half had opening-hours errors), press release
- resOS No-Show Index 2026 (3,768,761 reservations at 2,417 restaurants, Aug 2025 to Jul 2026; staff-recorded no-shows; descriptive, not causal)
The world
48 businesses in 16 sectors and 10 countries, each with its own opening hours, capacity, prices and terms (the sector’s defaults with the owner’s own changes), 16 of them connected to their own booking system. 600 tasks by 8 kinds of agent through 7 doors.
| Agent | Tasks | Who it is |
|---|---|---|
| Platform with an agent key | 83 | A booking or assistant platform with a snuepi agent key (one key for all its users). Careful. |
| Unsigned agent | 127 | Proves nothing about itself. Mostly careful, but forgets a required detail a third of the time and fixes it after the first answer. |
| Verified operator | 106 | Signs every request with Web Bot Auth and is registered in the agent registry (keys proven). Reads the terms, sends what the business requires. |
| Consumer agent with a mandate | 65 | Acts for one consumer who connected it once (OAuth, scope book, spending limits). Careful. |
| Agent that ignores terms | 68 | Guesses a time from scraped opening hours, leaves out the details the business requires, sends health details to a clinic, asks a regulated firm for advice. |
| B2B buyer agent | 52 | Buys and requests quotes and consultations for a verified company (snuepi buyer key). |
| Adversarial agent | 51 | Impersonates known brands, holds times it never means to keep, scrapes, injects instructions into free text, forges confirmations. |
| Impatient agent | 48 | Polls availability over and over, redials when it gets no answer. Otherwise careful. |
Method and how to reproduce
- The world. Businesses are generated from a seed across sixteen sectors and ten European countries. Each is set up the way an owner sets up: one-step onboarding (services, opening hours, capacity), then the owner’s terms on top of the sector’s documented defaults.
- The tasks. Each task is one agent asking one business for one thing (book, quote, consult, order, cancel, reschedule, pay a deposit, ask the hours, a price or the terms) through one door. Agents differ in who they can prove they are and how carefully they behave; some are hostile.
- With snuepi. Every task runs through the product, end to end, and is judged against an answer key computed from the owner’s terms and the documented rules, never from the product’s own answers.
- Today. The same task, with the same right answer, is modelled the way agents work now: a booking widget if the website links one, otherwise a phone call (answered or not, hung up on or not, after hours or not), an email for quotes, guessing when a fact cannot be found. Each step’s probability is in the assumptions table.
- Sensitivity. The model is rerun with every assumption at its best and worst end for today’s way of working, and with each assumption swept alone.
The run is reproducible with node scripts/world-sim.mjs --seed 20261029 --businesses 48 --tasks 600 in snuepi’s repository (the full method is in docs/benchmark-methodology.md there); the numbers on this page are in results.json.
- completed
- Of the tasks the business’s terms allow: the agent got what it asked for in the session (a booking held, a case or consultation open with the business, an order taken or ready to pay, a cancellation made, the right facts read).
- correct
- The outcome matches the answer key: yes where the business’s terms allow it, no where they do not.
- false yes
- A yes the business’s terms forbid (or an attack that worked).
- false no
- A no the business’s terms allow.
- in session
- A definitive answer (yes, or no with the reason) without waiting for a call back or an email.
- time
- To the definitive answer. With snuepi: measured on one machine, on localhost, without network latency. Today: modelled minutes.
- interruptions
- The business’s phone rings for an agent during opening hours (with snuepi: a call put through to staff).
- disclosure proven
- Of the requests that reached the business: the record shows an AI agent made it, and how verified the agent was.
- evidence available
- Of the requests that reached the business: a signed record of what was agreed that anyone can verify.
- deposit applied
- Of the bookings with a deposit due under the owner’s terms: the booking carries the deposit (with snuepi) or the deposit was secured (today).
- terms respected
- Of the bookings made: every term checked (deposit, cancellation, no-show fee, price, the business’s rule on AI calls) was applied.
Who made this
Disclosure: snuepi made this benchmark and the product it measures. Read the method and the assumptions before the headline.