The SmallOffice Brief Subscribe
Customer Service
Mon, Nov 2, 2026
Support

212 customers, refunded twice in 38 minutes

A business ran a customer support agent with a refund tool. Before launch, the agent passed an offline eval at 96%. The eval used 400 real, anonymised conversations, each run three times. One prompt line said: if a tool call fails, retry once. In production the payments system was slow. Its typical slow response was 4.8 seconds, and the tool timed out at 3 seconds. So calls marked failed had in fact gone through, and the retry paid a second refund.

How it wentIn 38 minutes at full traffic, 212 customers got two refunds each. Error rates looked normal. A customer ticket and a finance check found the problem, not the monitoring. The business is unnamed in the post.
Try it · based on the source's workflow
Before any agent at [business name] can use a tool that moves money, run this check. Test the tool with real response times, including slow and timed-out calls. Each conversation may call a money tool at most once, and every call carries a unique key so a retry cannot pay twice. Retries live in the tool layer, never in the prompt. Launch to [5%] of traffic first, and show me the test results and the first day's refund log before any wider launch.
A starting template based on the workflow the source describes. Review it before you use it; you are responsible for what it does.
Source: dev.to (Rakesh Kumar) · Topic: Customer Service · Tool: not named in the source · Other or custom cases
ShareXLinkedInForward to a friend
Cite this case212 customers, refunded twice in 38 minutes. The SmallOffice Brief, Mon, Nov 2, 2026. https://smalloffice.space/case-212-customers-refunded-twice-in-38-minutes
Share this win
212 customers, refunded twice in 38 minutes — from The SmallOffice Brief, a daily brief of real AI cases for small businesses: https://smalloffice.space/case-212-customers-refunded-twice-in-38-minutes
One bot, one job

Start with one job only. Fill in the three lines. Keep the bot on that job.

The one job this bot does: ______________________________

What it may do on its own: ______________________________

What it must ask me first: ______________________________

Before and after

Two lines from this case, side by side. No new figures.

Before · as the case describes it

A business ran a customer support agent with a refund tool. Before launch, the agent passed an offline eval at 96%. The eval used 400 real, anonymised conversations, each run three times. One prompt line said: if a tool call fails, retry once. In production the payments system was slow. Its typical slow response was 4.8 seconds, and the tool timed out at 3 seconds. So calls marked failed had in fact gone through, and the retry paid a second refund.

After · as the source reports it

In 38 minutes at full traffic, 212 customers got two refunds each. Error rates looked normal. A customer ticket and a finance check found the problem, not the monitoring. The business is unnamed in the post.

The after line is the source's claim, not our measurement.

smalloffice.space · Get a case like this in your inbox each morning. Subscribe free

Send this case to my inbox

Read this case on your phone now, and keep it for later. Type your email. We send this case to that address, and you get the next cases each morning too.

Additional Details

Published: Mon, Nov 2, 2026 · Updated: Mon, Nov 2, 2026 · Topic: Customer Service · Tool: not named in the source · Other or custom cases · Source: dev.to (Rakesh Kumar)
Next case: Email: two hours a day to 10 minutes. Get cases like this in your inbox each morning.
5 cases in your inbox each morningSubscribe free