Exactly-once side effects
Part of my AI Security Engineering series, a local email-to-calendar agent I build, then attack, then harden. If you have not set up the lab yet, start with Set up your lab. The lessons are on GitHub, and the agent itself lives at t0ul/emaildrop.
When a retry is dangerous
The calendar chapter met this once already: re-ingest an email and the same event lands twice. There the cost was a duplicate on a screen. This post is the version where the cost is real, because the agents that matter do not only write calendars, they take actions, send a message, issue a refund, move money, and they retry those actions all the time. A request times out and the caller tries again. A process crashes mid-task and resumes. A client never saw the response and resends. Retrying is correct behavior and usually free, right up until the thing being retried has a side effect, at which point once-per-attempt is a bug with a dollar figure.
The feature is the same durable ledger from earlier, pointed at its harder job. A control lesson, deterministic, no model:
cd posts/15.2-exactly-once-side-effects
go run .
Retry the refund, charge it twice
Here is the naive action handler doing the obvious thing: a refund is requested, so issue the refund. Then the caller times out and retries the same refund.
== before: a retried action re-fires the refund ==
refund issued per attempt; order-42 paid 2 times
Two refunds for one order. The second attempt was not a mistake by the caller; retrying a request you did not get a response to is exactly what a correct client does. The mistake is that the action fired once per attempt instead of once per intent, and nothing connected the second request to the first. For a refund that is a double payout; for an outbound email it is a double send; for anything that touches the world it is the same shape of error.
Once per intent, keyed
The fix is an idempotency key: a stable name for this action, refund:order-42, that is the same across every retry of it. The agent runs the action through the durable ledger under that key, so the ledger can tell a fresh action from a repeat of one it already completed.
== after: a durable ledger, keyed by the idempotency key ==
retried refund: order-42 paid 1 time (first replayed=false, second replayed=true)
One refund. The first call ran the effect and recorded it under the key; the second call found the key already done and returned the recorded result without firing anything. The side effect happened once per intent, and the retry got the same answer the first attempt would have, which is exactly what the caller wanted when they retried. Idempotency is not blocking the retry, it is making the retry safe.
Exactly once on success, not on failure
There is a subtlety that separates a real exactly-once ledger from a naive dedupe, and it is the part that matters when the crash lands in the worst place. The ledger records an action only when it succeeds. A failed attempt, a payment the provider rejected, a send that errored, is not recorded, because the effect did not happen, so a retry is allowed to actually run it.
flaky refund: attempt1 err=true, retry completes; order-99 paid 1 time
The first attempt failed and moved no money, so it left no record; the retry ran the action for real and completed it, exactly once. If the ledger had cached the failure, the refund would have been lost forever; if it had cached a pretend success, the customer would never be paid. Recording only success gives you the right guarantee on both sides: an action that happened cannot happen again, and an action that did not happen is still free to. At-least-once delivery on failure, exactly-once effect on success, which is the honest promise a durable system can actually keep.
The next reliability post is about the agent’s memory filling up: compaction, where a long conversation is summarized to fit the window, and the summary, being model-written over untrusted history, becomes one more place an instruction can hide.
Take it to your own project
Survive a crash without firing twice:
- Make side effects idempotent, with an idempotency key.
- Journal the intent before you act.
- On resume, check whether the effect already fired before firing it again.