Idempotent AI Agents: Retry-Safe Architecture for Ecommerce

The first time it happened, the brand's controller called at 11pm on a Tuesday. A customer had reported a $412 refund posting twice on the same credit card. Our LLM agent had called refund_order against the Shopify Admin API, the HTTP connection stalled for 47 seconds, the model decided the call had failed, and it retried. The first call had already landed. The customer got their money back twice, the brand was out the money, and the agent's trace looked clean because from the model's point of view, only one refund was ever issued.

Idempotent AI agents in ecommerce production are not a research problem. They are a production engineering problem with a specific shape, and the shape has gotten worse as more brands have let LLM agents touch real money. The thesis of this post is that the model itself cannot own retry safety. The harness around it has to, and the harness needs four layers: deterministic idempotency keys at the tool boundary, an intent ledger that records every dispatch, an outbox-and-worker pattern for external side effects, and a reconciliation job that catches what the live path missed. We have built this for two mid-market Shopify Plus brands and rescued it for a third. The patterns below come out of that work, including the parts that broke.

The business problem with retries in ecommerce

Retries are not optional. They are how distributed systems stay up. The infrastructure path between an LLM agent and a Shopify order touches at least four retry surfaces, each of which can fire the same logical write twice.

Shopify webhooks are delivered at-least-once. If your endpoint returns a 5xx or times out, Shopify retries the same payload, sometimes for hours. Our agent's queue runs on BullMQ 5, which retries jobs on worker restart and on unhandled exceptions. The LLM HTTP call to Anthropic or OpenAI can time out at the SDK level and the SDK can retry transparently. The model itself, when streaming reconnects after a network blip, will sometimes re-emit a tool call it already sent because the conversation state on its side reset.

Shopify's April 2026 changelog made idempotency keys mandatory on inventory adjustment and refund mutations in Admin API 2026-04. The previous version made them optional. The version before that did not support them at all. The platform's own engineering team has put a stake in the ground: writes that move money or stock have to carry a key, and the server will reject calls that do not. That is the right default, but it puts the burden of generating stable, deterministic keys on the caller, and an LLM agent is the worst possible client for that job because the model is non-deterministic by construction.

The cost shape of a duplicate write is heavy out of proportion to its rarity. A single duplicate refund pulls a customer-service agent into a reconciliation thread that takes an hour to close. A negative inventory on the warehouse floor produces a backorder email three days later and a customer-trust hit that takes months to recover. A duplicate order creates a duplicate fulfillment, which creates a duplicate shipping label, which is real money the brand will not get back. We have measured the operational cost of duplicate writes for one brand: 4-6 hours of finance team time per incident, plus the lost margin on the duplicate transaction. At even one incident per week, the math forces a fix.

We covered the broader integration failure modes in why your Shopify-to-NetSuite sync keeps breaking. This piece is the agent-specific extension: what changes when the writer is not a deterministic transformer but an LLM-driven agent that can re-decide to call the same tool.

Why AI agents make duplicate writes worse

Classical integrations have one source of duplicate writes: the inbound webhook redelivers, and the integration handler is not idempotent on the inbound event id. The fix is well known. The integration stores the Shopify event id in a Postgres table with a unique constraint, and a redelivery short-circuits before it touches NetSuite or any other external system.

LLM agents add three new sources, and the inbound-event-id pattern does not catch any of them.

The first is streaming reconnects mid-tool-call. The model emits tool_use: refund_order mid-stream. The SDK's HTTP/2 connection drops. The SDK reconnects and asks the model to resume. Depending on how the agent's state is checkpointed, the model can re-emit the same tool call because from its perspective, the prior call was never executed. We have seen this happen on the Anthropic SDK with stream=True when the network path between Railway and the API endpoint had a transient blip. The model's tool-call id was different on the resumed stream. The agent's dispatcher could not tell that the second tool_use was the same logical write as the first.

The second is the model deciding to retry on its own. We instrument every tool call with a structured response back to the model. If the response is delayed past the LLM's HTTP timeout, or if the response payload is missing for any reason, the model will sometimes call the tool again "to be safe." Anthropic and OpenAI both publish guidance that the SDK handles retries for you. That guidance is about HTTP-level retries inside the SDK. It does not stop the model itself from emitting a second tool-use block on a fresh turn, and we have logs showing this happen with Claude Sonnet, GPT-4.1, and Gemini 3.1 Flash. The model is doing what a well-trained agent should do when uncertain: re-attempt the action. That instinct collides with production state.

The third is parallel tool-call dispatch. Both Anthropic and OpenAI now support emitting multiple tool calls in a single response block, intended to be executed in parallel. We use this for read-side calls, where running get_customer, get_order_history, and get_inventory in parallel cuts agent latency in half. The problem appears when the model emits two write-side calls that touch the same resource. We have seen adjust_inventory(sku="WIDGET-A", delta=-3) and adjust_inventory(sku="WIDGET-A", delta=-3) emitted in the same parallel block because the model's reasoning chain split into two branches that both decided the adjustment was needed. The dispatcher fanned both calls out. Inventory went negative.

None of these three failure modes look like a duplicate inbound webhook. They look like distinct tool calls from the agent's perspective. An idempotency layer that keys on (webhook_event_id) will not catch them. The layer has to key on the logical intent the agent was trying to express, not on the call site that expressed it.

The model is doing what a well-trained agent should do when uncertain: re-attempt the action. That instinct collides with production state.

Architecture: the four layers of an idempotent agent harness

The design below builds up from the boundary inward. Each layer solves one specific source of duplicate writes. Skipping a layer is fine until it isn't, and we will show in the next section what specifically broke for us when we shipped without each one.

Tool-level idempotency keys

Every tool the agent can call against an external system computes its own idempotency key. The key is deterministic over (tool_name, args, intent_id). The intent_id is a stable UUID the orchestrator assigns when the agent begins reasoning about a user request. The orchestrator passes the same intent_id into every tool call the agent dispatches during that reasoning turn.

// services/idempotency.ts
import { createHash } from 'crypto';
 
export function computeIdempotencyKey(
  toolName: string,
  args: Record<string, unknown>,
  intentId: string,
): string {
  // Stable JSON sort so {a:1,b:2} and {b:2,a:1} hash the same
  const canonical = JSON.stringify(args, Object.keys(args).sort());
  const hash = createHash('sha256')
    .update(`${intentId}:${toolName}:${canonical}`)
    .digest('hex');
  return `idem_${hash.slice(0, 24)}`;
}

Two different tool calls with the same intent_id, same tool name, and same args produce the same key. The agent can emit the same tool_use block fifty times across a streaming reconnect storm; every emission resolves to one key. That key is what we pass to Shopify's @idempotent directive on the GraphQL mutation, and it is what we use to dedup inside our own dispatch layer.

The intent_id does the heavy lifting. It is not the tool-call id from the LLM SDK, because that id changes on a stream resume. It is not the conversation id, because the same conversation can have many independent intents. It is created when the orchestrator decides "this user message produces one logical outcome," and it sticks to every write the agent dispatches in that turn.

The intent ledger

A Postgres 16 table records every dispatched tool call: the intent id, the tool name, the idempotency key, the call status, and the canonical response from the external API. Before the dispatcher calls the external service, it checks the ledger. If the key exists with a terminal status, it returns the stored response without re-calling.

// services/intent-ledger.ts
export async function dispatchTool(
  toolName: string,
  args: Record<string, unknown>,
  intentId: string,
  externalCall: () => Promise<ToolResponse>,
): Promise<ToolResponse> {
  const key = computeIdempotencyKey(toolName, args, intentId);
 
  // Single-statement upsert: returns prior row if it exists,
  // otherwise inserts 'in_progress' and lets us proceed.
  const existing = await pg.oneOrNone(
    `INSERT INTO intent_ledger(idem_key, intent_id, tool_name, status)
     VALUES ($1, $2, $3, 'in_progress')
     ON CONFLICT (idem_key) DO UPDATE
       SET idem_key = EXCLUDED.idem_key
     RETURNING status, response`,
    [key, intentId, toolName],
  );
 
  if (existing && existing.status === 'completed') {
    return existing.response;
  }
  // ... call externalCall(), persist response, mark completed
}

The ON CONFLICT DO UPDATE is the load-bearing piece. A naive read-then-write check has a race between two parallel dispatches that both see no row, both decide to proceed, and both call the external API. The single-statement upsert with a unique constraint on idem_key forces the race into Postgres, which is built to win it. The first writer wins the row; the second writer sees the existing row and returns the stored response.

We use a partial index on (status) WHERE status = 'in_progress' to spot stuck dispatches. A row stuck in in_progress for more than 90 seconds is a worker that died mid-call, and the reconciliation job picks it up.

The outbox and worker pattern for external side effects

The intent ledger handles same-process retries. The outbox handles the case where the agent's process dies between deciding to call the tool and the call actually landing. The agent writes its intended side effect to an outbox table in the same database transaction that updates the conversation state. A separate worker process drains the outbox and makes the external call.

// services/outbox-worker.ts
async function drainOutbox() {
  const jobs = await pg.any(
    `SELECT id, tool_name, args, idem_key, attempts
     FROM agent_outbox
     WHERE status = 'pending' AND attempts < 5
     ORDER BY created_at
     FOR UPDATE SKIP LOCKED
     LIMIT 25`,
  );
  await Promise.all(jobs.map(async (job) => {
    try {
      // Shopify Admin API 2026-04 carries the key on the @idempotent directive
      const result = await shopify.callWithKey(job.tool_name, job.args, job.idem_key);
      await markCompleted(job.id, result);
    } catch (err) {
      await scheduleRetry(job.id, err);
    }
  }));
}

FOR UPDATE SKIP LOCKED is what lets multiple worker replicas drain the same outbox without stepping on each other. Postgres locks the rows the first worker grabbed; the second worker's query skips them and grabs the next batch. We run three worker replicas in production. They share the outbox, and the row-level locking keeps them coordinated.

The model can call enqueue_refund fifty times in a streaming hiccup. The outbox dedups on idem_key (unique constraint), so only one row ever lands. The worker drains that row once. Shopify sees one refund mutation, with one idempotency key. The customer sees one credit on their card.

This is the transactional outbox pattern AWS documents in their prescriptive guidance, applied to LLM tool calls. The wrinkle is that the "transaction" that writes to the outbox also includes the agent's conversation state update, so the model can never be told "your call succeeded" until the side effect is durably queued.

The reconciliation job

The layers above catch duplicate writes the live path knows about. They do not catch the case where the agent thought a call succeeded but the external system silently dropped it, or the case where a webhook redelivery slipped past the outbox dedup because of a database failover. A nightly reconciliation job pulls state from Shopify, NetSuite, and the agent's intent ledger, diffs them at the resource level, and emits alerts.

The reconciliation is the safety net, and we treat it that way. It does not auto-correct. It generates a list of discrepancies for the operations team to review, because by the time a discrepancy reaches reconciliation, the right action requires human judgment. A refund that landed in Shopify but not in our ledger could mean the agent retried successfully and our ledger write failed, or it could mean a CS rep issued the refund manually from the Shopify admin. The reconciliation cannot tell the difference. It tells a human to.

What broke when we skipped a layer

Each layer below got added because something specific broke. The order matters because every team we have worked with thinks they can skip one of these and "add it later if needed."

Skipping tool-level idempotency keys. We shipped the first version with just the intent ledger, keyed on the LLM SDK's tool-use id. The id changed on stream resumes. A Claude Sonnet 4.5 streaming reconnect during a refund flow produced two distinct tool-use ids for the same logical refund. The ledger saw them as different calls, dispatched both, and the customer got refunded twice. The fix was to compute the key deterministically from args and intent id, ignoring the SDK's id entirely.

Skipping the intent ledger. A second brand pushed back on the ledger because "Shopify's idempotency keys handle this already." That is true for Shopify, but the agent also writes to a CRM (HubSpot), a 3PL (ShipBob), and an email system (Klaviyo). None of those carried mandatory idempotency in their 2026 APIs. The agent retried a notify_customer call against Klaviyo seven times during a brief Klaviyo outage. Seven identical "your order has shipped" emails reached the same customer in 90 seconds. The brand's customer support manager was the first to flag it. We added the ledger that week.

Skipping the outbox. Our first agent ran tool calls inline. The model emitted tool_use, the dispatcher called Shopify, the response went back to the model, and the conversation state was updated. The conversation state lived in Redis with a 24-hour TTL. A Railway deploy restarted the agent process mid-flow, after the Shopify call landed but before the conversation state persisted. The model, when the agent resumed, did not know the refund had succeeded and called it again. This is what forced the outbox pattern. The outbox makes the side effect and the conversation state update part of the same atomic write, and the worker is the only thing that touches the external API.

Testing for idempotency

A test that proves an agent is retry-safe has to do more than assert "called Shopify once." The agent's tool calls have to be safe under arbitrary repetition and arbitrary partial-failure injection. We use a property-based testing approach with fast-check 4.

The test generates a trace of agent tool calls, then replays each call N times with random failure injection between attempts: timeout before response, timeout after response, 500 error, network reset. The assertion is that the final state in Shopify (queried via SuiteQL-style read) matches the state from a single clean run.

// tests/idempotency/refund-flow.test.ts
test.prop([genAgentTrace(), fc.integer({ min: 2, max: 10 })])(
  'refund flow is safe under N retries with failure injection',
  async (trace, retryCount) => {
    const cleanState = await runOnce(trace);
    const chaosState = await runWithChaos(trace, { retryCount, failureRate: 0.4 });
    expect(chaosState.refunds).toEqual(cleanState.refunds);
    expect(chaosState.inventory).toEqual(cleanState.inventory);
  },
);

This test caught the streaming-reconnect bug before it shipped to the second brand. The failure-injection harness ran a synthetic stream interruption mid-refund_order, and the assertion failed because two refunds posted in the chaos run versus one in the clean run. That was the signal that the SDK's tool-use id was the wrong key.

The property-based approach matters because the failure modes are combinatorial. Three sources of retries times five tool types times four failure-injection points is sixty distinct test cases, and the property-based generator covers them more honestly than hand-written scenarios.

What this architecture enabled

The wins were not subtle. The dollar-cost of duplicate writes dropped to zero across the measurement window. The customer-trust hits stopped. The brand's CS team stopped getting tickets that started with "I was charged twice." None of that is glamorous, but the alternative is a controller calling at 11pm, and we have heard that call once too many.

Observations

What worked. The four-layer model has held up under six months of production load across two brands. The intent ledger is the single most valuable layer because it gives us a queryable audit trail of every write the agent attempted, which has been worth its weight in finance team meetings. The outbox pattern is what makes the architecture survive Railway restarts and deploy windows without losing in-flight intents.

What didn't. Early on we tried to make the model itself responsible for generating idempotency keys, prompting it to include a key in every tool_use block. The model would generate keys, but they were not stable across retries because the model is non-deterministic. A retry would produce a different "deterministic" key, which defeated the entire point. The lesson is that idempotency cannot be the model's job. It has to be the harness's job.

What we'd do differently. We would build the reconciliation job first, before the live agent path. We made the same mistake described in the Shopify-to-NetSuite sync post: we built the satisfying live integration first and bolted on reconciliation later. The first three months were spent chasing live-path failures that the reconciliation would have caught in a single nightly run. We would also invest earlier in the property-based test harness; we wrote it after the second duplicate-refund incident, and the bugs it caught in its first week would have saved us those incidents. Anthropic's published tool-use guidance and OpenAI's tool-use docs both describe SDK-level retry behavior, but neither addresses the agent-level retry sources we cataloged above. That gap is where the harness has to do the work.

For the broader agent-harness context this fits into, the patterns in building harnesses for operational AI agents and the connector-selection framing in choosing a Shopify-NetSuite connector cover the surrounding architecture.

References

1. Shopify Engineering, "Making idempotency mandatory for inventory adjustments and refund mutations," Shopify developer changelog, April 2026. https://shopify.dev/changelog/making-idempotency-mandatory-for-inventory-adjustments-and-refund-mutations
2. AWS Prescriptive Guidance, "Transactional outbox pattern," AWS cloud design patterns, 2026. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
3. AWS Prescriptive Guidance, "Retry with backoff pattern," AWS cloud design patterns, 2026. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html
4. Anthropic, "Tool use with Claude," Claude API documentation, 2026. https://docs.anthropic.com/en/docs/build-with-claude/tool-use
5. OpenAI, "Using tools," OpenAI API documentation, 2026. https://developers.openai.com/api/docs/guides/tools
6. Confluent, "Exactly-Once Semantics is Possible: Here's How Apache Kafka Does it," Confluent blog. https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
7. Shopify GraphQL Admin API, "inventoryAdjustQuantities mutation reference," 2026. https://shopify.dev/docs/api/admin-graphql/latest/mutations/inventoryAdjustQuantities

Stack: Node.js 20, TypeScript, Postgres 16 for the intent ledger and outbox, BullMQ 5 for the worker queue, p-limit 5 for outbound concurrency governance against Shopify, fast-check 4 for property-based idempotency tests, deployed on Railway. The agents talk to Shopify Admin API 2026-04 and Anthropic's Claude Sonnet via the official SDK with streaming enabled. Customers were two mid-market Shopify Plus brands doing 18,000-40,000 orders a month.