March 16, 2026

When Webhooks Retry, Users Shouldn’t Notice: Idempotency Locks with Redis

Challenge	Approach	Results
Webhooks are at-least-once: duplicates and retries are normal Deliveries can arrive concurrently and out of order Duplicate work can leak into user-facing side effects (confusing state, repeated updates, inconsistent timelines)	Treat cache as a coordination primitive, not a read-optimization layer Acquire a short-lived idempotency lock per transfer-status before side effects Use deterministic keys + TTLs to stay safe and self-healing	More consistent user-facing outcomes during retry bursts Effectively-once processing per transfer status within a bounded window Lower database and event-bus pressure, plus fewer duplicate downstream actions

Why we wrote this

Fintech webhook pipelines live in the real world: vendor retries, network timeouts, and horizontal scaling are all expected. In money movement, “processing the same thing twice” is rarely a benign failure mode. It can create duplicate state transitions, duplicate event emissions, and operational noise that slows teams down.

This post shares a portable pattern in webhook processing: a Redis-backed idempotency lock with a bounded TTL.

What this feels like for end users

When duplicate or out-of-order webhook processing leaks into product behavior, the experience can feel inconsistent or untrustworthy, even if the underlying system eventually converges.

Common symptoms :

A status appears to change twice, or briefly “flip” between states
A user sees repeated updates for the same underlying action
Timelines look inconsistent across surfaces (UI, email, receipts, activity feeds)
Support and operations teams spend time explaining what is ultimately a delivery artifact

We treat these as correctness problems with user-visible impact, not just backend quirks.

The UX contract: one state change, one set of outcomes

A helpful way to frame webhook idempotency is as a product contract:

For a given entity and state transition, the system should emit one coherent set of outcomes.
Retries may happen, but retries should not multiply user-facing side effects.

The Redis lock pattern below is one practical way to enforce that contract at the edge of your system.

The problem: distributed systems + money movement

Most vendor webhooks should be assumed to have at-least-once delivery semantics:

If an upstream system doesn’t receive a timely acknowledgement, it will retry.
If upstream systems are degraded, they may replay.
If you scale workers, multiple instances can observe the same transfer update at nearly the same time.

This leads to classic distributed-systems hazards:

Non-deterministic ordering (status updates can arrive out of order)
Concurrent processing (two workers attempt the same unit of work)
Duplicate side effects (state updates, event publication, notifications)

The goal isn’t to stop retries, retries are healthy behavior. The goal is to make retries safe.

Why “cache” helps here (and what it is not)

In this workflow, Redis is not used as a generic caching layer to speed up reads. We use it as a low-latency coordination mechanism: a short-lived lock that prevents duplicate work from touching stateful systems.

A practical way to think about it: a Redis round trip is typically fast enough to absorb “retry bursts” before your database or event bus becomes the coordination point.

Typical access latency ranges (order-of-magnitude):

Access path	Approx. latency
CPU cache (L1/L2)	~1-10 ns
RAM	~50-150 ns
Local SSD read	~50-200 μs
Redis over network	~0.2-2 ms
Database read (indexed)	~2-10+ ms

Alt: Latency comparisons help explain why a fast, bounded Redis lock can suppress duplicates before slower systems are touched. (Values vary by workload and environment.)

The pattern: short-lived idempotency lock per transfer-status

We build an idempotency key for a specific “unit of work,” typically:

A stable transfer identifier (vendor transfer ID or internal correlation ID)
The transfer status (e.g., pending → posted → settled)
A service prefix (helpful when multiple services share Redis)

Why include the status in the key?

We usually want:

“Transfer X transitioned to PENDING” processed once
“Transfer X transitioned to SETTLED” processed once

…but we still need legitimate transitions to proceed. Including status gives us “effectively-once per status” while still allowing real state changes to flow through.

Architecture at a glance

At a high level, the webhook handler becomes “lock-first”. First, here’s the user-impact framing:

Alt: Without idempotency, retries can multiply user-facing side effects. With a lock, duplicates become no-ops.

Now the system view:

Alt: High-level flow, validate, acquire lock, then perform side effects. If the lock already exists, skip and return success.

Sequence diagram: duplicates arrive, one winner proceeds

The key property is that concurrent deliveries compete for the same atomic lock, and only one performs stateful work.

Alt: Only one worker acquires the idempotency lock. The duplicate delivery exits without side effects.

Implementation: what matters (without code details)

The implementation is intentionally simple:

Validate minimal webhook fields (so you don’t lock on malformed payloads).
Acquire an atomic idempotency lock using a deterministic key.
Perform all side effects only if the lock is acquired.
Return success for both “processed” and “duplicate” cases (the upstream system’s job is delivery; your job is correctness).

🧠A useful mental model: the lock protects the most expensive and most stateful part of the handler, database writes, event publication, and any external side effects

Key design choices

1) Deterministic key structure

We recommend a convention like:

servicePrefix:transferId:status

This helps:

Avoid collisions across services
Make keys debuggable and searchable
Keep the unit of work explicit

2) TTL: bounded safety with self-healing

The TTL should be:

Longer than typical vendor retry bursts
Short enough to recover automatically if a worker crashes
Not tied to the full lifecycle of the transfer

This keeps the system resilient: even if a lock is orphaned, it naturally expires and allows reprocessing.

3) What this guarantees (and what it doesn’t)

What this pattern provides:

Effectively-once processing per transfer-status within a bounded time window
Reduced duplicate pressure on durable systems during bursts
Lower probability of duplicate downstream side effects

What it does not claim to provide by itself:

Global exactly-once semantics across infinite time
Perfect ordering of distinct status transitions
A substitute for durable idempotency in downstream consumers

What this enables for end users

With duplicates suppressed at the edge, product behavior becomes more consistent:

Fewer repeated updates for the same underlying action
More stable timelines and state histories
Lower chance of confusing “double effects” during retry storms

What this enables for engineering teams

Downstream systems also become more predictable:

Cleaner state transition histories
Fewer duplicated “derived” events
Less reconciliation noise
More trustworthy operational signals

Most importantly: the system’s behavior under retries becomes easier for engineers to reason about.

Operational guardrails (recommended)

Even with idempotency locks, operations matter. A few practical guardrails help:

Metrics: count lock-acquired vs lock-exists outcomes
Logs: include non-sensitive correlation identifiers and status (avoid storing sensitive payload data)
Alerting: spikes in duplicates can indicate vendor degradation or network issues
Playbooks: document how to respond if retry storms occur

Design principles we followed

Prefer consistent user-facing outcomes over repeated best-effort updates
Treat retries as normal; make duplication harmless
Keep the mechanism simple enough that on-call engineers can reason about it quickly
Share patterns publicly without exposing proprietary code, topology, or sensitive logic </aside>

Key takeaways for engineering teams

Assume at-least-once delivery. Make duplicate deliveries safe by design.
Gate side effects, not parsing. Acquire a lock before database updates and event publishing.
Choose the right unit of work. Transfer ID + status is often a practical boundary.
Use TTLs to stay self-healing. Short-lived locks prevent permanent stuck states.
Share patterns, not internals. You can communicate architecture and trade-offs without exposing proprietary code, topology, or “secret sauce.”

✅ If your webhook handler can safely say “I’ve already processed this exact update,” retries become a reliability feature instead of an incident source.

Written by Emerson Costa