Building KYC That Customers Can Trust: Fast Retries, Consistent Decisions

🔁 How we made KYC processing idempotent so retries, duplicate events, and parallel workers do not create duplicate side effects. We relied on durable step state and database guarantees. This post focuses on patterns, not proprietary implementation details.

Why we wrote this

KYC (Know Your Customer) workflows are inherently messy: upstream systems retry, events can arrive out of order, and verification providers occasionally time out. If your pipeline assumes “exactly once,” it will eventually do the same work twice — and that’s where duplicate side effects and inconsistent state show up.

We wanted a design that:

• Treats retries as normal (not exceptional).
• Resumes cleanly after failures (without manual intervention).
• Stays safe under concurrency (multiple workers, multiple triggers).
• Remains debuggable (so engineers can explain outcomes quickly during incidents).

This post shares the engineering patterns behind an idempotent, resumable KYC pipeline — without exposing proprietary code, infrastructure, or vendor-specific secret sauce.

The member experience we optimized for: speed + consistency

When identity checks are slow or inconsistent, people don’t experience it as “distributed systems are hard.” They experience it as:

• Re-entering the same information after a timeout
• Seeing different outcomes across retries (“it worked yesterday”)
• Getting stuck in loops that feel arbitrary or unfair
• Losing trust that the system is treating them predictably

We optimized for two member-facing outcomes:

1. Speed: minimize repeated work and reduce time-to-complete verification, especially on mobile networks.
2. Fairness & consistency: make outcomes deterministic and stable across retries, duplicate events, and parallel processing.

⚖️ This post uses “retry” to mean safe reprocessing of the same request in the presence of duplicates, timeouts, and crashes. It does not mean retrying identity checks until someone passes. When a user does not meet verification requirements, the outcome is final or routed to review according to policy.

🔁 What we mean by “retry” in SoLo’s KYC experience: a member re-attempting verification after a prior attempt ended due to a technical error, a data issue (e.g., incomplete/incorrect information), or a denial. We design the backend so these repeats are safe and consistent, and we also apply fraud-prevention controls and rate limits so retries can’t be abused. We intentionally keep the specifics internal.

From system guarantees → member outcomes

Sequence: fast retries without “starting over”

This is the practical scenario we cared about most. Someone on mobile hits a timeout, retries, and expects progress, not a reset.

Alt: Every retry first checks durable step state. Completed steps are skipped. Incomplete steps run once and are recorded. Retries are for technical reliability and duplicate requests, not for changing a failed verification outcome.

TL;DR (the pattern)

The core idea is step-level durability:

1. Allow concurrent processing safely — multiple workers may process the same attempt, but DB uniqueness constraints and ON CONFLICT DO NOTHING ensure no duplicate writes occur.
2. Persist each step’s result (success/failure + minimal metadata).
3. Before running a step, check durable step state:
   • If already recorded, restore any required context and skip the step.
   • If not recorded, run it once and record the outcome.
4. Use database uniqueness constraints to enforce “no duplicates,” even under concurrent calls.

This gives you:

• Idempotency: safe to rerun the pipeline.
• Resumability: safe to restart mid-flow.
• Observability: a durable audit trail at step granularity.

What we are (and aren’t) solving

We are solving

• Verification workflows that must tolerate retries, duplicate events, and concurrency
• Avoiding inconsistent outcomes and duplicate side effects under those conditions
• Making progress resume instead of restarting, so the experience stays fast and predictable

We aren’t solving

• KYC policy design (what to verify), risk scoring, or decision thresholds
• Vendor/provider selection or vendor-specific integration details
• UI/UX copy and screen design (we focus on the backend guarantees that enable good UX)
• How to store raw documents or sensitive payloads (we intentionally minimize stored artifacts here)

A simple attempt lifecycle (state model)

To keep outcomes consistent, it helps to treat KYC as an attempt with a small, explicit state machine.

Alt: A minimal attempt lifecycle. The key is monotonic progress. Retries move forward or re-read state, and they don’t restart the attempt.

Consistency guarantees (invariants)

These invariants are what make the system feel fair and predictable to members:

• Completed means completed: once a step outcome is recorded, future retries do not re-execute that step (they only read it).
• No conflicting outcomes: the same attempt/step cannot end up with competing “truths” due to concurrency.
• Side effects are deduped: downstream actions (notifications, blocks, case creation) are keyed so they don’t fire twice.
• Deterministic progress view: retries return a consistent “where you are” view, even when events arrive out of order.

Failure modes we designed for (and the behavior we want)

Note: “retry” scenarios above include both technical/data issues and re-attempts after a denial. Retries are constrained by fraud-prevention controls and rate limits to prevent abuse.

Observability standards (what we measure)

For production workflows, idempotency should be measurable.

Metrics (examples)

• Time-to-verify p50/p95 (overall and per step)
• Step executed vs. skipped counts (skips are healthy in retry scenarios)
• Retry rate per attempt, duplicate-event rate
• Step failure rate (normalized reason codes)
• Resume success rate (attempts that recover after partial failure)

Logging & tracing (principles)

• Correlate logs with an attempt ID and step name
• Log decision codes, not sensitive payloads
• Prefer structured logs and short-lived trace IDs for incident debugging

Privacy & security notes (KYC-specific hygiene)

This pattern pairs well with standard KYC data practices:

• Data minimization: store only the metadata required for determinism and debugging
• No sensitive payloads in logs: redact or avoid PII, documents, and raw provider responses
• Least privilege: restrict access to step state to the minimal set of services/roles
• Retention: apply time-bound retention policies to step artifacts where possible

The problem: “at-least-once” meets “side effects”

Most KYC systems live in an “at-least-once” world:

• Message queues retry.
• Cron jobs rerun.
• Providers respond slowly.
• A service deploy interrupts work.

If your pipeline is a single monolithic function, a retry often means:

• repeating validations,
• re-writing the same records,
• re-sending the same downstream commands,
• and sometimes diverging state across tables.

The result isn’t just waste — it can become correctness risk.

Design goals (industry checklist)

1. Idempotent by default: every step can be retried safely.
2. Resumable: crashes and restarts continue from the last durable checkpoint.
3. Concurrency-safe: multiple workers can race without creating duplicates.
4. Minimal locking: avoid long-lived locks that reduce throughput or become incident fuel.
5. Auditable & debuggable: store enough context to answer “what happened?” quickly.

Architecture at a glance

At a high level, treat verification as an orchestration problem: a controller runs a sequence of steps, each step writes a durable result, and the database enforces uniqueness.

Alt: Orchestrator processes a verification attempt concurrently with other workers, runs steps, persists step outcomes idempotently, and produces a final outcome with an audit trail.

The key idea: step-level persistence

Instead of relying on “the workflow ran,” we persist “which steps ran” and “what they concluded,” so we can safely:

skip completed work,
resume after crashes,
prevent duplicate side effects.

What we persist (intentionally small)

For each step we store:

step name (e.g., “DOB”, “phone”, “address” — names are illustrative)
status (success/failed)
a small metadata payload needed to keep downstream behavior consistent (more on that below)

We explicitly avoid storing secrets or raw provider payloads in this pattern for external-facing designs. The goal is “enough to resume deterministically,” not “store everything.”

Sequence: safe concurrent processing

Idempotency is easier when the system enforces correctness at the data layer rather than at the coordination layer. Rather than preventing concurrent work, we allow it and rely on DB constraints to prevent duplicate writes:

Alt: Both workers process the same attempt concurrently. Safety is enforced at write time — the second INSERT is silently ignored by the database.

Database constraints: your concurrency safety net

When multiple workers race, correctness should not depend on timing. The database can enforce invariants like:

• one step outcome per (attempt, step, status, user) tuple (or whatever key fits your model),
• no duplicate error rows,
• no duplicate “block user” commands.

This pattern is powerful because:

• it is simple to reason about,
• it keeps correctness close to the data,
• and it works even when your application code is called concurrently.

Principle: if the DB can guarantee it with a constraint, prefer that over application-level locks.

Restoring context: why “metadata” matters

One subtle issue: skipping a step should not change downstream behavior.

Example: a step may compute a derived flag or intermediate status that later steps rely on. If we skip the step, downstream logic still needs the same context it would have had if the step executed live.

So we persist a minimal metadata shape such as:

• a status enum / outcome code,
• a small set of derived flags,
• and any normalization decisions required later.

When the step is skipped, the orchestrator:

• loads the metadata,
• rehydrates those flags into the in-memory working model,
• continues deterministically.

This prevents “skip drift” — where skipping completed work accidentally changes the final result.

Handling failures without duplicate side effects

The same step-level approach applies to failures:

• record the failure outcome,
• record a normalized error entry for debugging / UX,
• guard downstream actions so they are not re-triggered on retries.

A practical rule:

• if a failure already exists for a step (durably), treat subsequent retries as read-only with respect to side effects (they should observe, not re-emit).

Observability: make “what happened?” cheap to answer

Step-level records enable:

• a clear audit trail,
• easier incident review (“which step failed, and when?”),
• metrics like failure rates by step, attempt duration, retry counts,
• and “resume success” instrumentation.

At minimum, you can derive:

• attempts started / completed,
• steps executed / skipped,
• step failure reasons (normalized),
• time spent per step.

Implementation notes (kept intentionally generic)

To keep this post broadly useful (and safe for external publication), we’re avoiding:

• code snippets tied to our exact schema,
• vendor-specific provider details,
• infrastructure topology,
• internal naming and queues.

But the transferable implementation outline looks like:

• A durable store for attempts and step outcomes (often relational DB).
• An orchestrator that:
  • claims attempts,
  • checks step state,
  • runs steps conditionally,
  • persists outcomes idempotently.
• Uniqueness constraints backing “exactly-once side effects.”
• Structured logging + metrics around attempts and steps.

Common pitfalls (and how to avoid them)

1. Make retries read-first
   • Always check durable step state before executing work.
2. Persist failures, not just successes
   • Otherwise, retries can re-trigger the same failure side effects.
3. Deduplicate every side effect
   • If a step triggers an external action, it needs a dedupe key and/or durable guard.
4. Restore minimal context when skipping
   • Skipping must preserve deterministic downstream behavior (“skip drift” is real).
5. Avoid turning step state into a data lake
   • Store the smallest metadata needed for determinism, debugging, and fairness.

Key takeaways for engineers

• Assume at-least-once. Build for retries and duplicates from day one.
• Make each step a checkpoint. Durable step state is the simplest path to resumability.
• Use the database for invariants. Uniqueness constraints + idempotent writes beat timing-dependent logic.
• Skip safely. If you skip work, restore context so downstream behavior stays consistent.
• Design for debugging. Step-level audit trails reduce incident time-to-understand dramatically.

How to apply this pattern to your own workflow (checklist)

[ ] Identify the workflow’s side effects (writes, messages, notifications) and how to dedupe them

[ ] Define durable attempt + step concepts with a minimal state model

[ ] Make every retry read-before-write

[ ] Add DB-level invariants (uniqueness constraints) wherever possible

[ ] Record failures and normalize error codes for debugging

[ ] Instrument metrics for time-to-complete, retries, skips, and resume success

Closing

Idempotency isn’t a nice-to-have in verification workflows. It’s foundational. When your pipeline can safely retry, resume, and run under concurrency, reliability becomes a property of the system rather than a heroic on-call effort.

The best part is that these patterns are not KYC-specific. Any multi-step workflow with side effects, such as onboarding, risk checks, provisioning, or payments, benefits from the same “durable step state + DB guarantees” approach.