← BACK

Case Study

Idempotent Payment Orchestrator

Backend Architecture • Transactional Systems

Images

Gallery unavailable.

Overview

Designed and implemented a retry-safe payment processing pipeline with strict idempotency guarantees under concurrent transaction submissions. Idempotency keys at request boundary; outbox for provider calls; webhook reconciliation by event_id. Guarantee: no double charge under client retries, duplicate webhooks, or network failure between DB commit and provider call.

Client
   ↓
API (Django)
   ↓
PostgreSQL (transactions + idempotency)
   ↓
Outbox Table
   ↓
Worker (Celery)
   ↓
Payment Provider

Architecture & Design

Transactional Flow Overview

Internal System
Internal System

System Invariants

  • ·A payment intent cannot transition from failed to succeeded.
  • ·Reservation "paid" is set only after a verified webhook; frontend and redirect cannot set it.
  • ·Webhook events are processed idempotently by provider event_id.
  • ·Availability for a slot is updated under pessimistic lock (SELECT FOR UPDATE); no optimistic commit.
  • ·Idempotency keys are scoped per client and stored; duplicate key returns original response.
  • ·Payment and reservation state changes for a webhook occur in a single database transaction.

Architecture Decision Records

  • ADR-01Idempotency key required for all payment initiation requests
  • ADR-02Outbox for provider calls; no side effects inside request lifecycle
  • ADR-03Webhook processing idempotent by provider event_id
  • ADR-04Transaction boundaries: single DB transaction per state transition
  • ADR-05Reconciliation and failure mode handling

Scale & Constraints

Request volume
Client and provider retries; requests and webhooks can arrive duplicated or out of order.
Concurrency
Single writer per idempotency key; outbox for provider calls. No double charge under retries.
External dependencies
Payment provider API; webhooks. Network failures between DB commit and provider call possible.
Failure modes
Provider timeout or unreachable after commit → outbox retry. Duplicate webhook → idempotent by event_id. Client retry → same key returns stored outcome.
Data consistency
Payment state and outbox in same DB; commit before provider call or outbox. No double charge; idempotency key is sole source of outcome for request.

What was explicitly rejected

  • ✕

    Simple request-based processing without idempotency keys

    Retries and duplicate submissions would cause double charge; key at business layer is required.

  • ✕

    Handling retries only at HTTP layer

    Application state can still double-apply; idempotency must be enforced at orchestration layer with a stable key.

  • ✕

    Relying entirely on provider guarantees

    Provider semantics vary and may not guarantee exactly-once; we own the no-double-charge guarantee.

  • ✕

    Processing side effects inside request lifecycle

    If process dies after DB commit but before provider call, state is inconsistent; outbox decouples and allows retry without re-executing request.