Designing FinTech for the Day Everything Fails: Cyberattacks, Outages, Dependencies, and Recovery

A payment provider accepts a transaction, but its response never reaches your backend. The customer sees a timeout. An operator considers switching traffic to another provider. That recovery decision can create a duplicate charge. FinTech resilience requires more than restoring services. The architecture must preserve financial correctness while transaction outcomes remain uncertain. The Problem A local database transaction does not atomically include a payment provider’s processing decision. When a request times out, your backend knows it stopped waiting. It does not necessarily know whether money moved. Treating that uncertainty as failure can encourage another attempt against an obligation that was already collected. An explicit state such as OUTCOME_UNKNOWN gives the system something actionable to manage: the original provider reference, the next inquiry, the evidence received, and the threshold for manual review. Where It Gets Difficult 1. Atomic acceptance does not guarantee external execution Recording a payment intent and its outbox event in one database transaction closes a local consistency gap. However, a worker can submit the payment and crash before recording delivery. Processing the outbox again may repeat the external request. Stable provider identifiers, provider-specific idempotency behavior, and outcome resolution still matter. A unique request key also needs to work alongside an enforced business-operation identity. 2. Provider failover has two separate responsibilities Routing eligible new payments to another provider differs from resolving payments already submitted. Independent providers generally do not share an idempotency namespace. An unresolved payment should remain associated with its original provider reference until sufficient evidence establishes its outcome. 3. Restoring data does not restore the outside world A database recovery point may precede a payment that succeeded externally. Recreating and resubmitting that missing operation can repeat its financial effect. Recovery therefore needs reconciliation against provider records, authenticated callbacks, settlement evidence, and independently retained transaction evidence. 4. Cyber recovery requires a trusted environment During an infrastructure outage, moving traffic may restore availability. During a compromise, it may also carry stolen credentials or malicious configuration into the recovery environment. Isolation, credential revocation, validated deployment artifacts, and rebuilding from a trusted baseline become part of the recovery decision. Architectural Direction A practical design should: Preserve payment identity across retries and mobile-app restarts. Record accepted work and dispatch instructions atomically. Separate status inquiries from money-moving submissions. Apply controlled state transitions and idempotent ledger-posting controls. Reconcile uncertain operations before reopening capabilities that depend on their correctness. For example, after a provider timeout, the mobile app retrieves the existing payment rather than creating another one. The backend keeps investigating through the original provider reference. When an authenticated inquiry confirms success, the backend applies the permitted transition and records the corresponding ledger entry under a unique business reference. A duplicate callback must not create another posting. The Takeaway Replication supports availability. Backups provide historical recovery options. Reconciliation accounts for financial effects beyond the restored database. Recovery dashboards should therefore track unknown-outcome age, reconciliation breaks, settlement delays, and ledger-posting failures alongside infrastructure metrics. The recovery goal is an explainable financial state, including operations whose answers arrived late. Want the deeper architectural breakdown, implementation considerations, failure scenarios, and full reasoning? Read the full article on Medium