30 payment failure modes

The places where payment systems most often lose, duplicate or lose track of money. Each one is a real failure I have seen in production.

Tick an item only if you could show evidence for it today: a test, a query, a dashboard or a runbook. Anything unticked is worth a closer look.

0 of 30 covered

Idempotency

The same request, sent twice, must move money once.

1Every money-moving endpoint takes an idempotency keyReusing a key with a different payload is rejected, not silently processed.
2The key and its result are saved in the same transaction as the money movementA crash between the two cannot leave a charge without a stored result.
3Calls to your PSP or bank carry a stable idempotency keyYour own retries reuse the same key instead of generating a new one.
4Keys are kept longer than your longest retry windowIncluding queue redelivery, dead-letter replays and manual re-runs.
5Two concurrent requests with one key cannot both executeOne wins; the other waits for, or receives, the same result.

Retries and timeouts

A timeout means "I don't know", never "it failed".

6A timeout puts the payment in an unknown stateYou query the provider for the real status before retrying or failing it.
7Retries use backoff, jitter and a hard capA PSP outage does not turn into a retry storm when it recovers.
8Double clicks and resubmits cannot create a second paymentThe client sends the same key for the same checkout attempt.
9Payments stuck in pending are detectedAnything past its expected time raises an alert with an owner.

Webhooks and events

Assume every notification arrives late, twice, out of order, or not at all.

10Webhook signatures and timestamps are verifiedReplayed or forged notifications are rejected.
11Duplicate webhooks are deduplicated by the provider's event IDProcessing the same event twice changes nothing.
12Out-of-order events cannot move a payment backwardsAn "authorized" event arriving after "captured" is ignored, not applied.
13Webhooks are acknowledged only after state is durably savedOr written to an inbox table first and processed from there.
14Missed webhooks are recoveredA scheduled job polls or backfills status for anything unconfirmed.

Ledger

If it is not in the ledger, it did not happen.

15Every movement is double-entry and balances to zeroDebits equal credits for each transaction, enforced in code or the database.
16Ledger entries are append-onlyMistakes are corrected with reversing entries, never by updating or deleting rows.
17Balances can be rebuilt from entriesA regular job recomputes them and alerts on any difference from stored balances.
18Each entry links to its external referencePSP payment ID, bank reference or chain transaction hash.
19Fees, FX spread and chargebacks are booked separatelyNot netted silently into the settled amount.

Reconciliation

Your numbers are only right once someone else's numbers agree.

20Every PSP and bank account is reconciled dailyAgainst settlement files or statements, not just your own API responses.
21Unmatched items have an owner, an age and an alertNothing sits in "unreconciled" for weeks without someone noticing.
22Refunds, chargebacks, reversals and fees are reconciled tooNot only successful captures.
23Importing a settlement file twice does not double-count itFile ingestion is itself idempotent.

Concurrency and money math

Small rounding errors and race conditions add up at volume.

24Amounts are never stored or calculated as floatsIntegers in minor units, or fixed-precision decimals end to end.
25Decimal places come from the currency, not a hardcoded 2JPY has 0, KWD has 3, and crypto assets have up to 18.
26Balance check and debit happen atomicallyRow lock, conditional update or serializable transaction, so two debits cannot overdraw.
27The FX rate and rounding rule are stored with each conversionSo any amount can be recalculated and explained later.

Operations and audit

When something goes wrong, you can see it and explain it.

28Every payment state change has an audit trailWhat changed, when, and which user or service changed it.
29Manual adjustments need a second approverAnd go through the ledger like any other movement.
30You can say where any customer's money is in under five minutesWithout asking an engineer to write a query.

More than five unticked?

That is normal for a growing payment platform, and fixable. A Payment Integrity Audit traces these issues in your real code and data, with a fix for each one. Fixed price from €3,500, done in two weeks.

Book a 20-minute call

serhii@serhard.com serhard.com