Recording a payment twice: idempotency for money
Why a payment gets recorded twice when nobody double-tapped, what makes a good idempotency key, why check-then-insert is a race, what a retry should look like to the operator, why a reversal is a real ledger event, and whether a duplicate should be blocked, flagged or allowed.
A customer hands over ₹4,860. The cashier taps record. The connection is fine on their side and the response is lost on the way back, so the screen does not confirm. The cashier taps again, because the correct response to an unconfirmed money operation is to try again, not to walk away. Two records, one payment, one customer who now has two receipts and one of them is wrong.
This is the most common integrity failure in money software and it is almost never a bug. The system did exactly what it was asked to do, twice, because the same logical request arrived twice and nothing in the system could tell the second one from the first. Our piece on concurrency covers the general engineering shape; this one is narrower and about a single question: when does recording a payment twice stop being a mistake a person made and become a property of the system, and what should the person see at the counter when it happens?
Four ways the same money gets recorded twice
Worth separating, because they are different problems wearing one symptom, and the fix that handles one of them does nothing for the others. The human-level cause and the system-level fix are not the same question at all: telling a cashier not to double-tap addresses a human behaviour, while making the second tap harmless addresses the system. The second is the one that works when the cashier is tired, the queue is long, and the network is bad — which is exactly when it matters.
Four causes, and what actually fixes each
A double-tap at the counter
The user did nothing wrong; the response was lost. A key minted on the device and reused on the retry is the fix. Training is not a fix, because the behaviour being punished is the correct response to an ambiguous outcome.
A retry from a flaky connection
The client library resends because it did not get a response. This is the case the pattern was designed for: the retry carries the same key, the server recognises it, and it returns the original receipt instead of performing the work again.
Two people closing the same shift
Two supervisors, two terminals, no shared screen. A key does not help here, because these are two genuinely distinct operations that should not both exist. The fix is that the second close is refused or recorded as a separate event with a reason — never an overwrite of the first.
An import or a sync re-run
A bank statement is imported twice, or a queued batch replays. This is a different mechanism again: the key material has to come from the source — the bank's own reference, the value date and the amount — because no device was involved in the first submission and there is no client to mint one.
Why the human fix and the system fix are different problems
The instinct is to blame the operator, and the instinct is usually half right and entirely unhelpful. The operator's action was reasonable given the information they had: a screen that did not confirm, and a rule, correctly learned, that money operations get retried rather than abandoned. Blaming them produces a sign on the wall and a system that is one bad network day away from doing it again — and the duplicate is discovered later, at reconciliation, by someone who was not there and cannot reconstruct what happened.
The system-level fix has to hold even when the person is wrong, because people are sometimes wrong, and the two protections are not alternatives. Blocking the retry at the database level means a genuinely duplicated key can never create a second payment. Telling the operator that their tap was recognised as a repeat means the human who did double-tap learns it immediately instead of a month later from a variance report. Refusing a stale close means two shifts cannot both become the final word on a day's takings. A system that only does the first is safe and irritating; a system that only does the second is friendly and eventually wrong.
What makes a good idempotency key
The key is the system's only evidence that two requests are the same request, so its construction carries all the weight. It should be unique per logical operation, minted by the party that knows a retry is happening, and stable across every attempt of that operation. That means the terminal or branch, a client-side counter that only moves forward, and nothing that a legitimate second sale could reproduce.
Key construction compared (engineering reference — no measured outcomes)
| Candidate key | What happens | Verdict |
|---|---|---|
| Terminal + amount + method + timestamp to the second | Two genuine ₹500 cash sales in the same second on the same terminal collide, and the second one is silently refused | Wrong. It guesses, and it guesses in the direction of losing a real payment |
| Amount + counterparty + date | Every split payment becomes a duplicate. This is a duplicate-detection heuristic wearing a key's clothes | Wrong. It rejects legitimate repeat business — which is the failure mode that teaches staff to find workarounds |
| Terminal + device-side monotonic counter + operation type | A retry of one operation reuses the counter; the next sale moves it. Stable, unique, and knowable only by the device | The shape we would use. The device is the only party that can distinguish its own retry from a new sale |
| The bank's own reference (UTR) plus value date plus amount | Correct for imports, because the reference is issued by the source and is stable across re-runs of the same file | Right for imports. Useless at a counter, where the customer usually sends nothing |
| A server-generated id returned to the client | The client has no way to resend the same id after losing the response — the response is what carried it | Wrong for this purpose. It guarantees the failure it was meant to prevent |
Why checking whether it exists and then inserting is a race
The obvious implementation is a read followed by a write: look up the key, and if it is not there, create the payment. The problem is that the two requests are not queued behind each other. Both arrive within the same few hundred milliseconds, both execute their lookup before either has committed its insert, both see nothing, and both insert. The table ends up with two rows carrying the same key — which means the key was never actually enforced, only checked, and the check has a window in it.
There is no way to close that window with clever ordering inside the application, because the interleaving happens across two connections. The fix has to be the database's: a unique constraint on the key, and an insert that is expected to fail. The second insert does not silently overwrite anything — it is refused, and the handler reads the row the first request created and returns that original result, receipt number and all. The counter sees the same successful outcome they would have seen the first time, which is the whole point: a retry must look like success, not like an error the cashier now has to interpret at 14:03 with a queue behind them.
What happens on retry: three cases, not one
The three server responses to a repeated key
The key is already committed
Return the original result — the same payment id and the same receipt — and create nothing. The operator's second tap produces the screen they would have got from the first, which is what makes retrying safe enough to teach.
The key is still in flight
The first request has been accepted but not finished. Refuse the second cleanly rather than letting it proceed in parallel, and let the client retry once more; by then the original has committed and the first case applies.
The key is new
Do the work. This is the only branch that creates a payment, and it is the only branch that should be able to.
A reversal is a ledger event, not a silent edit
Idempotency prevents a duplicate from being created by a retry. It does nothing about a duplicate that was created by two genuinely different people, two shifts, or a re-run import that arrived without a usable reference. For those, the system needs a second mechanism, and this is where the reversal principle earns its place: cancelling a recorded money event is itself a recorded money event.
The distinction has a practical consequence for the counter. If the only way to undo a duplicate is to delete it, then the person who discovers the duplicate at 21:05 during shift close has to choose between leaving a known-wrong figure in the day's takings and destroying the evidence of what happened. Neither is acceptable, so the operation gets deferred, and deferred corrections are how a day's cash stops tying out. If instead the duplicate is cancelled by an entry that names the original, records who cancelled it and why, and leaves both visible, the correction takes seconds and the day's figure becomes true. The reversal has to be a first-class event in the same records as the payment, with its own id, its own author, and a reason that is not a free-text afterthought.
Blocked, flagged, or accepted with a warning — and what each costs
When a duplicate is detected there are three honest responses, and the right choice is not the same for every operation. A customer paying at the counter is not the same decision as a shift closing at nine at night, and treating them identically is how you end up with a system that is either unusable at closing time or unsafe at the counter.
The three responses to a detected duplicate (structural comparison — no measured outcomes)
| Response | What the operator sees | What it protects | What it costs |
|---|---|---|---|
| Blocked | A clear refusal naming the original payment, with its id and timestamp, and an option to view it | The strongest guarantee: no duplicate record can exist. Right for a customer payment, where a second receipt is a second document in the customer's hands | A dead stop at the worst moment if the detection is wrong. It pushes workarounds outward, and a workaround that bypasses the block is far worse than the duplicate |
| Flagged for approval | The payment is recorded and held out of the day's figures until a named person approves or rejects it | Records the money, delays the conclusion, and puts a human on the decision. Right for anything with a queue behind it | Nothing is settled until somebody looks. If nobody is accountable for the queue, the flagged item simply becomes the item nobody opens |
| Accepted with a warning | The payment goes through, a warning is displayed and logged, and the record carries the reason it was allowed | Keeps the business open when stopping would cost more than the risk — a supervisor re-closing a shift, a correcting entry after a mis-keyed amount | Accepts a record that may be wrong, on purpose, in writing. Only defensible where someone with authority is present and the operation is reversible by a real reversal |
Worked sequence: what each actor sees
The following is constructed to show the state each party holds at each moment, because most idempotency bugs are state bugs — somebody is holding a fact the system has already moved past. The counter, the device, the server and the manager are separate actors, and only two of them can see the key. All figures are invented for illustration.
14:02:09 — counter device. A sale of ₹4,860 is rung up. The device mints key T4-000871 from the terminal id and a counter that only moves forward. The key is held in a pending set that is not cleared until a success is confirmed.
14:02:10 — request. POST payment {key: T4-000871, amount: 4860, method: upi} is sent. The server inserts the payment under a unique constraint on the key and returns P-4412 with a receipt number.
14:02:10 — response lost. The reply does not reach the device. Nothing is wrong with the money: exactly ₹4,860 was received, and exactly one payment row exists.
14:02:14 — counter device. The cashier taps record again. The key is still in the pending set, so the same T4-000871 goes out. No second key is minted, because the device cannot tell a new sale from a retry until it has confirmation.
14:02:14 — server. The insert for T4-000871 is refused by the unique constraint. The handler reads the row created at 14:02:10 and returns P-4412 and the same receipt number. No second payment exists.
14:02:14 — counter. The screen shows the sale settled, with receipt P-4412. The cashier has no idea a retry happened, which is the correct experience.
14:02:16 — device. The success is confirmed, so T4-000871 leaves the pending set. Had it been cleared on a merely-sent request, the second tap would have minted a fresh key and created a genuine second payment — so the clearing rule is part of the design, not a detail.
21:04 — supervisor A. Shift 3 is closed. Totals, variance and the closing figure are written, attributed to A.
21:06 — supervisor B. B closes the same shift from another terminal. This is a second logical operation, so no key protects it. The correct behaviour is a refusal pointing at A's close, or a separate event recorded as B's own — never an overwrite of the day's figures.
21:08 — supervisor A. Notices the second attempt, cancels it with a reversal entry naming both closes, the reason, and their own authorisation. Both closes stay visible.
09:14 next day — re-run of yesterday's statement import. The same file is submitted again. If the bank reference on each credit is available, the reference plus value date plus amount forms the key and every row is recognised as already loaded. If any credit carries no usable reference, that row cannot be safely re-run and belongs in an exception list for a person — not in an automatic insert.
Two failure modes are worth naming from that sequence, because they are the ones that survive into production. The first is the pending set: if a key is cleared on a request that was merely sent rather than on a confirmed success, the protection disappears exactly when the network is worst. The second is the shift close, which a key cannot help with at all, and which is why the reversal path has to exist — the operations a key cannot cover are precisely the ones that need a correction mechanism rather than a prevention mechanism.
Duplicate detection is not idempotency, and confusing them is expensive
A system that flags two payments of the same amount to the same client on the same day as suspicious is doing something genuinely useful — and something quite different. Detection is a heuristic with false positives: small businesses take repeat payments routinely, a client settling two invoices in equal parts on one day is ordinary, and a rule that treats the second as a duplicate will train staff to record through it. Idempotency is not a heuristic at all. It is a fact carried on the request, and it is exact: the same key is the same operation, always, with no judgement involved.
The practical rule is that prevention should be exact and detection should be suspicious. Where an exact mechanism exists — a key from the device, a bank reference from the source — it should decide, and heuristics should be left to handle the residue: the credit that arrived with no reference, the payment re-keyed from a slip two days after the fact by someone who did not see the first entry, the closing adjustment that turns out to resemble yesterday's. That residue is real and cannot be engineered away, and the honest treatment for it is a named person and a queue, not a rule that guesses.
When the two payments are real: what an overpayment is as a fact
Retrying the tap and paying twice produce the same shape in the ledger, and they are different events. The distinction decides everything that follows, so it is worth being precise about what an overpayment actually is: not a verdict on the customer, and not an error state. It is an amount recorded against an invoice that exceeds the amount that invoice asked for. The customer's ledger and yours now disagree about a balance, and until something is written down, the difference is only visible in the two systems at once — the customer sees a credit, you see a surplus, and neither number is wrong.
Two money movements leaving one account is a real second payment, and it has to be treated as real money, not as a bug to be tidied away. That is the part most billing workflows get wrong: the instinct is to net the two entries down and leave one record, which makes the ledger agree with itself and the bank statement disagree with the ledger. A surplus that has been quietly absorbed is a surplus nobody can explain at quarter end.
The data-entry accident and the real second payment
There is one reliable discriminator, and it is not inside your system: whether the money actually left the customer's account twice. A bank statement, a UPI or card provider reference, or a settlement file settles it in a way that no internal heuristic ever can. Two entries carrying two different provider references are two payments. Two entries carrying the same provider reference are one payment recorded twice. This is the same conclusion the duplicate-detection section above reaches from the other direction, and it is worth stating twice because the failure mode is expensive in both directions: resolving a real second payment as a duplicate loses the customer's money, and resolving a duplicate as a real second payment invents a credit the customer never asked for.
Where that evidence does not exist — a cash payment described over the phone, a receipt keyed from a slip the next morning, a statement line with no reference at all — no system can decide this, and any product that appears to is guessing. The honest design is to flag the pair, attach what is known about each, and put a person on it. That is a slower answer than an automatic one, and it is the correct one.
Where this stops, honestly
Three boundaries. First, idempotency protects the write path and nothing else: it will not tell you that yesterday's takings were wrong, will not reconcile a payment against a bank statement, will not detect a duplicate created by two different people with two different keys, and it cannot tell you whether two payments were real. Second, a key is only as good as the party minting it, and there are situations where no party has one — a payment described over the phone, a cash receipt keyed from a slip tomorrow morning, a statement line with no reference. Those need detection and a person, and pretending otherwise is how a system ends up claiming a guarantee it cannot make. Third, none of this is financial control. Recording a payment correctly twice-over is not the same as verifying the money, and the reconciliation of what the counter recorded against what the provider or bank settled is a separate routine with its own discipline.
What we have not measured
We have no measured duplicate-payment rate, no measured rate of retries at a counter, no measured false-positive rate on any duplicate heuristic, and no measured number of reversals per thousand payments. Every figure in the worked sequence is arithmetic we constructed to make the state transitions legible, and the counter, the amounts and the timings are invented. We also do not claim that a well-designed key removes the problem: it removes the specific case where the same logical request arrives twice, and the instrumentation that would tell us how much of the residue remains — duplicate-key hits, reversals by reason, the count of flagged items a person actually opened — is the list we would want to be measuring before using the word protected about anything.
Frequently asked questions
Why does a payment get recorded twice if the cashier only taps once?
Usually because the response was lost, not because the tap was repeated. The device cannot tell a lost response from a failed request, so the honest operator behaviour is to retry — and if the server treats the retry as a new operation, the same money exists twice. The cashier did nothing wrong; the request was ambiguous. Telling people not to double-tap addresses a human behaviour and leaves the system unchanged for the next bad network day.
What is an idempotency key and who should create it?
A value carried on the request that identifies one logical operation, unique per operation and stable across every retry of it. It should be minted by the party that knows a retry is happening — normally the device, from a terminal id and a forward-only counter — because the server cannot tell a client's retry from a genuinely new sale. It should not be minted from the amount, the date and the counterparty, since two legitimate payments in that shape are ordinary.
Why is checking whether a payment exists and then inserting it unsafe?
Because the lookup and the insert are not atomic. Two retries arriving together can both complete their lookup before either commits its insert, so both see nothing and both insert, and the key is only ever checked rather than enforced. The protection has to come from the database: a unique constraint on the key, and an insert that is expected to fail, with the failure handler returning the original payment's result so the retry still looks like success to the operator.
Should a duplicate payment be blocked, flagged for approval, or just allowed with a warning?
It depends on the operation, and treating all three the same is the mistake. Block a customer payment that repeats a key, and return the original receipt rather than an error the cashier has to interpret mid-queue. Flag a second attempt to close a shift or re-import a statement, with a named owner for the queue. Allow an override only for someone with authority, only with a reason recorded, and only where a real reversal exists to undo it. On a dedicated Nox-Billings deployment, shift close and day-end reconciliation are set up and validated with you as assisted setup rather than handed over as a switch.
If a duplicate already exists, is deleting it acceptable?
No. A cancellation should itself be a recorded event that names the original, carries the author and the reason, and leaves both entries visible — the same principle as a storno, where the original and the reversal both stay in the books. Deleting the duplicate also deletes the only evidence of what happened, which means the day's takings stay wrong until somebody reconstructs the sequence from memory.
Does an idempotency key catch a payment recorded twice by two different people?
No, and no key can. Two supervisors, two terminals and two genuine intentions produce two keys, and prevention has to come from elsewhere: a refusal on a second close of the same period, a uniqueness constraint on the source's own reference for an import, and a correction path for everything else. Duplicate detection on amount and date is useful as a flag for a person, but it is a heuristic with false positives, and a system that leans on it to prevent duplicates will eventually refuse a real payment.