Every major high-throughput database now runs as microservices, not sure why people still act like things just grind to a halt when the network is involved.
Since when were payment networks latency sensitive? It’s usually 2 or more seconds to even get a payment up on the card terminal from the merchant POST system, then 2-5 seconds more from card presentation to getting approval back.
It's not simple though. In that 140ms the network is checking fraud rules, validating the card, checking available credit, applying rewards logic, and routing across multiple parties. The actual subtract-one-number-from-another takes microseconds. The rest is trust verification across organizational boundaries — which is the hard part of any payment system.
This is cool, but a part of me has to be flip, and humanity is kind of hilarious to me. All of this amazing infrastructure and hard work to essentially just very carefully and accurately subtract one number from another in a way that makes auditing easy.
"Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling."
Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring that all those things are correctly unwound in the case of refunds being issued.
As someone who recently got an Amex card (primarily for Air Canada lounge access), I've been impressed at how nice their app is compared to the five previous Canadian bank apps I've been exposed to in recent years (Scotia, BMO, RBC, Tangerine, CIBC). Some nice things I noticed in the Amex CA app that I haven't previously experienced:
- instant alerts on use, even when it was a non mobile pay transaction
- up to the minute transaction history in-app, including Aeroplan point accrual; all my other credit cards have a delay before new items appear.
- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
Apple Card has the same features, currently issued and operated by Goldman Sachs but to be transferred to Chase within the next two years. Hopefully the features and functionality will survive. I don't think Chase has them today, so I wonder.
It's atrocious how bad most bank and card apps are. I'm planning a switch to a new bank, and mobile app quality is a huge criterion. Bank of America and Wells Fargo get zero points from me.
The GTR design is smart — minimal protocol understanding, just enough to route. The temptation in payment systems is always to push more logic to the edge. Keeping the router thin is what made the canary ollback trivial.
The shadow traffic approach is the real insight here. You can't test a payment system with synthetic data — the edge cases are in the real transaction patterns. Replaying production traffic against a parallel system and diffing the results is the only way to know if you've achieved parity.
Their closing lesson — "patience and discipline, success is measured in reliability" — applies to any system where transactions are irreversible.
Oh Jesus Christ.
I’m surprised a network so sensitive to latency (as are payment networks), was able to achieve their latency SLAs with micro services.
Maybe Amex being a closed-loop network helps with latency?
I’ve heard anecdotally that it’s < 140 ms for payment networks.
Anyone, please correct me if you know better.
In practice, the POS sends a message to the acquirer processor -> hits the network -> is sent to the issuer processor, and back again.
https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-...
Hmmm.
Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring that all those things are correctly unwound in the case of refunds being issued.
As someone who recently got an Amex card (primarily for Air Canada lounge access), I've been impressed at how nice their app is compared to the five previous Canadian bank apps I've been exposed to in recent years (Scotia, BMO, RBC, Tangerine, CIBC). Some nice things I noticed in the Amex CA app that I haven't previously experienced:
- instant alerts on use, even when it was a non mobile pay transaction
- up to the minute transaction history in-app, including Aeroplan point accrual; all my other credit cards have a delay before new items appear.
- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
It's atrocious how bad most bank and card apps are. I'm planning a switch to a new bank, and mobile app quality is a huge criterion. Bank of America and Wells Fargo get zero points from me.
The shadow traffic approach is the real insight here. You can't test a payment system with synthetic data — the edge cases are in the real transaction patterns. Replaying production traffic against a parallel system and diffing the results is the only way to know if you've achieved parity.
Their closing lesson — "patience and discipline, success is measured in reliability" — applies to any system where transactions are irreversible.