The crawler that failed real payments
Some payments were ending in SYSTEM_ERROR for no reason we could see. The culprit was a text-to-speech feature loading our payment page on its own servers.
For a while, a small number of payments on our hosted checkout ended in SYSTEM_ERROR. Nothing in the frontend had changed. The backend looked healthy. The customers had done nothing unusual. The errors weren’t frequent enough to be an outage, but they were real payments failing, so we went looking.
What the logs showed
A payment page like ours makes a session call when it loads. That call sets up the transaction: what the customer is paying, which methods are allowed, what the page should look like. Then, when the customer presses pay, a second call submits the payment against that session.
In the failing cases, the logs showed something odd. After the customer had pressed pay, another session call arrived for the same transaction. It came from a different user agent and a different IP address. The backend treated it as the page being loaded again mid-payment, which put the transaction into a state it couldn’t recover from.
So who was loading our page a second time?
The answer
Some browsers and assistants offer a feature that reads a page aloud. Google’s version fetches the page on its own servers, renders it, and turns the text into speech. To render our page, it had to load it, and loading it meant making the session call. From our backend’s point of view, a second visitor had opened the same payment link in the middle of a payment.
Nobody on the team had imagined that a reading feature could fail a payment. It wasn’t a bug in our code in the usual sense. It was a gap between how we assumed the page would be used and how the real world actually uses web pages.
What I took from it
- The client is not always a person. Crawlers, previews, accessibility tools, link unfurlers and prefetchers all load pages. Anything with side effects on load deserves a second look.
- Side effects on GET are a trap. A page load should be safe to repeat. If loading a page can change the state of money, the system needs to tell the difference between “the customer opened it” and “something fetched it”.
- Read the boring fields. The user agent and IP were sitting in the logs from the start. The clue was there. We only had to stop assuming every request came from the customer.
It’s the bug I think about most, because there was nothing clever about the fix. The lesson was humility: there are factors you will never see in testing that still shape how your product behaves in production.