Last year one of my clients took a hit I'll never forget.
I was processing analytical data for them. About 8 to 12 million records daily. The data itself was fine — there was just an enormous amount of it. The client wanted us to back-calculate values that, in my view, shouldn't have been back-calculated in the first place. I told them six separate times that we shouldn't be doing this at all. Each time they confirmed: do it anyway. So I delivered exactly what they asked for.
The one clue I had: I needed to spin up 70 machines at once, all calculating for several hours. If this had been running on our own database, fixed pricing, no problem. But this was Google Cloud. Resources scaled up automatically. Everything kept responding fine. No slowdown, no warning, no alert.
I found out something was wrong when a payment reminder came through — the company card got declined. I ignored it. Not my department.
Three days later, the actual invoice arrived. They normally pay about $40 a month. This one was $57,000.
Complete shock. I went cold — not because I'd stepped outside the brief, but because I'd never seen a cloud bill behave like that. Up until then the client was paying around $40 a month. We had no reference point for what a full backfill of this scale would cost — the only way to find out was to run it. The obvious move, in hindsight, was to run a single week first and watch the meter before committing to the whole backfill. Even if a week burned $4,000, that's a number you can live with. Nobody suggested it — not me, not anyone on the client side. The decision was just: do it. So we did it.
The client happened to be on vacation by the sea. I called him, explained the situation, and said I'd file a dispute with Google. He took it better than I expected. Maybe the sun helped.
The dispute took 6 months. Google eventually compensated about $13,000. The rest is still being worked out.
No one tells you about these moments. You hear about the wins, the launches, the milestones. Not the phone call where you tell a client their cloud bill just exploded. Not the 6 months of waiting, checking email, hoping Google says yes.
It happened because cloud pricing is a black box. No usage cap, no notification threshold, nothing. The system just scales and bills. By the time you notice, the damage is done.
I set billing alerts on every cloud project since. Hard caps where possible. But honestly, the first spike alone was around $24,000 and Google's billing data is so delayed that no alert would have caught it in time. By the time the dashboard updates, the money is already gone.
But the real lesson isn't technical. It's that these things happen — not because someone was careless, not because the brief was wrong, but because cloud systems are designed to be invisible until they explode. If you've never seen that explosion before, no amount of warnings prepares you for the size of it.
