TrenchOps 🐎

Insights from the tech trenches

Hardware

Cover Image

The Cloud Is Full

Nobody planned for the day elasticity ran out

For fifteen years, "the cloud is infinite" was the most reliable lie in enterprise IT. Not a malicious lie. A useful one. It let architects design for peak load without capacity planning, let CFOs treat compute as opex, and let me tell customers "just scale out" with a straight face.

Then customers in North America and the UK started filing tickets that read like they came from 2004. "Cannot provision." "Cannot reschedule." "Try another region." Across all three hyperscalers, in specific regions, at specific instance families. AWS reportedly told its own engineers to conserve compute "however they can" - internal teams waiting days for CPUs, because customer workloads come first. Which is the correct priority and also a sentence that should not exist in a business built on the promise of on-demand.

Ten years ago I would have laughed at anyone predicting this. The whole pitch was that Amazon, Google and Microsoft had so much spare iron that your workload was a rounding error. And they did. They just sold the rounding error to language models.

The number that shouldn't exist

Here is the part that should make every architect uncomfortable. Cast AI's 2026 telemetry across 23,000+ production clusters puts average GPU utilization at about 5%. On AKS, 2%. On EKS, 5%. On GKE, 6%.

Read more
Cover Image

Many Eyes, Zero Readers

A hardware wallet built keys from a random number generator it never once called. For five years. The source code was public the entire time.

In a 25-minute window, someone swept roughly 594 bitcoin - about $38 million - out of some 500 unrelated wallets. No phishing. No malware. No supply chain tampering. No 5$ wrench.

Every one of those wallets was protected by a device sold on a single, admirable promise: keys generated offline, on isolated hardware, using a true hardware random number generator, Bitcoin-only to shrink the attack surface, no cloud, no companion app, and open-source firmware you can verify yourself.

The hardware RNG was on the board. Powered. Functional.

For roughly 1,900 days, the firmware never asked it for a number.

The bug is one character wide

The vendor did something entirely sensible: they switched off MicroPython's built-in random path with a build flag, because they had written their own wrapper around the microcontroller's true RNG. Textbook embedded hygiene.

Then a cryptographic support library checked that flag with #ifndef - which asks "does this macro exist?" rather than "is it set to something other than zero?"

The macro existed. Its value was zero. So the library concluded hardware entropy was available and happily bound itself to MicroPython's random function. MicroPython, having read the same macro correctly, had compiled a non-cryptographic software fallback instead of the hardware peripheral.

Read more
Cover Image

The 6700€ Backup Strategy

(When "It's Cheaper This Way" Is the Most Expensive Lie in IT)

I was sixteen, still an apprentice, fresh out of school, and my biggest concern in life was whether I had enough money left for the new Pokémon cards after lunch.

Then HR called in a panic. One of their desktop PCs wouldn't boot. Classic clicking death rattle from a 10k RPM hard drive that had clearly given up on life. I walked in, confirmed the drive was toast, and casually told the lady we'd just swap the machine and restore her data from the network share.

"There are no network shares." she whispered. "This PC holds every single interview question for the 100 applicants coming in tomorrow."

No backup.
No off-site copy.
No second machine.
Just one lonely desktop in the corner of HR, guarding the future hiring plan of the entire company.
And tomorrow? One hundred candidates were showing up for interviews.

She was hysterical. I was sixteen. So I did what any smart-mouthed apprentice would do: I joked that we could send the dead disk to a professional data recovery lab... but it would be stupidly expensive.

She didn't laugh. She nodded frantically. "YES - do that. Right now!"

Read more