The Day 8.5 Million Computers Died

Zach Hughes
4 min readJul 26, 2024

--

I awoke on Friday, July 19th to a phone full of missed calls and texts. Rubbing the sleep from my eyes, I went to my work computer, and jiggled my mouse to wake it up, only to see the blue screen of death.

I returned the calls and quickly got up to speed on the technical carnage impacting the planet.

Some of you never heard of CrowdStrike before last Friday, but most of us in enterprise technology know it well.

Some of you never imagined that our entire digital existence could hang in the balance of a bad update from an obscure software company, but I’ve always realized this possibility.

Our digital world is a whole lot more fragile than you think it should be.

I mean, it’s 2024. We should all be a little tired of hearing about “unprecedented” events, by now, right? It’s almost like “unprecedented” is the new precedent.

“If you do not expect the unexpected, you will not find it; for it is hard to be sought out and difficult” — Heraclitus.

So, yeah. I’ve spent a lot of hours on this, but not as many as some. Plenty has already been written about this event, so what unique perspective can I offer on this topic? What leadership lesson is there in this mess?

My day job as a technology executive is spent in strategy and planning. How useful are strategies and plans when most of your computers get stuck in a blue screen loop that you never saw coming?

Sure, we have disaster recovery plans and we test our backups, like many companies do, but who on Earth planned for this exact scenario? No one. I guarantee it. No one.

No one pulled out the old plan they practiced that read, “Read this in case CrowdStrike blue screens most of our servers and workstations.”

Do you know how to recover from a terrible event like this or anything else you can or can’t imagine? You can’t plan your way out of it. You can’t build enough contingencies to make it so nothing will ever happen. You can’t implement enough testing to prevent it. Occasionally, bad things will happen that you didn’t see coming and you just have to respond.

I can’t be more proud of how the team at CHS handled this event. To say they rose to the occasion doesn’t begin to explain it. I’m really having a hard time putting it into words because if you hadn’t experienced it, you really couldn’t know.

But if you know, you know.

Even if I can’t really explain what it was like, I do have a pretty good idea of why it was possible, and herein lies the leadership lesson.

Purpose

We create connections to empower agriculture. Those connections went down. It’s not just important for the techies to do their job and get the computers working. The whole purpose of the company is to create connections to empower agriculture. Agriculture stops unless we act. No one needed to be reminded of what was at stake. Everyone knows why we are here.

The decision to act wasn’t a decision at all. It didn’t matter if you were on vacation, in a foreign country, or in the wilderness with a bad cell connection. We create connections to empower agriculture. Everyone here knows their purpose and they care about it deeply.

Collaboration at the speed of trust

I witnessed a lot of capable leadership in action. There wasn’t just one leader calling all of the shots and everyone else doing what they were told. We divided and conquered, then collaborated and synchronized frequently. This obstacle was too complex and too high-stakes for single-threaded decision-making.

We progressed. We made judgment calls. We communicated. We adjusted and course-corrected. This is only possible in a highly collaborative high-trust environment. That cannot be conjured at the moment. It must exist in the culture ahead of time.

Brilliance under pressure

We all rely on tech partners, but when things get really crazy, I’m glad I’m surrounded by extremely smart team members who can think on their feet. I witnessed genius-level problem-solving. I work with engineers who looked at a solution that was “good enough” and then worked to make it more automated, scalable, faster, and more reliable.

I watched my sharp colleagues develop intricate support processes out of thin air, and evolve them to become more effective and efficient.

That’s my list. Perhaps it’s not comprehensive. I’m sure I could go on, but you get the gist. Sometimes in our line of work, the unexpected happens and puts us flat on our backs.

Some teams rise to the occasion and some don’t. What makes the difference is what you do as a leader in the normal steady times. Lead with purpose. Create a culture of collaboration and trust. Hire and retain wicked-smart people. That’s the formula.

I’m not looking forward to the next CrowdStrike-type event. Whatever it is, it’ll probably be like nothing we’ve seen before. That’s okay. We’ll be ready because that’s who we are.

Read this article on my blog site or listen to it on my podcast🎙️

--

--

Zach Hughes
Zach Hughes

Written by Zach Hughes

Technology Leader at CHS. Passionate about leadership and innovation. Posts are my own.

No responses yet