I felt sorry for myself on Thursday because Spirit was four hours late FLL to ORD (impressive considering that they had no mechanical or weather problems). On Friday, however, CrowdStrike managed to disable the entire U.S. airline industry. Can we agree that there should be a new word in English: CrowdStruck, meaning a systemic meltdown caused by a diversity, equity, and inclusion-oriented enterprise? From CrowdStrike’s web site:
It seems fair to say that they achieved their goal of “challenging the status quo” (the status quo being servers that had been up and running for years).
Considering that the U.S. Secret Service was apparently more focused on DEI than on keeping Donald Trump alive, the word could be used in the following sentence: “Donald Trump might need a new ear after being CrowdStruck in Pennsylvania.” (Loosely related… I received the photo below from a deeply closeted Trump-supporting academic.)
Readers: Please share your stories about being CrowdStruck in the comments. How did you experience the meltdown of IT services (except for Elon Musk’s X!).
My own CrowdStruck experience was limited to not being able to check in at the Doubletree here in Milwaukee. They couldn’t make keys for any new guests all day and had to send employees up to open doors for any guest who wanted to get into a room. They finally got their systems back by around 9 pm and will spend the weekend catching up.
Speaking of Milwaukee, here are some of the billboards that the righteous paid for on a highway leading into town:
The Third Ward and some other parts of town that we’ve seen so far are quite pleasant. I can understand why some Chicagoans are considering fleeing here (though I can’t understand why or how they’d stay through the winter!).
Correction: it’s 10 am and they still can’t make keys! “We are waiting on Hilton,” says the front desk manager.
I think the USPTO is living the CrowdStruck Nightmare. Maybe the AC/DC “Thunderstruck” song can be rewritten to include:
I was caught in the middle of a Security track (CrowdStruck)
I look ’round, and I knew there was no turning back (CrowdStruck)
My mind raced and I thought, what can I do? (CrowdStruck)
And I knew there was no help, no help from you (CrowdStruck)
YOU’VE BEEN CROWDSTRUCK!
https://www.uspto.gov/blog/system411/
Current Status
Saturday Jul 20, 2024
Current Events
Based on the updated best estimates, roughly 6,500 employees could be impacted by the CrowdStrike outage.
As of 10 a.m. ET, our IT Service Desk technicians are currently listening to and processing employee voicemails left between 7- 8 a.m. ET on Friday, July 19, 2024.
Due to the nature of the issue, our technicians can’t remote into a user’s workstation and remediate. Therefore, the resolution time per employee is ranging between 25 minutes and 35 minutes with an average of 100 requests for service resolved per hour.
At this time there is no agency-wide solution we can deploy to all computers, and we must continue to resolve employee issues on a 1-to-1 basis with employee interaction.
Surprised so many animals have gotten to a level of accepting mandatory updates with no option to disable them on their own computers. Greenspun bricking his iphone camera by updating it was the clearest warning that you’re eventually going to have to own what you pay for or lose it.
My phone was automatically updated about a month before the camera failure. The fix to the phone was entirely hardware replacement and did not involve any software updates or rollbacks.
However much one may dislike Microsoft, I am not sure why some blame the company for an unrelated company (CrowdStrike) misstep amplified by mindless corporate IT directives to update the glorified Norton antivirus like software without due testing.
I kept hearing Microsoft all over the news, but it wasn’t their fault. A Crowdstrike driver running in kernel mode tried to access a null pointer. There’s no recovery from that. It didn’t take Crowdstrike long to debug their code (apparently not hard to reproduce!) but the fix involves going into safe mode and must be done manually on all affected computers.
I live a zero-Windows life and while pottering about at home, I have only noticed this crisis in the news. This includes some amusing photos of BSODs on sophisticated-looking public screens.
Saw a picture of all the arrival/departure displays at an airport with a BSOD.
Unlike our southerner boarders, those BSODs are coming to the USA legally!
Thankfully I read about CrowdStrike GEFGW update in the morning in deplorable media, so I refused first thing in the morning pushed patch with malicious update (hours after deplorable media reported it) and was not affected personally and could continue working; however other folks were affected and my schedule went into disarray.
At my BIG company, a lot of us were impacted and directed to IT center in India who had no clue what to do. The good news, we have a local “IT” person — she is manages accounts, sever and software updates, and anything has to do with networking — she got us up and going by noon time (getting the Bitlocker key was the hardest part). The bad news, our local “IT” person is retiring in few months! We know we are screwed when it comes to IT issues.
Notice how I put “IT” in quotes, her job is not in IT, but she is our bridge to the IT team in India. She knows more about IT then any IT person in India.
I lost two laptops to Bitlocker. Both Bestbuy/Dell vintage. Computers encrypted themselves, and turns out neither Bestbuy nor Dell had provided the keys to the customer (me!).
It was possible to recover the computer doing a hard system reinstall, but all the data was gone forever. I had the option of using the system again, but chose to dispose of the computers knowing that they had a factory-installed malicious operating system.
My son is a nurse manager at a local hospital here in Maskachu$etts. He got a call at 1 am and was asked to come to the hospital. Everyone at the hospital switched over to using papers to keep track (prescriptions, vital, notes, etc.) My son was responsible to audit and track the paperwork. By 3 pm, the IT department handed out memory sticks to managers, with instructions, to get the systems back online. By late evening, the hospital was up and running again and the crises was declared over.
I worked the nightshift. I new something widespread was going on when several of our computers in the Emergency Department showed blue screens and our radiologist said his home computer was blue.
Only our idle computers updated, so we were able to keep working on the ones we were using, and our radiologist had to stay up all night because we couldn’t send images to the night-reading service.
I found the fix on the web about an hour after it hit us, but I couldn’t delete the update file because I don’t have admin privileges. I called IT and told her what the problem was and the screenshot of the fix, but she just said, “I think it may be that Microsoft problem,” and stayed at home.
On the dayshift IT sent people over to fix each machine individually while I slept at home.