skysight-inc/HackerNews huggingface dataset:
There sure are a lot of mission-critical systems and companies hit by this. I am surprised that auto-updates are enabled. I read about some large companies/services in my country being affected, but also a few which are unaffected. Maybe they have hired a good IT provider.
That's what you get for letting a company install a root kit on your servers and desktops ;-)

I mean, don't they do canary updates on CrowdStrike too? Every Windows admin has done this for the last 5+ years, test Windows updates on a small number of systems to see if they are stable. Why not do the same for 3rd party software?
I blame the Copilot
Yes, software industry as we know would not exists if companies where held liable for all damages. But in the current state of affairs they have little incentive to improve software quality - when incident like this happens they can suffer an insignificant short term valuation loss but unless it happens too often they can continue businesses as usual.

Many companies paying lip service to quality/reliability but internal incentives almost always go against maintenance and quality of service work (and instead reward new projects, features e. t. c.).
July 2024 global cyber outages
Would not shock me for AV companies to immediately work around that if it were to be implemented. “You want our protection all of the time, even if the attacker is corrupting your drivers!”
It's kind of surprising so much infra was using windows servers or windows cloud VMs for these things. I assumed these systems would all be Linux VMS in Azure/AWS/GCP at this point.

on https://azure.status.microsoft/en-gb/status the message is currently:

> We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state.
Which is actually Counter Strike 4
Unleashing 100 Mpps with Fd.io VPP on GCP x86
Difficult on mobile and I'm without access to k PC ATM.

Basically, my predecessor hat build a python program which had N modules which where applied to a data structure - maybe think of correction steps. First spell checking, than turn manual headlines into actual headlines etc.
Originally, they operated on a global data structure, and every new module required calling it on the global structure, so extension was difficult. Worse, every module hat internal state, e.g. the number of spelling mistakes. Reporting these things at the end was cumbersome. There where "required" functions in each module, but it was difficult to newcomers to discover who they where.
We changed it so that every step was a class which inherited from a base class, so adding a new step was as easy as inheriting from the base class. Furthermore, we added Auto discovery, so just adding the class in a module was sufficient to get it executed.
>Linux isn't exactly reliable either...

That's certainly a perspective.
Centralisation in a nutshell. Monopolies so big that they become globally fragile. CloudFlare outages break a lot of the internet, and now we can see, Windows-based updates bricking machines across the world.

We've all pushed bad updates but how was this not tested?
I feel like posting this as if it's a given means you are unaware of history. Just because something exists now doesn't mean it is always determined.
Because it's not a Windows update.
As a non-US person I have the opposite perspective. The US is almost the only place where companies can spring up and just be better than thousands of other companies, and bring me more value. Obviously that brings issues, but fundamentally for me at least I use Amazon because it's the best, even though I wish a local alternative would be better.
Because it's not a Windows update perhaps
This company has post-apocalyptic style photos to make you panic-buy their solution.

https://ibb.co/Bc6n527

"62 minutes could bring your business down"

I guess they could bring all the businesses down much quicker.

https://www.crowdstrike.com/en-us/#teaser-79minutes-adversar...

(Repeating my comment because other story is duped)
Gaia: Milky Way's last major collision was surprisingly recent
I'm actually working on something like this:

https://github.com/project-kiosk/kiosk/tree/v3

I'm still deep in the trenches, though. That project is like my personal zen garden of deadline-free software development, so don't expect a release soon. Happy if someone would be interested in contributing, though :)
> checkbox software

Never heard this name before. Thx.
That’s very different though. That’s avoidable. We all can easily have our services running in different data centers around the world. Heck, the non-amateurs out there all have their services running in different Amazon data centers around the world. So you can get that even from a single provider. Hardware redundancy is just that cheap nowadays.

This CS thing, there’s no way around. You use it and they screw up, you get hit. Period. You don’t failover to another data center in Europe or Asia. You just go down.

Hardware, even cloud hardware, is rarely the issue. Probably especially cloud hardware is not an issue because failover is so inexpensive relative to software.

Software is a different issue entirely. How many of us will develop, shadow run, and maintain a parallel service written on a separate OS? My guess is “not many”. That’s the redundancy we’re talking about to avoid something like this. You’d have to be using a different OS and not using CS anywhere in that new software stack. (Though not using CS wouldn’t be much of a problem if the OS is different but I think you see what I mean.)

Amazon, implementing failover for your hardware is a few clicks. But if you want to implement an identical service with different software, you better have a spare dev team somewhere.
Some of these services go even further. One time, our IT department was being sales-bombed with a service that would remove our actual login credentials to servers, and then "for security" we'd access said servers using a MITM website kind of thing that would be behind our corporate AD-login. I didn't even find out the full intricate details before telling them to "nope this the fuck out" and stay away with a 10-ft pole.

It's like these people have nothing better to do with their time and just absolutely have to have to design and build a product for the sake of it, and then dump it on marketing for > 0 amounts of sales through pretty-much wearing IT departments down. Or in the case of this Crowdstrike thing, through the protection racket known as security audit compliance.
I did approximately this recently, but on a Linux machine on GCP. It sucked far worse than it should have: apparently GCP cannot reliably “stop” a VM in a timely manner. And you can’t detach a boot disk from a VM that isn’t “stopped”, nor can you multi-attach it, nor can you (AFAICT) convince a VM to boot off an alternate disk.

I used to have this crazy idea that fancy cloud vendors had competent management tools. Like maybe I could issue an API call to boot an existing instance from an alternate disk or HTTPS netboot URL. Or to insta-stop a VM and get block-level access to its disk via API, even if I had to pay for the instance while doing this.

And I’m not sure that it’s possible to do this sort of recovery at all without blowing away local SSD. There’s a “preview” feature for this on GCP, which seems to be barely supported, and I bet it adds massive latency to the process. Throwing away one’s local SSD on every single machine in a deployment sounds like a great way to cause potentially catastrophic resource usage when everything starts back up.

Hmm, I wonder if you’re even guaranteed to be able to get your instance back after stopping it.

WTF. Why can’t I have any means to access the boot disk of an instance, in a timely manner? Or any better means to recover an instance?

Is AWS any better?
It gets worse if your machines have bitlocker active, lots of typing required. And it gets even worse if your servers that store the bitlocker keys also have bitlocker active and are also held captive by crowstrike lol
What are you saying? Am I holding it wrong?
That school running their HVAC infra on an Amiga must be pretty happy.
It's a shame. I'd love to visit St. Petersburg, but the window of history where I could do that in relative ease and safety is closed.

Many years ago a friend went to Syria in its brief window of peace, and visited some of the incredibly well preserved Roman sites. I hope they're still OK.
I am not against trying to protect against people who do that. The problem is that you pay XDR big bucks to stop a lot more than that, and this mostly doesn't work.
I think the system is rather a one where if you offer critical services then you're not allowed to use a software that hasn't been developed up to a particular high standard.

So if you develop your compression library it can't be used by anyone running critical infra unless you stamp it "critical certified", which in turn will make you liable for some quality issues with your software.
Waiting in a queue at an airport in Palma, Mallorca, Spain right now and the check in staff are currently flipping through printed sheets of paper to check us all in. It's going to be a very long wait.
The corporate training always says you are responsible for your chosen cloud vendor's problems, but in reality everyone assumes they can successfully blame the vendor.
The generally accepted (but not well tested) legal position is that it's ok to have a proprietary kernel module that is dynamically loaded.

You can, for instance, ask a running kernel if it is "tainted" by having loaded a non-GPL module.
If it works the same way - absolutely.
Pushing an update to all clients worldwide simultaneously isn't "move fast"?
Perversely, this may make many companies no longer invest in this type of cyber security software. Which may lead to a whole host of other problems...
It works this way because we, as a society, decided we wanted it to.
Yeah, these events will be fun once new product liability directive (that includes sw) comes into force.
Rub' Al Khali (The Empty Quarter)
Socializing Homo Economicus
Hey hey, Silicon Valley just bought themselves a VP to ensure no regulation.
Redis.com's "Free" Ebooks
Netherlands is somewhat affected: Two main airports, Rotterdam harbour, a few hospitals and news reporting.

Surprisingly: banks, government, police, fire department, railways, buses are mostly unaffected.

Maybe they have a good IT department/provider.
Sadly, I'm going to have to update examples in my blog post... https://www.evalapply.org/posts/software-debt/index.html#sof....

Software debt is networked.

I'm writing this in the wake of the aftermath of the disclosure of the log4j zero-day vulnerability. But this is only a recent example of just one kind of networked risk.

With managed services we effectively add one more level to the Inception world of our software organisation. We outsource nice big chunks of supply chain risk management, but we in-source a different risk of depending critically on entities that we do not control and cannot fix if they fail.

Not to mention the fact that change ripples through the parallel yet deeply enmeshed dimensions of cyberspace and meatspace. Code running on hardware is inexorably tied to concepts running in wetware. Of course, at this level of abstraction, the notion applies to any field of human endeavour. Yet, it is so much more true of software. Because software is essentially the thoughts of people being played on repeat.
As someone said earlier in these comments the software is required if you want to operate with government entities. So until that requirement changes it is not going anywhere and continues to print money for the company.
presumably they handle credit card transactions
The inconsistent use of pronouns (it Vs them) and asking for the player's name but then almost exclusively referring to "the player" spoiled it for me a bit. But I liked the overall idea.
Ryanair – when every page is a dark pattern
It's one of those things that if you need to ask why, you'll never understand :-)
Don't buy tills that run Windows. You must be crazy.
Most likely. I think laptops are particularly gnarly, especially when they have both an apu and a discrete gpu. While manufacturers use windows' amenities for adding their own drivers and modifications so that they ensure that the OS understands the topology of the hardware (so that the product doesn't get mass RMA'd), there's no such incentive to go out of your way to make Linux support it.
Yes, but that's like 1% of the actual surface area for "running a script". I am not a Windows expert but on, say, Linux you can overwrite a script that someone has already run, or modify a script that is already running, or use an interpreter that your antivirus doesn't know about, or sit around and wait for a script to get run and then try to swap yourself into the authorization that gets granted for that, or…there's a whole lot of things. I assume Windows has most of the same problems. My confidence in Carbon Black stopping this is quite low.
Most of the large deployments I've seen don't use pre-boot PINs, because of the difficulty of managing them with users - they just use TPM and occasionally network unlock.

So might save a few people, but I suspect not many.
Setting aside the utter fecklessness if not outright perniciousness of cybersecurity products such as this, I hope this incident (re-)triggers a discussion of our increasing dependence on computing technology in our lives, its utter inescapability, and our ever-growing inability to function without it in modern society.

Not everything needs to be done through a computer, and we are seeing the effects now of organizing our systems such that the only way to interface with them is through a digital device or a smartphone, with no alternative. Such are the consequences of moving everything "into the cloud" and onto digital devices as a result of easy monetary policy and the concomitant digital gold rush where everyone and their dog scrambled to turn everything into a smartphone app.
Crowdstrike marketing slogan on their website: "A radical new approach proven to stop breaches". I'll give them that: Putting all Windows computers within a company into an endless BSOD loop is a very radical approach to stop breaches. :)
They make malware that steals funds from corporations (willingly!) so these corps can tick a security checkbox for some certification investors have been told is paramount; it's just disguised as security software.
From a purely "winning in the market" perspective, I think there-abouts. For optimal, I might go up a little bit more, since there are multiple products, but not a lot more. Some of the CI/CD stuff is pretty disconnected from the core product.

There are very, very strong individual pressures to overhire. "I managed a 500-person team" sounds a lot better on a resume or otherwise than "I managed a 5-person team" (even if the 5-person team did more).

I also wouldn't go all cash. I'd do the normal $150-$200k cash, and the rest in equity. I'd be very, very generous with equity.

There are also time pressures, whereas many things take time to build. 5 people x 4 years >> 20 people x 1 year, at least if kept productive (which is sometimes hard with longer timelines)
> "50k windows-based endpoints or so. All down."

I'm a dev rather than infra guy, but I'm pretty sure everywhere I've worked which has a large server estate has always done rolling patch updates, i.e. over multiple days (if critical) or multiple weekends (if routine), not blast every single machine everywhere all at once.
Maybe they have rolled back the update and Windows boxes in Europe are no longer pulling it?
Welcome to the enterprise. Where “lift and shift” was sold to corporate CTO’s as better than maintaining their own IT infrastructure.
How does such a huge company do “full deploys” like this?
At this number of endpoints, only a few % should have been updated (and faced the problems) before a full rolout

This is not a small startup with some SaaS, these guys are in most computers of too many huge companies. Not rolling out the updates to everyone at the same time seems just too obvious
Sure, and all those rules were created when a building without multiple exists burned down and many people died.

Maybe this crowdstrike outage will be the "burning building" that will bring change :)
Like any tool, you need to learn how to use it and iterate with it. Trying it once is not enough.
Wonderful read, thanks for posting!
You say that as it's some bad thing, but it's just other words for "use boring tech".

Yes, there could be reasons to choose a lesser-known product, but they better be really good reasons.

Because there are multiple general reasons in the other direction, and incidents like this are actually one of those reasons: they could happen with any product, but now you have a bigger community sharing heads-ups and workarounds, and vendor's incident response might also be better when the whole world is on fire, not only a couple of companies.
Don't forget about the BSD's. We should not rely on a single Finnish man.
are you sure about that? anti cheat stuff is pretty invasive these days
Is this something that could be solved by building AI code review directly into git clients? I can't help thinking Claude 3 would have caught this.
masters/phd gives you a big portion of time and access to journal papers to really do the research. It's like a muscle you have to train, I'm not sure it's easily done while working full time. How much of your day can be spent reading journal papers?

In my Masters degree I learned how to do proper academic research, literature reviews, paper writing, building up your body of knowledge from textbooks as the starting point to papers published in the past, to papers just being published now, and by that point you're at the cutting edge.

By the time you finish a masters or PhD you should more or less know which academics around the world are working on your niche area and what they are publishing, and read all of their stuff.

By the end of my masters I knew the ~10 people in the world who cared about my topic and had read all of their most recent papers.
Operator overloading isn't to blame.

If I overload shift-left `<<` for a completely different concept such as "piping", that's my mistake. That's like writing a normal function or method and calling it `foo()` when it has nothing to do with the concept of fooing.

That said, unless you are writing a math library or some container, there's not many good uses for operator overloading.
A Digital Model of the Ancient City of Babylon
We've had production outages caused by Microsoft Defender our RHEL boxes :(
Crowdstrike: What % of sys admins can remotely boot to safe mode, given BSOD?
What kinds of setups would make this possible?

For those who can't do it, what is their least bad option?
This isn't some global conspiracy, it's just incentives and economies of scale. When it's cheaper to pay a hyperexpert to handle your security, why wouldn't you?

The fact that physical distance is no longer a limit to who you do business with means that you can select the cheapest vendor globally, but then that vendor has an incentive to hyperspecialize (because everyone goes to them for this one thing), which means that even more people go to them.

Avoiding once-in-a-century events just isn't something we're willing to pay the extra cost for, except now we have around twenty places where these once-in-a-century events can happen, which kind of makes them more frequent.

How much stuff do you host on Hetzner instead of AWS?
How about people in charge of choosing these clown solutions - both crowdstrike and windows?
Peter de Schrijver RIP
Before reaching the "pushed out to every client without authorization" stage, a kernel driver/module should have been tested. Tested by Microsoft, not by "a third party security vendor shitting in the kernel" that some criminally negligent manager decided to trust.
Is it KLM, or just the KLM flights operated by Delta (of which there are a decent chunk)?
America has much stricter rules about this than other places, actually.
A convenient alibi?
> 78% or 78 degrees proof?

It's always percent ABV (alcohol by volume). As you can see in this table, typical values for rakia (rachiu in Romanian) are 40–86% ( https://en.wikipedia.org/wiki/Alcohol_by_volume ).

Degrees proof is a term originating in English tax law where the drink catching fire was deemed 100 proof. But it's only used in the US now.
Many in regulated industries actually are forced to use crapware like Crowdstrike, to tick a box on the security checkbox.
I think the enterprise software ecosystem currently is not really "all eggs in one basket", but rather you have a whole bunch of baskets, some of them you are not even aware of, some are full of eggs, some have grenades in them instead, some are buckets instead. All baskets are being constantly bombarded with a barrage of eggs from unknown sources, sometimes the eggs explode for inexplicable reasons. Oh yeah and sometimes the baskets themselves disintegrate all at once for no apparent reason.
For sure, but Crowdstrike eventually hosts a subproject some random person in nebraska has been thanklessly maintaining since 2003 :p
Sorry I confused them with SolarWinds. Strike that
possible workaround if you have bitlocker.

https://x.com/attilabubby/status/1814216589559861673
> the single biggest security improvement

Clearly not, unless you don't count a world-wide economic and societal disruption of unprecedented scale a security incident.

> This could have been any security vendor

...that apparently deploys Kernel Extensions to millions of Windows devices at once, without any staggering.

> there is a reason why they are so widely used to begin with

Because companies need to check a box, and purchasing CrowdStrike checks that box.
OTOH... I remember an O365 outage in London a few years ago.

You're down? Great, so are your competitors, your customers, and your suppliers. Head to the pub. Actually, you'll probably get more real value there, as your competitors, customers and suppliers are at that same pub. Insurance multinationals have been founded from less.

That didn't affect any OT though, so it was more just proof that 90% of work carried out via O365 adds no real value. Knowing where the planes are probably is important.
But it checks a box somewhere so it has to be good. How can anyone run an IT infrastructure without buying a dozen "security" solutions in this day and age? Just designing your network with security in mind instead? Are you nuts?
There's only one piece of software which (with adaptations) runs every Airbus plane. The cost of developing and modifying that -- which is enormous -- is amortized over all the Airbus planes sold. (I can't speak about Boeing)

What failed today is a bunch of Windows stuff, of which there is a vast amount of software produced by huge numbers of companies, all of very variable quality and age.
I suspect that deploying a new image via PXE will be the fastest resolution.

Ideally one you just built without crowdstrike.

If you have important data trapped in userspace, print out some bitlocker keys and get in the car lmao.
The other aspect of risk management is an acceptance that something going wrong isn't necessarily a reason to change what you are doing. If the plan was tacitly to run something at a 99% uptime, then incidents causing 1% downtime can be ignored.

We are going to get hit by some terrible outage eventually (I hope someone is tracking things like what happens if a big war breaks out and the GPS constellations all go down together). But having 10x providers won't help against the big IT-related threats which are things like grid outages and suchlike having cascading effects into food supplies.
Rolling out updates in an A/B test slowly is the only way to reduce the occurrence of such issues _significantly_. There's no other way, literally, nothing.
Yes, the CS Falcon agent caused a kernel panic on RHEL about a month ago.