skysight-inc/HackerNews huggingface dataset:
Ask HN: What is the Backup plan for another outage like this?
With windows and other OS being installed on many devices for daily usage, what can be done to reduce centralised point of failure? Todays outage made it clear that windows is used on many critical systems, and another outage like this can happen again which will result in many problems.
If it’s illegal, how about not sending illegal invoices to people? Just an idea.
I have no objection to calling "almost all criminals vote Democratic" a "claim" rather than a "statistic," although I'm not sure what it changes. I still doubt the claim, and, even if there is no pre-defined numeric threshold at which one can say precisely that one more criminal voting Democratic would mean "almost all" do, and one less would mean the contrary, then there are certainly quantitative data that could be judged to refute it—for example, if fewer than half do—or that could be judged to be evidence for it—or example, if 90% do. I think it is reasonable to ask for a reference to some confirmatory evidence for such a statement.
This.Is.Pathetic.

Seriously. Software should NOT be this bad that your fix begins with reboot up to X times.
We still play Jazz standards from 100 years ago. It seems if you want to be able to have your own take on something you should be able to do it the intended way, capture l'espirit, and then impart your own.

Also it's much cooler to switch back and forth between styles.

Synthwave is pretty cool, but it'd also be pretty cool to hear a Moog play a piece from an old composer instead of a simple drone.
Love it see it, I think most "designers" like this tools are behind paywalls and subscription email platforms. This looks great thank you for sharing.
Yes that seems to be the case - the blocking page is headed with "Ministero dell'Interno -
Dipartimento della Pubblica Sicurezza - Direzione Centrale per la Polizia Scientifica e la Sicurezza Cibernetica - Servizio Polizia Postale e Sicurezza Cibernetica" along with official looking government logos but text underneath (in Italian and English) talks about a collaboration between govt agencies and ISPs. From mobile cell service it was not blocked.

The text does however mention it's a specific measure against child pornography, not re-hosting copyright content.
How else are you going to update your grocery list while operating the lift?
It's not a serious issue, as you see they clearly have all the fancy bling bling logos on their site. Processes were followed. ISO standard numbers were chanted. It's a completely isolated _accident_ there's no scale at all here, and they could have done nothing to prevent it, duh. And going forward they will hire a Chief This Never Happens Again Officer and everything will continue to be good.
Al Michaels gonna prove to be a more valuable IP than Oswald the Lucky Rabbit.
They aren't deleting the knowledge of the species, just changing the name. They aren't saying everything is racist, just that racial slurs are.

I'm pretty sure you knew both of these things when you wrote your comment too, so I'm struggling to understand what is going on here.

Maybe you should rephrase as like "I don't mind a little racial slurring, it adds spice to science" or something? "It keeps us grounded in the scientific tradition of personal naming of species, and the racism is an acceptable cost for it" would at least be honest.
> what would the dream free market implementation even look like here?

Every company's dream: Free labor and captive audiences.
What % of those sysadmins are then going to turn around and script something to auto-approve those updates, once they realize that they are A) requested at inconvenient times and B) are related to security?

Who's going to take the risk of appearing to have sat on an important update, while the org they support is ravaged by ThreatOfTheDay, because they thought they knew better than a multi-billion dollar, tops-in-their-field company?

(I'm not necessarily saying that's actually objectively correct, but I can't imagine that many folks are willing to risk the downside)
Article states they had fewer biomarkers of aging, and showed less age-associated obesity, fraility, and metabolic decline. So their "youth" period extended longer.

Actually the point of healthspan vs lifespan is made in the introduction -- it was a justification for using mice for this study, which have clear health phenotypes that are analogous to humans.
Yes correct, SMS was 160 so 140 was used to provide room for the username.

This is back when the main UI of Twitter was via texts.
Ukraine rushes to create AI-enabled war drones
That being said, when Super GMs play a bad move against a lower rated GM, they quite often gets a pass. They don't capitalize, simply because they assume the move is excellent.
I've been wondering the same. I did just see [1], where it's apparently trying to read memory from an unmapped address, but I haven't seen anything about how r8 got to the point of having said unmapped address.

[1]: https://x.com/patrickwardle/status/1814343502886477857
That data doesn't appear to have personal names or anything else that can be used to connect them to a specific individual.

It's just all the addresses they deliver to with things like building type, schedules, route ids, etc.
It's not the first time they pull something similar...1 month ago:
"CrowdStrike bug maxes out 100% of CPU, requires Windows reboots" - https://www.thestack.technology/crowdstrike-bug-maxes-out-10...

75 Billion dollars valuation, CNBC Analysts praising the company this morning on how well the company is run!...When in reality they can't master the most basic of the phased deployment methodologies known for 20 years...

Hundreds of handsomely paid CTO's, at companies with billions of dollars in valuations, critical healthcare, airlines, who can't master the most basic of the
concepts of "Everything fails all the time"...

This whole industry is depressing....
I am, and believe I am (mostly) sane, empathetic, and relatively intelligent.

Its fine if things aren't perfect. We're a country with a lot of very different people with very different beliefs. Things are going to go wrong, but they tend toward getting better with time.
Well cryptolockers have actually compromised various hospitals and I remember the first one was in the United Kingdom .
This seems really optimistic.
The laser was 30 years old at the time and kept in expectation that it would go to a museum, perhaps to the Smithsonian or equivalent. I didn't use it in my work. My advisor's retired technician would still come in once every couple of weeks because of his love for the Labs. Towards the end of my stay he and I put in fresh gas tanks and fired it up. All the 1960s vintage glass, power supplies, mechanical and optomechanical systems worked and it lased like a champ and made firebrick beam dumps glow.

I don't know what became of it.
> People is going stupider by the minute.

*are

And no, they're not.
Unless that threat was a 0day bug that allows anyone to SSH to any machine with any public key, it was not worth pushing it out in haste. Full stop. No excuses.
Because security experts seem to have this slight dismissive attitude about companies' and individuals' attempts to do security, while not usually having answers or providing secure systems.
OS is irrelevant in this case, and CrowdStrike deserves all the blame. They literally brought down Linux systems earlier this year. https://news.ycombinator.com/item?id=41005936
Your IT wizard is probably wrong. There is a fix that involves booting into safe mode and deleting a file.

Unless you have an encrypted file system this should be a relatively trivial fix.
Clearly that's how they ended up with the current team. They hired for culture fit. Anyone who worries too much is out.

You bet they have an amazing perfect top-notch hiring pipeline, many rounds of interviews, and whatever you could wish for! (No, no ... the subcontractors writing code are not in scope for this, duh.)
Also a major point in the Black Swan. In the Black Swan, Taleb describes that it is better for banks to fail more often than for them to be protected from any adversity. Eventually they will become "too big to fail". If something is too big to fail, you are fragile to a catastrophic failure.
Just completed a standing 24 due to this outage. My B-Shift brothers and sisters had to monitor the radios all night for their units to be called for emergencies. I heard every dispatch that went out.

We were back in the 1960's with paper and pen for everything, no updates on nature of call, no address information, nothing... find out when you show up and hope the scene is secure. It was wild as it was coupled to a relatively intense Monsoon storm.
Feels like there is some sort of lesson we should learn as a society.
With web, at small scale (which honestly is 95% of the world), you just version and back up everything. We push updates that break stuff from time to time. If it's bad enough, we just hit a button and roll back the change. The nerves are basically a sign that you need to have an easy rollback process in place, once you have it, you sleep easy and things are fun.
I’m all for successful businesses operating within the parameters of the law, but is it not also correct to expect some adherence to a minimum ethical standard?

Exploitation is what it is. Legal or not, it’s gross and it’s what these companies have been doing for years without consequences.

The rates aren’t even really accurately reflected in those per-minute tables. There are also a lot of service charges and other fees, blocks of time must be purchased with minimum amounts ($20 minimum is not uncommon), and then fees are taken from the prepaid funds as they are used, causing the balance to decline much faster than one might expect, and allowing the service providers to further conceal their deceptive billing practices.

Actual average rates can easily exceed $0.50/min, and it shouldn’t be surprising that the folks who depend on these services to maintain family and relationships are frequently not the most flush with cash. This has been a brazen redistribution of
funds from those who have the least resources, to those who have the least conscience.

Somewhat relevant, video calls have been hailed as improving the ability for incarcerated individuals to keep in touch with their loved ones. This is also a cynical lie. Video calls have been used nearly across the board as an excuse to end in-person visitation. It’s cruel, and should be stopped. Some minimum visitation should be afforded to inmates, particularly since many of them are pre-trial and presumed innocent, and in any case their families and loved ones deserve to maintain contact with them, not to mention it’s a positive reinforcement towards rehabilitation and reducing recidivism.
I'm so sorry, TIL.
There's a brokenness spectrum. Here are some points on it:

- operational and configured

- operational and at factory defaults

- broken, remote fixable

- crowdstruck (broken remotely by vendor, but not fixable remotely)

- bricked

Usage:

> don't let them install updates or they'll crowdstrike it.
I love that https://naya.lol/unicorn is the cats page. This is wonderful. Sharing this in the company Slack like a boomer.
Sorry to hear this! I'm a journalist covering this mess and wondering if we could talk. Am at sarah.needleman@wsj.com
Impact of US–China Tensions on US Science: Evidence from the NIH Investigations
I understand the logic of this but it is somewhat based on the assumption - which most industries have in droves - that people in THAT industry are the competent bullwhark against stupidity.

I consulted for a company for a while where the 'sysadmin' was the owner's mother - who bought laptops from walmart. Not only could she NOT have approved updates like this, even if she could she would have she wouldn't have had any knowledge whatsoever with which to make a determination if it worked.

In an abstraction, the problem really is with externalities. These approaches to updates exist because people who CAN'T do what you describe are likely a more dominant part of the threat model than this happening to people you do describe. The resulting fix, as we're seeing, is very reliable until it isn't...and if the isn't is enormous in scale the systems aren't setup to fail gracefully.

If you want to make a rule...require graceful failure.
While true, high ceilings with light scoops could be the rage.
> Does a temporary moat from a single proprietary driver justify a 10x valuation?

glances over at OpenCL's mangled corpse

Yep
There is a way with AWS, but it carries risk. You can force detach an instance's volume while it's in the shutting down state, but if you re-attach it to another machine, you risk the possibility of a double-write/data corruption while the instance is still shutting down.

As for "throwing away local SSD", that only happens on AWS with instance store volumes which used to be called ephemeral volumes as the storage was directly attached to the host you were running on and if you did a stop/start of an ebs-backed instance, you were likely to get sent to a different host (vs. a restart API call, which would make an ACPI soft command and after a duration...I think it was 5 minutes, iirc, the hypervisor would kill the instance and restart it on the same host).

When the instance would get sent to a different host, it would get different instance storage and the old instance storage would be wiped from the previous host and you'd be provisioned new instance storage on the new host.

However, with EBS-volumes, those travel from host to host across stop/start cycles and they're attached via very low latency across the network from EBS servers and presented as a local block device to the instance. It's not quite as fast as local instance store, but it's fast enough for almost every use case if you get enough IOPS provisioned either through direct provisioning + correct instance size OR through a large enough drive + large enough instance to maximjze the connection to EBS (there's a table and stuff detailing IOPs, throughput, and instance size in the docs).

Also, support can detach the volume as well if the instance is stuck shutting down and doesn't get manually shut down by the API after a timeout.

None of this is by any means "ideal", but the complexity of these systems is immense and what they're capable of at the scale they operate is actually pretty impressive.

The key is...lots of the things you talk about are do-able at small scale, but when you add more and more operations and complexity to the tool stack on interacting with systems, you add a lot of back-end network overhead, which leads to extreme congestion, even in very high speed networks (it's an exponential scaling problem).

The "ideal" way to deal with these systems is to do regular interval backups off-host (ie. object/blob storage or NFS/NAS/similar) and then just blow away anything that breaks and do a quick restore to the new, fixed instance.

It's obviously easier said than done and most shops still on some level think about VMs/instances as pets, rather than cattle or have hurdles that make treating them as cattle much more challenging, but manual recovery in the cloud, in general, should just be avoided in favor of spinning up something new and re-deploying to it.
IMO, if you want to write code for anything mission critical you should need some kind of state certification, especially when you are writing code for stuff that is used by govt., hospitals, finance etc.
Texan construction workers put a rocket up Team SpaceX over 'unpaid bills'
Can confirm this is the norm in machine shops. I encounter systems running dos, 3.1, 95, 2k and mostly XP constantly. I rather prefer the old dos systems of the obsolete stuff. Less variables. It is easier and more reliable to freeze the tech in time than it is to manage updates.
Fair enough, but given how crap utilities are at securing user accounts, I do not want to have an account with them. Just send me the invoice and I'll pay it. End of story.
The automation world runs on windows
>It never caught any legit threat, but constantly flagged our own code

When I worked in large enterprise it got to the point that if a piece of my app infrastructure started acting weird the blackbox security agents on the machines were the first thing I suspected. Can't tell you how many times they've blocked legit traffic or blown up a host by failing to install an update or logging it to death. Best part is when I would reach out to the teams responsible for the agents they would always blame us, saying we didn't update, or weren't managing logs etc. Mind you these agents were not installed or managed by us in any way, were supposed to auto update, and nothing else on the system outran the logrotate utility. Large enterprise IT security is all about checking boxes and generating paperwork and jobs. Most of the people I've interacted with on it have never even logged into a system or cloud console. By the end I took to openly calling them the compliance team instead of the security team.
CrowdStrike Falcon may ship as a native package, but after that it completely self-updates to whatever they think you should be running. Often, I have to ask IT to ask CS to revert my version because the "current" one doesn't work on my up-to-date kernel/glibc/etc. The quality of code that they ship is pretty appalling.
> our IT wizard says the fixes wont work on lathes/CNC systems

Why, whY, WHY...are these things connected to the internet?!
This infuriates me because it’s automatic if a merchant has “fast checkout” enabled.
I would be fine with jumpers, ye.
Wow !! good to know real reason for non-staggered release of the software ...

> Crowdstrike bit my company with a false positive that severely broke the entire production fleet because they pushed the change everywhere all at once instead of staggering it out. We pushed them hard in the RCA to implement staggered deployments of their changes. They sent back a 50 page document explaining why they couldn't which basically came down to "that would slow down blocks of true positives" - which is technically true but from followup conversations quite clear that is was not the real reason. The real reason is that they weren't ready to invest the engineering effort into doing this.
Probably has to do this early on in the boot process so as not to require a reboot after update because of Windows’s silly pessimistic file locking.
Fine crowdstrike for 10% their companies value. Only way to ensure they won't try to kill people in the future.
So in essence just a flavor of insurance?

Shouldn't that hit Crowdstrike's stock price much more than it has then? (so far I see ~11% down which is definitely a lot but it looks like they will survive).
Can someone share a link to a copy of the offending channel file. Now a crowd strike customer, but interested in poking at its contents. Thanks!
CNCs might not allow direct Windows access for end-users and require on-premise support from the manufacturer. Our cnc can be remotely serviced… if Windows boots.
I would add that a lot of ppl in this industry also just blindly follow the herd too, without any independent thinking.

Oh, everyone is using Crowdstrike? I guess i have to do so too!

Oh, everyone is using Kubernetes? I guess i better start migrating our services to it too!

Oh, everyone is using this fancy Vercel stuff? We better use it too!

Oh everyone is migrating their workloads to the cloud even tho we dont need to and it costs 5x more?!! We better do so too!!
My only wish is that they had put an oculink port on it.
Most stuff is fine in France.
I'm basically skeptical of any longevity study in mice or yeast or C. Elegans because I think humans already have superior longevity factors over other animals because of our culture: we can live 3x reproductive age and still contribute to the survival of our descendants and our tribe.

Metabolic adaptations that prolong the life of mice might be just bringing them closer to where we are as opposed to something that would work for us.
This borked our dispatch/911 call center then as well. However, it wasn't as bad as this one. This outage put our entire public safety system into the stone age and with that we were at stone age efficiency.
Having read through the entire thread twice now, I cannot shake the impression that Telekom is persistently trying to avoid implementing any meaningful mitigation procedures themselves.
Across the entire thread they communicate under the assumption that

- they didn't make any mistake in delaying the revocation,

- they should be able (given enough insistence from their customers) to do so again at their own discretion in the future, and

- their inadequate handling of the incident and their apparent laissez-faire approach to the requirements for a CA should be accepted and deemed satisfactory by everyone else.
Let's crash on integer overflows that aren't a security bug
Could you please upload it to some place and share a link. Curious to examine the contents.
Ham radio is about whatever you want it to be. It can be about designing and building your own radio then seeing if you can make a contact with it. Same can be said about designing and building your own antennas.

It can be about designing your own digital protocol or software to decode other published digital modes. It can be about attempting to make a contact at the lowest power output with the lowest noise floor at the farthest distance possible for any given situation.

None of these things really require talking to anyone about anything. All that's required is someone to respond with their callsign, location and signal report. 99% of what I've done with ham radio over the last 20 years is mostly just that. Tinkering with radios, antennas, and different digital modes, not sitting and chatting with people.

Maybe you find that all boring too, which is fine, but the hobby is what you make it.
Tech has become such an unbelievable house of cards full of various people covering their asses by offloading these tasks to third party trusted actors.

Consider the recent npm supply chain attack a few weeks ago, or the attempted SSH attack before that, or the solar winds attack before that.

This type of thing is institutionally supported, and in some cases when you’re working with with the government, practically required.

We’re going to see more of this.
I was in my 20s during the peak hysteria of post-9/11 and GWOT. I had to cope with the hysteria hyped 24/7 by media and DHS of a constant terror threat to determine if it was real.

The fact that global infra is so flimsy and vulnerable brought me tremendous relief. If the terror threats were real, we would have been experiencing infrastructure attacks daily.

I remember driving through rural California thinking if the terrorist cells were everywhere, they could trivially <attack critical infra that I don't want to be flagged by the FBI for>

I've read a lot of cyber security books like Countdown to Zero Day, Sandworm, Ghost in the Wires and each one brings me relief. Many of our industrial systems have the most flimsy, pathetic , unencrypted & uncredentialed wireless control protocols that are vulnerable to remote attack.

The fact that we rarely see incidents like this, and when they do happen, they are due to gross negligence rather than malice, is a tremendous relief.