skysight-inc/HackerNews huggingface dataset:
> You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated.

That seems like the system GP just described, including the hairiness you can get into.

> Why is it a requirement to be able to instantly revoke a person's badge access everywhere in real time anyway?

With 140,000 employees and hundreds of locations across the globe it becomes a question of dealing with hundreds of local databases and dealing with people that need to be in more than 1 or 1 central database (be it truly centralized or cache coherent).

Also think beyond the "you're fired and standing right in front of me" use case of revocation for why it's a requirement it be instant. A lost/stolen/replicated card can have traveled the world in the time it takes to be realized compromised.
Well, it's talking to satellites 400 kms away.
FB is in the spy, stock is practically unsinkable at this point what with everyones retirement plan buying and holding it.
Facebook has to make some public statement; the shareholders will demand it.

I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd.

I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that Facebook would admit such if it were true unless they were legally required to do so.
Seems like they're back online
Upcoming tweet: "Smoke could be seeen at FB data center, but employees said it's nothing to worry about, it's a workaround to get the fire alarm to trigger and override all doors to open.".

Did Nest smoke detectors have to go to the cloud?
This would make for a very good deep-dive technical discussion in an interview setting, I'm using this.
> ... but as of 22:28 UTC Facebook appears to be ...

Someone assumed London==UTC, when London is 1 hour ahead :) that was actually 21:28 UTC
What is Facebook
?
It has both optional and panicking ways to index, but it steers users away from using indexing in the first place.

The `for` loop is based on the Iterator trait. Iterators typically optimize better than indexing, and can't panic.
Are you ok, Sir?
I don't think he has the strength.
I think it really is people coming to hn specifically to read about fb being down.

What kills me is that we'd be humming just fine if the software I've been working on for so-long-I'm-embarrassed-to-tell-you had been rolled out by now. At this point the only solutions are non-technical. A la the Serenity Prayer, and waiting till tomorrow.

Edit: it does seem to be better for community if things break occasionally, though. People can relate to sheepish devs under excessive load. Nobody relates to a perfectly oiled machine.
Well everyone, that was a small glimpse into what it’s like when shitty third world countries just turn off the internet. Combine and centralize more everyone.
Factors? Sure. But I don't think its the big scandal everyone tries to make it out as. Which you can kind of tell because the story never includes any of the pertinent facts, its just boiled down to the most sensational version.
Nothing important for the world. But at least the employees could be able to enter the building.
Can't wait for the post mortem technical post
Facebook and Messenger are working now.

Instagram and WhatsApp - not yet.
Fwiw if you are an outsider, having gone through this exact issue myself recently start with venv:
https://docs.python.org/3/library/venv.html

It is a standard library module and serves as the base functionality, most the other things you list are third party tools that build on the venv concept in various ways.

Once you grok venv, you will be in a position to understand if those tools can provide value to your particular situation.
Facebook automates network peering to enable smoother web traffic management
not really though
And it clearly isn’t that as they don’t award them every year do they. Pathetic strawman
first rule of Blind, never talk about Blind
What I wonder if this isn’t a scheme on Facebook part to show the world what it feels like without it. At first you might think it’ll “wake up” the addicts and whatnot but you can’t just cut off an addict from their supply and expect them to just be ok with it. Instead they’ll relapse harder without help and might dig in deeper, never wanting to feel the withdraw again. I wonder if there’s a phenomenon called Social Withdrawn Symptoms.
Turn down the UI scaling. That works well enough for me. I also run Nova7 as my launcher, so I can control icon and text sizes independent of the UI scaling.
Openess.

Apple is the #1 most restrictive company in the world. "the ability to install Linux" with zero commits and support from Apple themselves, all reverse engineered. Meanwhile Intel and AMD both contribute to the Linux kernel.

How much kool aid do you have to drink to think that 'ability' is actually something you can say is a feature with Apple devices?

Come back when I can unlock the bootloader of an iDevice and we'll talk about the 'ability' to install Linux.
"No one will notice the understatement if the problem is solved fast enough."
Instagram, Facebook and messenger working for me. Whatsapp still down though, for me anyway. Cant like or comments posts. I get this message in dev tools: "A server error field_exception occured."
They must have shut it off and turned it back on
Yeah but reading about it but also being able to communicate about it on the largest network (the one in question too) are 2 separate phenomena. No one can go on there right now and say I'm deleting my account, who's with me?
This is very true. I tell everyone who'll listen that every competent engineer should be well versed in the nuances of feedback in complex systems (https://en.wikipedia.org/wiki/Feedback).

The most successful systems rely on the property of feedback (https://en.wikipedia.org/wiki/Feedback): evolution, untrained learning, genetic algorithms, the diagonal arguments (https://en.wikipedia.org/wiki/Diagonal_argument), artificial general intelligence (https://en.wikipedia.org/wiki/Technological_singularity), financial markets according to no less than George Soros (https://en.wikipedia.org/wiki/Reflexivity_(social_theory)#In...), etc.

That said, virtuous cycles can't exist without vicious cycles. I think we as a society need to do a lot more work into helping people understand and model feedback in complex systems, because at scales like Facebook's it's impossible for any one person to truly understand the hidden causal loops until it goes wrong. You only need to look at something like the Lotka-Volterra equations (https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...) to see how deeply counterintuitive these system dynamics can be (e.g. "increasing the food available to the prey caused the predator's population to destabilize": https://en.wikipedia.org/wiki/Paradox_of_enrichment).
We are a dying breed... A few days ago my daughter asked me "will you send me the file on Whatsapp or Discord?". I replied I will send an email. She went "oh, you mean on Gmail?" :-D
There's "separate infrastructure" such as separate domain names, and then there's "host the status page on AWS".

Normal "separate infrastructure" would have failed here. And an AWS-hosted status page wouldn't have been of any use; apart from "everything is down" the status page wouldn't have known anything either. Simply having the status page be down is all the signal necessary for such a catastrophic outage.
If past outages are anything to go by there’s no chance of this seeing the light of day.
Nothing is ever a solution, only a hurdle.
Timezones are the most annoying thing... right after encoding
it is not... at least I can't login... https://i.imgur.com/tr3ElGL.png (screenshot taken at 18:22 10/4/2021 Eastern time) ... I that think I have never seen a monopoly been down this long before
Learn the differences between APIs vs. Webhooks and when to use them
mark accidentally logged out. first time he'd done it since it started.
This is why firefighters practice breaking into buildings.
Very disappointing.

Hopefully some enterprising ransomware gang can do the world a service and encrypt all of FB's data for them, possibly upgrade all of their servers and backups to being /dev/null hosts entirely.

A man can dream.
People join the military out of patriotism.

54.3% of under 18s in Belgium have a migrant background and any sense of national pride in the native population is maligned to accommodate them.

The Belgians I talk to want to leave. They don't recognise their neighbourhoods any more.

Belgium and many countries in western Europe are going to struggle attracting recruits in the years ahead.
Who's "mark"?
The people working on Facebook infra are true Software Architects.

I can't certainly imagine the pressure of those involved in causing and fixing this issue.
I was hoping it would stay down for longer haha
lots of ISPs straight up ignore ttl
> shitty third world countries

let's tease out your imperialistic racism more. which countries are you talking about?
My tinfoil hat theory is they allow the jailbreaking scene to persist instead of shoring up holes within a second of each jailbreak's release because they poach tweak ideas. When looking at the overlap between popular early jailbreak tweaks and features that were added to iOS over the years, this really doesn't seem farfetched.
Not that fatal. I think at least some recursive servers will do 'collapsed forwarding', where additional requests to resolve the same name while the first request is in progress will wait for the first request to finish and send the same results to all clients at that point. Although, perhaps that's just wishful thinking on my part.

Then you have port limits, usually each request goes out on a new port, a recursive resolver can only have 64k requests outstanding to any given authoritative (or upstream) server IP for each IP the recursive uses. Facebook runs with 4 hostnames listed, so that's a limit of 256k requests outstanding, 512k if your recursive does IPv4 and v6 (and 1 M if they're also making whatsapp requests).

DNS services for both domains appear to be back up by the way.

On the authoritative side, it's not too hard to manage this load. If you can't handle the big crush to start with, drop all requests, and then accept all the requests from 1.0.0.0/8, and add one /8 at a time as CPU permits until you're allowing everything. Once you handle the initial crush from a resolver, it should go back to normal load, and there should be some distribution of load across the various /8s. I wouldn't expect it to be evenly distributed, but it should be even enough.

Disclosure: I worked at WhatsApp, but left August 2019. I don't know anything about this outage other than idle speculation. I don't know if FB has a procedure to slow start DNS, but the theory is simple; the practice is complicated by the DNS ips being used in Anycast.
That should not be the case. Any existing user should be in the local cache. Most companies will have a controller in some control room somewhere which manages the door readers. New users yes. Former users should also need a connection. But existing users should not need a round-trip ping.
Very humbly, I don't think she's as smart or knowledgeable as most people think she is. I had two group conversations with her, and in both cases I just failed to understand her arguments. She was also wrong on both cases, in retrospect.

An example? vCloud Air can be credible only when it will have a global presence. vCloud Air then proceeds to build/buy locations in Japan, Germany and Australia. One year later, vCloud Air is no more. Guess the global presence didn't matter that much. The problem was the product, not the locations.

Sorry, but my opinion on analysts that never built a company or never really practiced has never been particularly high. It might just be my own personal bias.
Refers to Mark Zuckerberg, founder of Facebook
Will they learn? No.
Don’t listen to what successful people say. Watch what they do.
Migrating to OpenSearch with CloudFormation
So the U.S. couldn't defeat the Taliban because Pakistan didn't allow it. I must say that I didn't the U.S to be deterred by Pakistan.
https://www.theatlantic.com/science/archive/2021/05/spike-pr...

If there was a tradeoff between making the vaccine safer or more effective, I guess I would agree to making it safer. That's what it seems like they did by focusing on the spike protein, which was said to not be a defining characteristic of coronaviruses in general. You should understand that the spike protein they did produce may not be nearly close enough to resembling an easily recognizable virus for the immune system to do exactly what we want it to do. This is also one of the reasons that natural immunity seems to be so much stronger than vaccine-acquired immunity. The vaccine might be called good enough by some, but the delta variant overwhelming all our efforts so soon has to be called less than ideal. In fact, the vaccine was supposed to be the light at the end of the tunnel, but it's obviously not. Little has changed, and we aren't doing all that much besides chasing after a magical vaccination or booster shot rate even as Israel is showing us that it's all for naught and mutations are still appearing.

Teenage boys are at a higher risk of heart complications from the vaccine. If one of them dies from a sudden heart condition, are we supposed to say something that make it seem like getting the jab was the honorable thing to do? Imagine if a parent went through the trouble of proving that their child has already been infected with the virus and has a family history of heart conditions or something. With the current, inflexible thinking about vaccine mandates, the powerful might end up pressuring or forcing them to get vaccinated anyways.

I think you only implied it, but the idea that the unvaccinated were the source of mutations was based largely on estimates that the viral load would be higher when they get infected. That turned out to not be the case when looking at the high viral load of people with breakthrough infections. In fact, as a lockdown skeptic, I think it's becoming clear that locking down indefinitely and losing sight of hospital capacity was a mistake that can explain our current plight. The perfectly healthy and young population normally become our shield from viruses each year. The vulnerable were the ones that needed to avoid infection, and they can't do it forever (and a booster shot may still not even be enough). Then again, we listened to Dr. Neil Ferguson and told Dr. Sunetra Gupta that she didn't know what she was talking about, so I am sure you see it differently.
This one in particular seems interesting. Felt like one of the longer outages with soo many different sites effected.
Facebook going down meant more than just a social network being unavailable
Damn, nothing good ever lasts huh?

Even if it was only a few hours things have significantly improved in my life in the period it went down. Zero anti-vax spam from FB sent to me via texts from older relatives.
During the time Facebook (and related properties) have been unavailable, I can't help but feel a little better about the world. There is so much hate, arguments, misinformation, angry, and harmful messaging on FB platforms. It's nice to know all of that crap is just....gone. At least for a little while.
Getting "Query Error" if I click Notifications and after dismissing it, it keeps coming back.
https://www.aljazeera.com/amp/news/2021/3/3/mapping-internet...
There's some explanation by CloudFlare available [0]. But for sure that'd be interesting to read facebooks version.

[0]: https://blog.cloudflare.com/october-2021-facebook-outage/
It’s called broetry.
Thank you, I'm glad you liked it!

It's not difficult at all as we have our own plugin. After signing-up just go to Welcome -> connect your blogging platform -> WordPress. You will find a step-by-step guide on how to connect it.

Converting audio is super-easy with this WordPress plugin.
Check every time, but read from cache upon failure and update cache on success.
On Firefox you can use containers. Two tabs, one logged in, one logged out.
"only a few hours" for Facebook is literally billions.
For comparison Netflix spends about $1.2 Billion dollars per year on AWS (Amazon Web Services). Their total revenue last year was 27.5B. So that means that AWS fees account for 4.3% of Netflix's revenue.

AWS is streaming video, operating hundreds of edge location CDN locations, and everything else running on expensive server infrastructure. AWS has real serious costs with offering that service, to the point that Netflix has yet to determine that it is better to build it themselves.

Apple by comparison demands 15% which is 3.5x as much of a cut. Apple provides credit card processing, and approval into the App Store (plus according to a comment in this email, it sounds like they keep Netflix in the App Store promotion rotation as "free" ad-space). This isn't anywhere near the value that AWS provides, despite demanding a much higher premium. Of course this is the "generous" 15% IAP cut, instead of the normal 30% which is even more atrocious.

Apple seems to maintain the expectation that they deserve 30% of revenue from massive corporations like Netflix, Amazon, etc.. Even the "generous" 15% fee is quite atrocious, especially considering the fact that Apple provides next to no value for Netflix, if anything they are a pain in their side or a necessary evil for Netflix.

Apple is wildly delusional on the value that they provide. It is absolutely no surprise to me that Netflix would consider not allowing IAP subscriptions at all. They already have the marketing power to force people to subscribe at their website. I am surprised Netflix is honestly being as understanding on this issue as they seem to be.
The difference is whether it is removed for being wrong versus for political/ideological purposes. You would not call a newspaper editor who refuses to publish dishonest journalism a censor, would you?

Obviously there is a stance going around that 'calling things wrong' is being done for political/ideological purposes, but that doesn't make fact-checking censorship _in principle_.
Notes as Facebook comes back up:

"registrarsafe.com" is back up. It is, indeed, Facebook's very own registrar for Facebook's own domains. "RegistrarSEC, LLC and RegistrarSafe, LLC are ICANN-accredited registrars formed in Delaware and are wholly-owned subsidiaries of Facebook, Inc. We are not accepting retail domain name registrations." Their address is Facebook HQ in Menlo Park.

That's what you have to do to really own a domain.
A massive service outage like this could help in some ways. Bypassing usual change control, bypassing logged auth systems, missing logs as log servers were inaccessible, bypassing physical access controls in DCs to implement emergency fixes...

I have absolutely no beliefs in any direction at this point but a major service outage (I can't imagine how it could have been much worse) could act as cover for either internal nefarious actions or even a bad actor if they were quick/prepared enough to exploit the situation.
It’s giving an SSL error for me.
Well, for FB there are quite a lot of comments suggesting for it to stay down. So Signal actually got off quite easy.
I feel sorry for someone who has such a dogmatic view of the world.
Unfortunately Twitter was still alive and QAnon conspiracy accounts were making fun of Facebook. Maybe next time.
All doors are glass with the right combination of a halligan bar, an axe and a gasoline powered saw.

And I guess beyond that point, walls are glass. Or you need explosives.
A Gmail outage would be barely an inconvenience as Gmail plays a minor role in Google's disaster response.

Disclaimer: Ex-Googler who used to work on disaster reponse. Opinions are my own.
Which in many cases is morally questionable or downright illegal.
I see it as similar to Snowden, in the sense that everybody kind of knew (actually guessed) but now we actually know. It doesn't come as a shock, but it's important information to have since it can be now argued with authority.
For those you understand Swiss German:
Mani Matter – I han es Zündhölzli azündt https://www.youtube.com/watch?v=PkGatIgXERI
All the hate and comments but you forgot what keeps it on. The ads. All the ads.
The Video Partner Program which takes 15% went into effect in 2016.
> Every controlled door is wired to a relay panel. That panel connects to a access control db

Absolutely not. E.g. HID is a well-documented protocol and you can do whatever you want using the RS232 interface they come with (depending on the model). I personally wrote a gateway that would dynamically and directly check the access with my own software.
It can be nice to know what’s going on, or at least provide ETAs if known. At least that’s what I do for my users when things go down.

Not sure if FB would provide any actionable info, but can be nice to know if you should hang around for the service to return or take a nap to be ready for when it does return.
I don't get so much 'terrible' stuff, I deleted my account but re-signed up for a local cause.

After ignoring "region to the West of me" local news, I've had to delete about 40 other regions spanning farther outwards. FB keep pushing more data for you to interact with which in itself is not exactly healthy.

And it's easy to say ignore it, the simple fact it's in front of me, in the part of the screen where I'm reading new information is the problem.

It's not radicalising, it's just an absolute time and life suck. If it weren't for COVID and lockdown I'm sure my local cause would prefer not to use Facebook.
Not OP, but many times. Really makes you think hard about log messages after an upset customer has to read them line by line over the phone.

One was particularly painful, as it was a "funny" log message I had added the code when something went wrong. Lesson learned was to never add funny / stupid / goofy fail messages in the logs. You will regret it sooner or later.
One step at a time. ;)
New Zealand drops ‘Covid zero’ goal after outbreak
Lots of people interested in the event and normal web response queries their sql database for user data like upvotes which is their limiter.

Logout which I believe they just forced for all users sends you to the cache and it'll be fast. dang@ mentioned it during the giant S3 outage a few years ago.
Is this just a thrown away observation or meant to be some sort of insight?

Determinism isn't exactly useful as a personal guideline, or anything. So I don't know why some people keep digging that hole.
And HackerNews seems to be much faster as soon as FB is back up. Not sure if this is just me.
Could anyone explain this so people with no DNS knowledge could understand?
I'm not sure if this discovery will ever amount to a "beyond reasonable doubt" standard of proof of a cover-up, especially in an internationally accepted judicial setting, but I do wonder if, in general, there needs to be a legal principle that hiding evidence of something has to be punished as harshly as the worst plausible crime that the evidence could be hiding.

Such a principle would, admittedly, lead to many defendants being punished for crimes they didn't actually commit (i.e. they were only covering up lesser crimes), but without such a principle, a guilty person will always face a rational choice of covering up their crime and risking only an additional "destruction of evidence" charge (for example), as opposed to being caught for the actual crime and incurring the (presumably harsher) punishment for that.

For such a principle to pass the smell test, though, it should first and foremost apply to government record laws, covering everything from politician's official emails to police body-cam footage. A society which elects politicians willing to be held to that standard can be assumed to be mature enough to have this principle applied to other criminals.
worst scammers are the one who ask if you know amazon.com and close the call when you say no :(