skysight-inc/HackerNews huggingface dataset:
Breaking the glass to get in to fix the service is totally a good business move.

A few hundred bucks of glass Vs a billion wiped off the share price if the service is down for a day and all the user's go find alternatives.
I'm not sure if that is true (and I hope it is not cause that would be fatal) but I read somewhere that with facebook being down also means all internal infrastructure of facebook isn't available at the moment (chats, communication) including remote control tools for the BGP Routers. Therefor they require people to get physical access to the router while many people are working from home cause of the pandemic.
I'm afraid I can't let you into the building today, Dave.
> Well-managed companies blame processes rather than people,

We're six hours without a route to their network, and counting. I think we can safely rule out well-managed.
Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.
Ask HN: Is this a distraction while FB deletes incriminating evidence?
Speculation:

"There's an argument to be made that, while Facebook is off-line, they're sweeping through their storage arrays looking for anything and everything that could implicate them in a hearing, and disposing of that evidence."

-- https://mathstodon.xyz/web/statuses/107045234735019624

Entire conversation: https://www.solipsys.co.uk/Chartodon/107045387088565341.svg
I had an hour+ video meeting on it a little while ago.
This is a good list, but I find that people are often confused about this requirement:

> Similar to encryption in transit, practicing data encryption at rest—using standards like AES-256—will ensure that stored data is not accessible to any application or user unless they present the decryption key.

A lot of people seem to have the attitude "Oh, my DB is in Amazon RDS, and they say everything is encrypted, so I'm good". However, if your DB creds get hacked, they can still read all the info in your DB.

What should really be required is a tokenization or app-side encryption scheme, ideally using something like AWS or GCP KMS so that all decryption requests are logged and monitored, for any even remotely sensitive data that is saved to the DB. That can also make it much easier for developers, because it's safe to give out read-only user access if you're sure they can't read any sensitive private data.
I have a Darden-signed copy - one of my proudest possessions!
Seeing this also.

Copyright 2020. I'll just assume it's not cached :)

Guess now it resolves they'll be dealing with a thundering herd.
Hmm. I wish cars wouldn't exist so my cat wouldn't have been ran over by one.
Rest in peace Netscape, once the most beautiful browser in the world (2019)
I'm not sure it's possible to speculate in a way which is generic over all possible infrastructures. You'll also hit the inevitable tradeoff of security (which tends towards minimal privilege, aka single points of failure) vs reliability (which favours 'escape hatches' such as you mentioned, which tend to be very dangerous from a security standpoint).
Because last week's Apple revulsion ran its course. This week it is Facebook's turn.

We'll circle back to Google next week, and repeat.
Facebook cannot be compared to any other industry. Facebook's ethical problems is not about the services they provide, it is about the quantity of data collected with consent(ignorance). I hope everyone working inside is ethical as believed by the upper management.
Maybe they should not have a single point of failure. FAANG is not good for our economy’s resilience.
Zucks personal conference room has 3 glass walls, so I’ve been amusing myself imagining him just throwing a chair through one of the walls
Can someone explain why it is also down when trying to access it via Tor using its onion address: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t...

Or when trying ips directly:
https://www.lifewire.com/what-is-the-ip-address-of-facebook-...

I would have expected a DNS issue to not affect either of these.

I can understand the onionsite being down if facebook implemented it the way a thirdparty would (a proxy server accessing facebook.com) instead of actually having it integrated into its infrastructure as a first class citizen.
It's remarkable the effect even short TTL caching can have given enough traffic. I recall once caching a value that was being accessed on every page load with a TTL of 1s resulting in a >99% reduction in query volume, and that's nowhere near Facebook/internet backbone scale.
now if only tiktok would fail
ISPs keep floating charging Netflix for their own customers' data traffic; by the same logic DNS operators should be charging Facebook.
A lot of the core 'ops like' teams at FB use IRC on a daily basis.

When I worked there, I wasn't aware of any 'test once per year' concept or directive.

Of course, FB is a really big place, so things are different in different areas.
Not trying to make fun of this person, but I thought there was actually some enjoyable insight hidden in here if you can get past the language barrier:

"I had a Pegasus Mail on my desk top, no Greek temple lemme tell ya, it asks you questions you don't know the answers to, the path, the whatever, but I don't know coding, Earthlink net smapauth server codes bewilder. Gad, what coding? What language? What words?"

At the very least I aspire to write software whose disappearance could inspire a decades-long grudge.
I unfortunately cannot edit the parent comment anymore but several people pointed out that I didn't back up my claim or provided any credentials so here they are:

Google has multiple independent procedures for coordination during disasters. A global DNS outage (mentioned in https://news.ycombinator.com/item?id=28751140) was considered and has been taken into account.

I do not attempt to hide my identity here, quite the opposite: my HN profile contains my real name. Until recently a part of my job was to ensure that Google is prepared for various disasterous scenarios and that Googlers can coordinate the response independently from Google's infrastructure. I authored one of the fallback communication procedures that would likely be exercised today if Google's network experienced a global outage. Of course Google has a whole team of fantastic human beings who are deeply involved in disaster preparedness (miss you!). I am pretty sure they are going to analyze what happened to Facebook today in light of Google's emergency plans.

While this topic is really fascinating, I am unfortunately not at liberty to disclose the details as they belong to my previous employer. But when I stumble upon factually incorrect comments on HN that I am in a position to correct, why not do that?
Agreed, it’s also worth mentioning that at the end of every cloud is real physical hardware, and that is decidedly less flexible than cloud, if you locked yourself out of a physical switch or router you have many fewer options.
> There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time.

Luckily this isn't a pure IT system, but bound to physical bodies moving around. Some delay is acceptable. Most likely the person is already in anyways.
Too bad it’s full of lies:

“Some users”

“May be impacting”

“May be experiencing issues”
datacenterS
Is there a way to secure BGP? Some kind of BGP table backup and restore?
I don't think it's cocky or 20/20 hindsight. Companies I've worked for specifically set up IRC in part because "our entire network is down, worldwide" can happen and you need a way to communicate.
You're awesome dang! Thanks for all you do. That things are up as much as they are is pretty great.

I, and probably a bunch of others, would be pretty interested in a write up of the challenges and how you solved them if you could find time after things settle.
Well, and all their systems presumably being disconnected from each other for the better part of a day.
This distinction is not definitive. None of the normal Linux distributions makes it.
FB seems to be finally loading for me, after nearly 6 hours.

This will be a highly discussed topic for a bit.
Kinda sorta. There are four DNS servers for Facebook: 129.134.30.12, 129.134.31.12, 185.89.218.12, and 185.89.219.12.

Of those, only 185.89.219.12 is up right now (Edit All four DNS servers are now up). For people who want to add Facebook to hosts.txt, the A record (IP) I’m getting right now is 157.240.11.35 (it was 31.13.70.36)
Lots of people drawing conclusions between recent leaks and some sort of attempt to cover them up.

I'd suggest more likely is a disgruntled engineer performing some sort of sabotage because of the content of the leaks.
If its any consolation, I find SMS and telephone to be remarkably robust in these situations. In college during football games, the campus population would swell to probably 200k people within a few square miles each with a cell phone in a pocket. 3g and LTE would be worthless. Campus area wifi would be worthless. The only thing that would work is shutting all that off your phone and resorting to SMS and calling people over EDGE, but it worked flawlessly even with all the people stressing out a handful of towers at once.
If only the cancer apps could be shut down permanently
"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com."

The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)
We and cloud providers charge different things as they charge for the number of processed minutes and we charge for articles.

The thing is, with cloud providers you need to use API and we built the whole UI and automated the process so that it's just more user-friendly.

But if you know how to convert articles just using cloud providers and not BlogAudio, then most likely you'll pay less.
Netflix knows how to build on top of a house of cards.
>How long does the logging need to be on?

This would depend on the warrant.

>If you turn on logging, does it turn on logging for all? (Seems likely yes)

No, the warrant must have a specific (non-drag-net) scope. Broadly scoped warrants are fought to reduce the scope to something specific (reduced time-frame, reduced users, etc.). This is common, even with Google/FB. I'm not sure why you assume yes.

>Therefore one warrant can put all other privacy-desiring users at risk...

No, not really.
Facebook also used brute force to get facebookcorewwwi.onion on Tor a while back. https://en.wikipedia.org/wiki/Facebook_onion_address
When I worked there a few years ago, the routing tables were being updated almost all day every day, primarily via automated processes.
It helps if your dsl line is is bridging at layer 2 in the osi model using rotated psks, so it won't be impacted by dns/bgp/auth/routing failures. That's why you need to put it in a panic room.
If it bleeds we can kill it!
I'll have a go

Legaltech - namely, automating the legal process behind startup funding rounds. We have lawyers on staff to help if needed, but we automate 95% of the work.
I read it brought down the Vodafone network in Czechia, one of the major providers there.
Canada: Won't Get Fooled Again?
It's not the divisiveness that upsets me, I agree with "And in many cases getting divided is our only hope of ever arriving at the truth." It's the fact that facebook is propagandizing people because it's shoving the divisive content in their face nonstop without understanding if it's disinformation first.

There will always be false content on social media, but I don't think that means facebook gets a pass for actively encouraging it.
Nah, a ticking time-bomb would "explode" on Christmas (or Aïd El Kebir, etc.), whenever most of the employees who could do something about it are absent.
The market have never been more ripe for a new player.
“ Now seeing facebook.com and other addresses resolving via DNS.”

CF CTO - https://twitter.com/jgrahamc/status/1445136845986308100?s=21
I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook.

For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.
Margin calls don't really exist as far as loans are concerned. Once you have agreed collateral, and an agreed schedule of payment, you only get in trouble if you miss a payment, regardless of how the collateral fluctuates in value.
I live in Europe. In Amsterdam. I don't get what you mean.

I mean, I get that you don't want your wife being called a filthy whore, I get that you want to be left alone at a bus stop, I get that you want women to dress however they want. I want all of those things too. I just don't see how getting rid of open spaces would accomplish any of that. Are all of these things things that can't happen on narrow streets?
No. But FB's DNS is anycasted.

FB's eggs were all in one basket, and the basket broke.
The usual boilerplate phrases coming straight from the masters of PR
I think that that difference we draw - the line between having responsibility for your actions and not - is pretty arbitrary. I think it's good that it exists since when we're at our most vulnerable it exists to protect us from ourselves (and some people spend their entire lives in such a state if they have a debilitating developmental condition) - but it mostly exists to give all of us folks in society a good deal of comfort that if things ever go really off the rails we'll be given more of an allowance to recover - thus preventing some doom spiraling where feeling mentally unbalanced makes you paranoid about what you're going to mess up in your life which leads to more unbalancing.

And, on the otherside, we are a funny animal that takes comfort in the fact that we control our own destinies so, even if we don't - we'd like to empower ourselves as if we were in some manner. At a basic level our misunderstanding might be the fact that the ability to choose equates responsibility - without any responsibility (good and bad) your choices have no power - choosing to not do the shitty thing lets you avoid a penalty and so you can be happy with yourself. This equation of choice and responsibility is why punishment is so necessary (both big-j Justice like cops jail and the like - and little-j justice like being shunned, not getting a raise and failing to find an equal partnership in a relationship) so responsibility exists to add meaning to our lives. In fact denying it to people permanently (i.e. the developmentally disabled) does come with a strong cost to their sense of self - a lot of folks who live in support systems can struggle to define themselves, but, at the same time, neurodiversity means that different folks can find contentness in different ways.

It's all terribly complicated I think.
Agricultural automation is part of why Uyghur slavery is so profitable (similar to how the cotton gin led to a revitalization of slavery in the US- with a modicum of training, a single slaves productivity was drastically increased)

Closer to home, assuming you're in the US, the glut of cheap corn syrup plays a big role in our obesity epidemic (especially in underprivileged communities). Obesity and related issues are some of the top killers in the US.
You forgot the chat boxes inside other apps like Google docs, Gmail, YouTube, etc.
Alas, if Facebook went down for good, they would probably be replaced by something even worse.
I think you're supposed to try HVAC first.

https://xkcd.com/705/
Maybe now’s not the best time. https://www.vice.com/en/article/z3xpm8/company-that-routes-b...
The classics. Another one of my favourites is "increased error rates" when a service is completely down.
In all cases, yes, the hypothetical employee contributes to FB-spread societal damage.

For example, if I design a boring mechanical part for a bomb my labor is integral to when that bomb kills civilians. So, if you are in the defense industry, believe in the net benefits of the missions that you support. And likewise for FB.

Claiming one is a meaningless cog in a machine, FB or otherwise, and one is therefore absolved of the responsibility of one's downstream impact is disgusting cowardice.
It might take even longer, though the all encompassing impact of the pandemic might bring it to the forefront. Although there are outliers, at this point the average time from discovery to Nobel recognition is over 20 years for the most recent prizes: https://www.economist.com/graphic-detail/2020/10/09/the-nobe...
None of those are lies, they are simply unhelpfully unspecific.

I understand your frustration at imprecise communication - but if you are advocating for precise and accurate phrasing, the least you can do is follow your own advice.
not sure how that is relevant analogy... but if you choose to have an outdoor cat then that's the risk you accepted. There's a reason some people choose to keep their cats indoors.
Yup, not a coincidence what with the damning new report from 60 minutes.

https://www.cnn.com/2021/10/03/tech/facebook-whistleblower-6...
This is not true when your the registrar (as in this case) in fact your entire system could be down and you’d still have access to the registries system to do this update
At 21:44 UTC, facebook.com resolves for me.
Is the status page always only limited to facebook for business properties? Seems like 90+% of their services are missing from this page
I’m just gonna say this. Disclaimer I have no knowledge nor evidence whatsoever that this may be the reality. But speculation seems to be the order of the day…

Seems like cutting their ASN off from the world would be a great way to cut off any would-be Discovery Volunteers that might try to collect evidence 4chan style to support the whistleblower’s case.
While I would definitely entertain extreme tax tiers at high revenue limits (like $10B/yr), it is important to note that there is value in Apple making money.

There are a lot of jobs created between the $10B and $1T levels. And as citizens of Apple's home nation, it greatly benefits us that those high-quality jobs are available.

A 100% tax (or even 80%) tax would discourage a lot of products from existing. Apple would probably move down to simply offering just phones and apps only and removing everything else. I know that sounds good in theory, but there are indirect benefits for these other services that they offer, in addition to jobs. It also bolster's California's economy all the way down to bodegas, bars, restaurants, and hobby shops.

So an alternative tax like 60% on revenues above $50B that can not be deducted against might be a more reasonable alternative.
Immortality.
I found a comment that was factually incorrect and I felt competent to comment on that. Regrettably, I wrote just one sentence and clicked reply without providing any credentials to back up my claim. Not that I try to hide my identity, as danhak pointed out in https://news.ycombinator.com/item?id=28751644, my full name and URL of my personal website are only a click away.

I have replied to my initial comment with provide some additonal context: https://news.ycombinator.com/edit?id=28752431. Hope that helps.
> I kind of want to ask him about the discrepancy between his online persona vs real life, but I don't want to offend him.

Struggling people are allowed to have, and share, enjoyable moments. Going through a divorce doesn't mean someone must be miserable 100% of the time, nor that you have to hate your spouse. Nobody is obligated to share the negative parts of their relationships and lives online, even if they're struggling.

Having a perfectly happy and problem-free marriage isn't a prerequisite for posting a happy family photo online with your kids. Nor should it be!

I don't understand why this is even a question. Just support your friend. Don't try to poke fun at the one moment of joy he shared online or call him out for some perceived discrepancy just because you know his struggles. It's not hypocritical for someone to have fun while going through tough times. It's bizarre that you'd even consider asking someone about this.
What is Blind? Or shouldn't I ask?
The fact that we have no "free will" and that there's always a reason for our actions at the end of the day doesn't constitute useful advice for automata like us trying to do better in the world nonetheless.

Except maybe in how we think about justice perhaps. But that's not useful now.
tracert 129.134.30.12

Tracing route to a.ns.facebook.com [129.134.30.12]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms eehub.home [192.168.1.254]
2 3 ms 3 ms 3 ms 172.16.14.63
3 * 5 ms 3 ms 213.121.98.145
4 5 ms 3 ms 4 ms 213.121.98.144
5 17 ms 8 ms 18 ms 87.237.20.142
6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149]
7 * * * Request timed out.
8 * * * Request timed out.
9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102]
15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143]
18 91 ms 93 ms 90 ms 157.240.36.97
19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12]

Trace complete.

this is what i got now
You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated.

Why is it a requirement to be able to instantly revoke a person's badge access eeverywhere in real time anyway?
Location: Edinburgh, UK

Remote: Yes

Willing to relocate: Maybe

Technologies: React, Angular, TypeScript, JavaScript, Node.js, GraphQL, REST, HTML5, CSS3, Ruby, Ruby on Rails, Laravel, PHP, Perl, SQL

Résumé/CV: https://1drv.ms/b/s!AusItvVCO-zsgeUVj-eDReOaRjaT8g

Email: righter.barrows0z@icloud.com
At the time, there was no guidelines to use the mask. Actually, people still thought that masks would be worse than not using at the time. This was at the beginning of the pandemic.

This has to be an emotional decision as well, I am not a doctor, I have to follow my gut before there is data to analyze.
It sorta sounds like Facebook is Too Big To Fail.

Yet another reason to dismantle it.
And where did evangelicals get the idea to ban alcohol?

They didn't read in the Bible that alcohol was bad. They looked around at society and blamed a lot of its ills on problematic alcohol use.

Some background here: https://www.thegospelcoalition.org/article/9-things-you-shou...

Everybody gets this causal arrow strangely backwards. Alcohol causes people to sin; that's why Christians opposed its use. It's the negative effects of alcohol that led people to oppose its use, not "religion" per se (or even in any kind of explicit scriptural or liturgical sense).
Hey now, no need to be selective, let's knock them both out and get rid of the worst of people on both sides!
Yep, with FB and IG down, people spending time on a much more important site.... This site.