skysight-inc/HackerNews huggingface dataset:
tracert 129.134.30.12

Tracing route to a.ns.facebook.com [129.134.30.12]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms eehub.home [192.168.1.254]
2 3 ms 3 ms 3 ms 172.16.14.63
3 \* 5 ms 3 ms 213.121.98.145
4 5 ms 3 ms 4 ms 213.121.98.144
5 17 ms 8 ms 18 ms 87.237.20.142
6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149]
7 \* \* \* Request timed out.
8 \* \* \* Request timed out.
9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102]
15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143]
18 91 ms 93 ms 90 ms 157.240.36.97
19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12]

Trace complete.
Arguably someone working on automating farming systems would be reducing demand for labor in that work.

So really, "isn't this putting impoverished people out of the only job they can get?"
Honestly having buildings automatically lock down like this sounds like an absolute worst practice. FB headquarters is in california. What happens when the big one hits? server is killed and no one can get inside and look for survivors? Shouldn't need a functional network connection to be able to open the door of a building, but here we are I guess.
We also have RAM.

Edit: but we can't GC it fast enough. On a day like today I wonder if always going to disk mightn't be better.
For a similar setup, see https://usesthis.com/interviews/joey.hess/ and https://joeyh.name/offgrid/ .
I would rather purge it from this planet and throw Zuck into jail.
Researchers Discover Evidence of a Major Coronavirus Epidemic 20k Years Ago
Just checked it out - awesome tool - decent voice quality. Is it difficult to integrate this with Wordpress?
You can get through to a web server, but that web server uses DNS records or those routes to hit other services necessary to render the page. So the server you hit will also time out eventually and return a 500
Thanks! I just edited the parent's link and didn't notice.
Football game's are planned events, and the carriers plan capacity accordingly. I remember sms+cell phone going down on several music festivals with 20.000+ participants (especially when it ended at around midnight). Only some carriers supported those "sponteneous" gatherings in the middle of nowhere with mobile cell towers that would keep connectivity going - but low-cost carriers never did.
> I'm bothered by this comment.

I'm sorry you're bothered by my comment; it's simply an expression of how I feel.

> PICO-8 is one of the most vibrant communities in existence for beginners making games. Thousands of people have made thousands of games with this software....licensing allows people who purchase the engine to distribute it in their own games.

Windows is also very popular, and it is also proprietary software. It enables lots of other people's jobs to exist, but that doesn't mean that it's ethical to deprive its users of the source code to the software that's running on their machine.

>The price point to purchase is extremely cheap, supports the person that created it...

I don't have a problem with donating money to help out a project; I have a problem with the software not respecting the four freedoms (liberties) of free software. (https://www.gnu.org/philosophy/free-sw.en.html)

> It is one of the most open, empowering, user-supporting pieces of software...

It isn't open and it isn't user-supporting or empowering. Users don't have access to the source code, and in that way the developer holds power over the users. Bruce Perens, who really popularized the idea of 'open source' once said "When I say 'Open Source', I mean the same thing that Richard means when he says 'Free Software'."

> there are plenty of open-source projects that are actively user-hostile.

would you like to provide some specific examples?

> aren't saying that out of any interest for how that would make the project better or create a better community, but instead are saying it from a standpoint that only considers how it would benefit them personally.

A better community would be able to improve the software itself with access to the source code. Also, simply because it-being-libre-software would benefit me personally doesn't mean it wouldn't also benefit the rest of the community equally as much.
Came to say the same thing about 2020 date! The source of the document is very different to anything else FB produce. They are handling the layout with this js code for instance.

if (window.innerHeight < 80 || window.innerWidth < 80) {
// Blank if window is too small
document.body.style.display = 'none';
};
if (window.innerWidth < 200 || window.innerHeight < 150) {
document.getElementById('back-link').style.display = 'none';
document.getElementById('help-link').style.display = 'none';
};
if (window.innerWidth < 200) {
document.getElementById('sorry').style.fontSize = '16px';
};
if (window.innerWidth < 150) {
document.getElementById('promise').style.display = 'none';
};
if (window.innerHeight < 150) {
document.getElementById('sorry').style.margin = '4px 0 0 0';
document.getElementById('sorry').style.fontSize = '14px';
document.getElementById('promise').style.display = 'none';
};
Aren't we still communicating on HN, even though the possibly largest network is down? Can you send email?
The Facebook website is also partially back up. I'm seeing a few status updates, but I can't load new posts by scrolling and it just generally seems broken.

Edit: Must be intermittent, on refresh I'm getting the "Sorry, something went wrong." page again.
Yup, whatsapp is back for me.
I just got the login page. It was fun while it lasted.
I enjoyed the comment. You might like my short story along the same lines from a few years ago: https://bowaggoner.com/writeups/robust.html
Any idea what is the explanation for this?
DNS is back, looks like systems are still coming online.
Excellent opportunity to set up a DNS-by-mail service. Just send me a letter with the names you want and I’ll get back to you within 3 to 5 business days!
As they should. That said, USPS is using Visa gift cards. What a nice way to collect data and monetize it. But if no one sees that, or realizes it, it's less sketchy?
As an embedded linux developer, what Android brings is an unified UI(after all those x-windows variants plus Qt GUI). As long as the below list remains to be OSS, Android can be used in many non-realtime devices that need a GUI(with/without touchscreen):

* the linux kernel
* the ART(java jvm) which most GUI apps depending on
* base code to support the multimedia stack(video/audio)
* NDK(c/c++)

There are many devices can use Android for applications that have no need for google play store apps at all.
I thought Telegram offered e2e encryption.
That’s pretty much why during the downtime visit instagram.com showed a 503 from AWS instead.
just ran a dig A facebook.com, it's resolving and it wasn't a little while ago, obviously something is working again. same with insta and whatsapp
Still wasn't clear enough with my analogy! I was thinking more like a dam failure due to operator/designer error, not sabotage (but who knows). The damage is really small signs initially, followed by rolling catastrophic failures.
Facebook seems to be coming back up.

Which makes sense. Not much was down. Just DNS and BGP.

As soon as they get past the login transient, things should be OK.
See e.g. https://www.digwebinterface.com/?hostnames=facebook.com&ns=a... for responses from different nameservers.
Whatsapp is (and especially was) way better at sending photos than whatever SMS app installed on one's phone. It also used to work with crappier phones, for which there was no easy way (that I know of) of installing a SMS app that would send photos. Plus, you have Whatsapp private groups, which I don't know if you can emulate using SMS.
Okay, but as the value of the collateral approaches 0 your lender asks you to increase your collateral correct?
> nuclear war

I think you need some convincing to keep your SREs on-site in case of a nuclear war ;)
From article:
[This chart shows] the availability of the DNS name 'facebook.com' on Cloudflare's DNS resolver 1.1.1.1. It stopped being available at around 15:50 UTC and returned at 21:20 UTC.
The new phone books are here!!
I was thinking the same...
Coworking spaces are a thing, and - being tired from WFH - I am currently enjoying working in those. As much as I wish I could build something like this, this is a good enough option. The price is a bit steep in the large city where I live: 500 CAD for dedicated desk and about 1000 CAD for 1 person office. On the flip side you have all the niceties of the modern office, including espresso machines and sometimes beer and carbonated water on tap! Curious to see if all the extra office space from people not wanting to return to the office depresses the prices.
I literally built a color scheme doing this manually last week! Results here were basically as good as what I came up with.
Even if we accept that's true, it still makes no difference to the fact that this outage has harmed those people.
This person has a POTS line in their current location, and a modem, and the software stack to use it, and Instagram has POTS lines and modems and software that connect to their networks? Wow. How well do Instagram and their internal applications work over 56K?
Doesn't help when everyone just uses Gmail.
I don't know much about this but from the limited amount of I've read it is probably only a portion of the equity owned, and generally when borrowing against an asset the lender will not give you 100% of that assets value to protect from downside risk. Another probability is that it was adjusted in the past, potentially year(s) ago, and FB's stock price a year ago was almost $100 a share less than current so a $10 drop is not a big deal in the long term.

You raise an interesting question though and I'd like to know the answer as well!
I currently have a $150k margin loan, and it absolutely will lead to forced liquidation if the collateral drops in value: Interactive Brokers was clear about that.

Separately, my bank tried to sell me a "Pledged Asset Line of Credit" that would also have required the collateral to maintain a certain value or there would be forced liquidations.

Can you link an example of a bank or similar that lets you borrow against stock or options collateral without margin call risk?
Maybe I should have marked it as satire.

OTOH, produced out of carbon sucked out of the atmosphere by whichever (nuclear) means could make it almost carbon neutral!1!!
Discussed 4 months ago:

Metastable Failures in Distributed Systems - https://news.ycombinator.com/item?id=27506167 - June 2021 (11 comments)

...but on a day like today we dare not mark it as a dupe.
Can we nominate them for a Nobel Peace Prize?
That isn't access control, that is a an intern project. Those readers aren't wired to devices that could trap you in a burning building, they simply provide a rube goldberg solution to the traditional sign-in sheet.
They're suggesting employees should work 80hrs a week instead of 40 and they should fire the bottom 5-10%. The only note on compensation is maybe give them more stock options.

Sounds like a lovely place to work.
It's back as of approx 14:47 PST.
They wouldn’t need to bring the site down to do all that. That’s not how scaled infrastructure works.
Yeah as part of my job I often have to work with our DNS team to provision say a subdomain or get some domain verified. They’ve got like…three people…trying to service thousands of teams across the enterprise. I do not envy their job at all.
I love how instagram's error message is literally text in a body element "Oops, an error occured"
I worked at eBay many years ago. At the time, the process to get into the datacenter involved multiple ID checks by both humans and computers and multiple authentication factors. The process took about 10 minutes from walking in the front door to actually being in the cage.

One day we had a major outage. The first few engineers to arrive were started on the 10 minute process. As more engineers arrived, a line started to form out the door. Then the VP showed up. He told the poor security guard that if they didn't open all the doors right now, eBay would find a new datacenter (we were probably more than 1/2 the space at the time).

They used the emergency fire release to just unlock all the doors at once, and then called the fire department and told them to ignore the alarm. So now all the doors were open and if there were a fire no one would come.

That day, if you happen to know where the eBay datacenter was, you could just walk right in. The only check against it was that most of use who had DC access knew everyone else who did and could call out imposters, but you'd have to stop what you were doing and think about it first.

The point is, there is always a way to avoid security when needed.
I think that it is possible to restore the previous state but the question is, if it makes sense. When do you decide that it was a failure? Facebook explicitly (even so automatically) told all others that they shouldn't use that routes anymore.

When it comes to Facebooks side I guess they do have backups of their BGP config. Applying them (probably remotely) however seems to be harder then expected when the whole infrastructure is down.
"So basically everything I used to be able to get in a phone book."

Assuming you had phone books from every city/region in every country. Thats a lot of phone books so you must have had a large warehouse to store them all. Then there is the fact that phone books did not list number for every individual. Multiple persons routinely shared the same number.

The comparison sounds apt in theory but in practice it isn't. Try looking these Facebook users up in the phone books of their respective locales, via the telcos' online phone books or directory assistance. Then, using what you find, tell me their email address, gender and Facebook user ID.

Good luck.

The problem with this argument, "all email addresses are public", which I see regularly on HN, is that information does not become "public" and lose its "private" designation if it is published without consent or lawful purpose. If someone steals secrets and publishes them, they are still secrets.

Whether this information from Facebook is truly "private" I cannot say but I do think it is possible to have email addresses that are not made public.

The recent NSO iMessage story was interesting because the exploit seemed to rely on NSO getting lists of mobile phone numbers for the targets. Not email addresses. Yet iMessage will work without a phone number, with no SIM inserted. Perhaps the targets chose to use phone numbers for iMessage, not email addresses.

Consider what happens if someone creates a Gmail address but never uses it to send mail, and never shares the address with anyone, except Facebook. If this person does not make their Facebook profile public, how is this address public information. Google does not publish a list of every Gmail address. According to the logic of the parent comment, they might just as well. Email addresses are "public", right. Because some HN commenters think they are.

What happened when someone scraped Apple's servers to obtain the email addresses of Apple iPad users. Did federal prosecutors think the information was "public" or "private". The media called the incident "theft of e-mail addresses".^1

1. http://www.nbcnews.com/id/41196595
Yes, even Facebook falls prey to the wrong copyright year. Anyway, I got further now to a page that says "Account Temporarily Unavailable." and has the old Facebook layout. Would love a peek inside the Facebook codebase to see how this happens, hah!
FYI TFA is just a blank dark scrollable page without js here on FF w/noscript.
Cloudflare has a useful tool for measuring if your ISP is using RPKI.[0] For Facebook, this is the latest I could find for their implementation of BGP.[1][2]

[0] https://isbgpsafeyet.com/

[1] https://engineering.fb.com/2021/05/13/data-center-engineerin...

[2] https://www.usenix.org/conference/nsdi21/presentation/abhash...
Considering how much IRCops were paid back in the day (mostly zero as they were volunteers) and what a single senior engineer at FB makes, I'm sure you will find 3-4 people spread amongst the world willing to share this 250k+ salary amongst them.
The fun part of BGP is they apparently make a lot of use of it within their network, not just advertising routes externally.

https://engineering.fb.com/2021/05/13/data-center-engineerin...

(and yes, fb.com resolves)
facebook.com resolves again!

> ping facebook.com

PING facebook.com (31.13.83.36) 56(84) bytes of data.

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=1 ttl=54 time=12.2 ms

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=2 ttl=54 time=12.1 ms

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=3 ttl=54 time=11.7 ms

Can't yet traceroute to a.ns.facebook.com tho
lol, it's not dns (fb dns is back), it's not coincidence, it's not people coming to hn specifically to read about fb being down. it's the digital equivalent of the consumer goods shortages resulting from changed behavior during the lockdowns of 2020. the whole consumer/attention-economy internet is creaking under the load and it's kinda hilarious. https://downdetector.com/
That might be true for a house or a small scale loan, but once you are dealing with billions I doubt that’s the case. I assume that it works as follows: you have $1b in stock, Bank gives you $500m line of credit. If stock goes down enough they force a sale, but they only sell against what you have actually Utilized in your line of credit. If you are Mark Zuckerberg and worth more than $100 billion, you probably don’t have any issues. If you add up all of his houses and planes and cars it probably doesn’t add up to more than 1% of that. He’s fine.
It was for me too, but not for everyone at work, and it seems down again, so at best it's unstable.
www.teamblind.com

Enjoy.
It's actually the main reason I despise Apple for how it locks down iOS : it's not that I care specifically about that, but it creates cover for this kind of thing. The bar has been raised across the board now for what is tolerated.
Looks like someone found the light switch and turned everything back on!
> Reddit r/Sysadmin user that claims to be on the "Recovery Team"

They have time to make public posts, and think it's a good idea?

Sure, I'm on the 'Recovery Team' too! How about you?
When a backend engineer gets asked to do some frontend work...
I don't understand why archiving and linking source material isn't the norm.
this.

I'm a dev/tech blogger, and I can say, following the right people is half of the success.

Not just because of the things they say, but because they can share your work.

Every time my work was shared by an influencer in the field, I got one or multiple job/project offers and over the years I made a good living out of it.

"being good at tech" only helps in the next step, when you need to proof that you can write better about a tech topic than the average marketeer/content creator.
Facebook app, Instagram and Messenger are working for me.
> I think the the point of the anecdote is that what you see online in no way reflects people's actual struggles.

> Unfortunately this fact is not made explicit anywhere

People don't wear their struggles on their sleeve in person, either. People are allowed to have fun even while struggling elsewhere in their lives. There's no rule that people aren't allowed to thrive or be happy in public if they're struggling with something private.

I don't understand why anyone would think that social media would be any different.
At what point is this an occupational safety hazard akin to blocking a fire door? Ridiculous that something like access to a building requires a network at all, but leave it to technology companies to overengineer a solution and underthink about the implications.
They've had that 2020 for all of their outages this year. I was seeing it before it came back up partially, and now I'm getting a new error page design with 2021.
What does IAP stand for in this context?
Goodreads.com | Software Development Engineer / Senior Software Development Engineer | ONSITE San Francisco, California / Seattle, Washington / Arlington, Virginia | Full-time

Goodreads.com is the worlds largest community of readers and authors. Join Goodreads to solve problems at a massive scale and serve customers all over the world. Interested in high throughput web development? Backend tools used by other teams? Networking or Security? Goodreads is a chance to do it all.

Apply here - https://www.goodreads.com/jobs#openPositions
Agreed, website loading here, still no whatsapp though.
Facebook Is Back Online
We kind of got off topic, I was arguing that if you were concerned about internal systems being down (including your monitoring/alerting) something like pager duty would be fine as a backup. Even at huge scale that backup doesn’t need to watch everything.

I don’t think it’s particularly relevant to this issue with fb. I suspect they didn’t need a monitoring system to know things were going badly.
Aaaand this is why you host status pages on completely distinct infra, including domain names.
Software related conspiracy theories are the worst
Not necessarily. Depending whether internal systems have been affected, say if their internal DNS went down along with these, many other things might be affected.

Applications and appliances that have exponential backoffs might be minutes or hours away from a recovery without manual intervention; other things might be in a broken state for having given up e.g. mounting a network drive, etc.

Failures, especially on such fundamental networking technologies like DNS and routing, and especially those that take a long time to resolve, have a tendency to cascade into catastrophic failues in many other systems.
Facebook Free Basics was quickly blocked in India https://www.reuters.com/article/us-india-regulator-netneutra...
AFAIK the whistleblower already provided all the internal reports to the SEC and possibly the media. While a coverup is not impossible it would be incredibly high legal risk to put on employees. Also, such a massive BGP outage will draw a lot of attention and investigations into their networks and calls from investors. This has already impacted their stock value.

If I were them and planned to cover things up, I would do it quietly and discretely using someone with the highest privs possible and that knew how to cover their tracks. It is highly unlikely anyone would know the data is gone until someone went looking for it.

So no, I don't think this is a likely scenario.
Many of these businesses are small single-person operations with little tech understanding that are using, for example, FB marketplace. They wouldn't even know what you meant by "single point of failure."
Not for a second would I have believed they actually delete information. (which is why I've avoided them). Even this site can leak at any time.

This whole industry is meant to harvest user information. Richard Stallman tried to warn us. Jaron Lanier called it "siren servers", because they hypnotized people into using their services. I hope this encourages people to question this and and to look for alternatives.
Here's a reason that seems obvious to me: Cults of personality are dangerous, for cult members and for the community. Among other things, they are delusional; they aren't reality. Also, they give far too much power to the personality.

Why has that obvious knowledge been forgotten? Have we forgotten to be skeptical?