skysight-inc/HackerNews huggingface dataset:
I think this paper is super important, and anybody who designs or runs big systems should read it and take the core point to heart. As system designers, we're very used to thinking about systems as 'stable' and 'unstable', where stability is good, and instability is bad. What this paper points out is that many kinds of distributed systems have multiple 'stable' modes, some of which are modes where the system is stable (in a control theory sense), but not doing any useful work from the client's perspective. This is dangerous, because the system won't kick itself out of this "stable but down" mode without something changing: human input, a control plane taking action, etc.

I don't think this paper covers anything particularly new, but writing it down in this form, with the evidence they present, is very valuable. Hopefully this paper will deepen the conversation about applying control theory to distributed systems design and control problems, and allow a more theoretical approach to be taken to the design of these systems to avoid common causes of instability and bistability.

One of the authors has a great summary of the paper on his blog: http://charap.co/metastable-failures-in-distributed-systems/

I wrote a summary and discussion too: https://brooker.co.za/blog/2021/05/24/metastable.html
Facebook NSDI '21 – Running BGP in Data Centers at Scale
Previous discussions:

https://news.ycombinator.com/item?id=26580568 (March 25, 2021 — 1 points, 2 comments)

https://news.ycombinator.com/item?id=10344601 (October 7, 2015 — 132 points, 108 comments)
Looks like DNS resolution is back now, but the site's still down.
Locking the doors to shred the evidence!

/s But seriously, such odd timing with Pandora Papers and with the Whistleblower.
She said she wanted FB to do something to stop misinformation and hate speech but what we've seen from Reddit is that "are mRNA vaccines actually safe?" becomes misinformation and "we shouldn't perform elective life-altering surgery on pre-teen children" becomes hate speech. There's not much I applaud Facebook for, but not listening to this woman is one of the few I do.
I'm one of those who had no idea about the whistleblower story, but I learned of it through reading about Facebook network outage.
DSW 2021: Blockchains and Building Dapps with Solidity on Ethereum
So then when I'm on some kind of blocked WiFi and nothing resolves, and I switch to a properly working WiFi your code will continue to fail?

It's not so simple to cache misses - you don't know if it's a real miss or some kind of error.

For example if Facebook cached the miss, then even when they are back up nothing would connect.
Strangely, https://research.fb.com/ is UP.
> revoke key digital records that tell computers and other Internet-enabled devices how to find these destinations online.

[Ed note: our readers can't be relied up on to know DNS is, but let's see what they make of this techno-babble.]
Looking forward to HN dismissing NFTs for the next decade while they continue to grow and gain usefulness.
I have a hunch that the "How BGP can break the Internet" will get updated in the near future :^)
No one.

Cloudflare that write a report of some else network outage!

AWESOW COMPANY!!!
Thanks, great write up!
`These spirits may make you willing, but your body will be weak'.
This info had a logical purpose also. Phone books without location and address would be mostly useless because lots of people share the same name.

To find the phone number of a friend having just "John Smith, USA" would be impossible (or would annoy thousands of people and require months of calls).
Simply because the average reader of *Hacker News* knows more about the tech industry than the average user of social media platforms.
Amazon is starting Black Friday deals early. Like, now
From your linked SO post, the accepted answer concludes:

"In summary, SERVFAIL is unlikely to be cached, but even if cached, it'll be at most a double- or even a single-digit number of seconds."

That would be fatal right now, wouldn't it? That would mean every major ISP's DNS server right now forwards millions of identical DNS resolve requests to the (currently null-routed) Facebook DNS servers. These must be millions, as heck, every larger website uses FB tracking tools, "like buttons" etc. Are they at least smart enough to throttle based on a domain/ip hash? Else it could happen that DNS servers of major ISP are soon overloaded as (constantly failing and thus uncached) requests to FB DNS would eat up all bandwidth/ressources?
Hmm... I'm always reminded of my professor telling me that it's never the fault of who pressed the button, responsibility lay upon who decided to make them able to press a button that can cause such catastrophic issues.

Somebody from my engineering class had an internship at DuPonts main facility/production line. Was implementing something that managed to complety shut down production for an entire shift & cause a large fire, ended up being something in the millions worth of damages from production loss and fire damage.

Intern wasn't even yelled at IIRC. He actually went on to do some very helpful things the rest of the internship. But man, did the person who let an intern be in the position to single handedly cause such a mess get absolutely fucked by his superiors.
I don't disagree with the article at all, I think it's conclusions are probably largely right.

But it's hard to take seriously when the very first figure has a logarithmic y-axis without any callout in the discussion, exaggerating the appearance of the negative effects.
100,000 sketchy payday loaners and check cashers gasp to breath...
Logically the USPS will have a handshake set-up process...
Which sucks because SMS is an increasing attack vector in current times and iMessage had a fun pile of zero days this year.

Also US based here luckily everyone I talk to is spread across signal/telegram/discord/googlechat for the most part.
Because only people with a high enough conviction of benefits of moderate drinking are commenting here and arguing for. However I have no idea how they do become so strongly convinced in that, given the low quality of the data that exists, the general reproducibility crisis in bio-sciences, precedent of complete compromise of a large study, things like https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-..., the argument that if moderate amounts of ethanol were beneficial we would have just evolved to synthesize it, etc.

I think the most likely outcome is boring: moderate amounts of alcohol are moderately harmful, but not harmful enough to be immensely obvious and not harmful enough to outweigh the non-biological benefits some people extract from it.
Thanks! That saved me. I was trying to decipher this acronym based on the context where it was mentioned with no luck.
There is already a nice writeup on the current incident from Cloudflare at https://blog.cloudflare.com/october-2021-facebook-outage/

They key observations:

"Due to Facebook stopping announcing their DNS prefix routes through BGP, our and everyone else's DNS resolvers had no way to connect to their nameservers. Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.

But that's not all. Now human behavior and application logic kicks in and causes another exponential effect. A tsunami of additional DNS traffic follows.

This happened in part because apps won't accept an error for an answer and start retrying, sometimes aggressively, and in part because end-users also won't take an error for an answer and start reloading the pages, or killing and relaunching their apps, sometimes also aggressively."
Telegram has the great advantage that you can have it working on multiple devices at the same time. Whatsapp's web and desktop solutions are terrible.

On the other hand, voice calls on Whatsapp are far better.
Its alive!

drill @1.1.1.1 www.facebook.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 2172
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; www.facebook.com. IN A

;; ANSWER SECTION:
www.facebook.com. 3401 IN CNAME star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 3403 IN A 31.13.72.36
It sounds like the perfect time honestly. If you fuck something up, you have the whole week to fix it.
pings to a.ns.facebook.com are no longer timing out
I think this is great, I just really hope you will be okay when Discord builds this internally.
Well, everything is bit slow for me. I'm in the UK on Virgin Media, using either Google DNS or the VM ones (I'm not sure and can't be bothered to look).

What has just happened, and it can't be coincidence, is that I lost internet connectivity about 1 hour ago, and had to reboot my Cable Modem to get it back.
Decent list and a nice way to educate whilst pitching your product.

A lot of these points are applicable outside of Fintech. If you have any type of data that if leaked could ruin your business then it's worth a look over.
Well, You cant access a city if the freeway has been bombed...

Remember the 'information super-highway'? Yeah it gets carpet bombed constantly....
It's fun that you should mention that, because assuming it is true that webview is best upgraded out-of-aosp (which I guess I kinda agree with), then why does AOSP ship with a webview?

AOSP is lacking a browser, not a webview.
>Done wrong, it would extend the FB outage longer.

Let's hope it's done wrong.
Twitter, Google, Apple and Amazon are not equivalent to Facebook.

Facebook is 100x as malevolent as those four combined.
Sure. As long as you plan for disaster.

The place where I worked had failure trees for every critical app and service. The goal for incident management was to triage and have an initial escalation for the right group within 15 minutes. When I left they were like 96% on target overall and 100% for infrastructure.
Spam Phonecalls in Canada Inhibit Answering
I love that when you had to think of a random improbable event, you thought of a cocaine meteor. But ... hell YES!
The feds will just subpoena Google or the app author.
Yep, they were called Reverse Pages. You would buy the CD, and then just read the database yourself. This allowed you to type an address and you could get their phone number etc
I really like that take on this topic, my personal story about it was at the university.

While studying older students would start talking, how they get a good grade/easy questions because they gave cookies/whatever or how some prof was super cool for them in some way.

Of course thing was they probably were in a group that prof liked, they had a good connection, maybe someone from the group had good jokes that made that prof laugh. Maybe they were showing up with whole group to his classes and no one made a fuss or other silly stuff at his lectures.

Then freshmen would pick up the story and would try to "trick their way" with the prof that would of course backfire as they totally missed the context of why for the older guys prof was happy with some cookies to ask them "easy" questions.

Just being in shape without any context - like how much in shape - might backfire when some girls think you are a buff so probably aggressive and get scared instead of interested. So just trying to talk up a girl on the street while buffed might scare her much more.

I would not focus on "natural skill" but I would focus on understanding that there is infinite amount of details to life and whole context is missing if someone gives high level advice.
I imagine the authentication systems for the datacenters are more rigorous than most Facebook buildings, right?
While this is a massive inconvenience, I don't see how messaging apps like this are a government problem if they go offline. These are not state run businesses.
In volume.
It seems to be a side-effect of Facebook being down, attracting so much attention to HN that it might as well be a DDoS attack.

Sorry everyone—there are performance improvements in the works which I hope will make a big difference, but it's not something we can roll out today.

p.s. You can read HN in read-only mode if you log out. Should be way faster.
Yep, they were called Reverse Grey Pages. You would buy the CD, and then just read the database yourself. This allowed you to type an address and you could get their phone number etc
Mythia | Android Developer | Remote or San Francisco | Full Time | https://www.mythia.com/

Salary: $60k to $110k USD | Equity: 0.25% - 1%

Mythia is a rewards card for gamers. Instead of airline miles, users get free PlayStations and video games. Users can team up with friends and tackle weekly challenges for more rewards.

Our leadership team is highly technical. We have 90%+ unit test coverage, well documented APIs, a smooth CI/CD process, and 1 week sprints.

You will be fully responsible for our Android app. This is a unique opportunity to own a mission critical part of a fast growing startup.

We're funded by Y Combinator, as well as the founders of Tinder, Nerd Wallet, Scale.ai, and more.

Interested? Email us with "HackerNews" in the subject line: careers@mythia.com
That's a spot on summary.
Due to DNS being busted, all internal FB services/tooling that they'd use to push DNS config updates are probably completely inaccessible. Someone at FB will have to manually SSH into a production host (assuming they can even identify the right one), and issue some commands to repopulate the DNS records. They'll probably have to do this without any access to internal wikis, documentation, or code.

Keeping those poor network engineers in our thoughts.
> I submit if you need to qualify your use of the word "laptop" with fully eight adjectives or adjectival phrases to avoid ambiguity with pre-existing products, that it's probably not "revolutionary" by definition.

I count five: It's the first "widely popular", "fully-functional", "powerful", "general purpose" ARM-based laptop ever being "produced on a massive, industrial scale"

My counterpoint: it's revolutionary because it's the first ARM-based laptop which doesn't have to make a ton of compromises to make it out the door.

It's got mass-market appeal ("widely popular", unlike the PineBook Pro).

It's not limited in functional scope ("fully functional", unlike Chromebooks).

It's not limited in computing power ("powerful", unlike any other ARM laptop).

It's not restricted to a subset of tasks ("general-purpose", unlike the iPad Pro when treated as a laptop).

It's being mass-produced for retail, i.e. it's not a limited-run or a prototype ("produced on a massive, industrial scale").

Every other attempt at an ARM laptop has made one or more of these compromises; the M1 Macbooks (and the M1 Mini) don't have any of these compromises, meaning that it's fit-for-purpose for the vast majority of laptop users (those for whom a Macbook would have sufficed before the M1 line).

I think being good and useful is pretty revolutionary; I haven't seen another ARM laptop offer that, and those are pretty important features.
How can I prove that someone has intent of doing something? You can take a look at the news revenue over the years and the efforts for getting paid for displaying headline in Google.
Never having explicitly queried fb.com before I never noticed how they (face:b00c) got clever with their IPv6 address:

2a03:2880:f1ff:83:face:b00c:0:25de
Can you explain/rephrase? Are you just talking about surveillance in particular? If so, I apologize for not being clearer... I meant heavy-handed as in forced disappearances, labor camps, censorship, behavioral modification, etc. (or in the US, forced reproduction/sterilization, manipulation of educational curricula, etc.)

China's recent attempts to curb the influences of their Big Tech and online gaming sectors come to mind, vs the fuck-all we've done with ours over the past couple decades. Soon Facebook is going to become a supranational organization that can drive new laws just by enraging enough people through algorithmic manipulation... our watchdogs are whimpering puppies against that kind of power. Our government is way weaker in terms of its ability to regulate business or personal behavior.

And for the common person, sure, our government might know everything we're doing (they all do, these days), but by and large it does not really care. The data it collects is often so disorganized even its own agencies don't know how to share it with each other, much less use it to systematically oppress -- for now. Our discourse and dissent is THRIVING, free speech is alive and well, and we have so much freedom we've self-organized into alternate reality bubbles, absent state guidance and with a deliberate disregard of expert opinion. China doesn't allow its society to fracture like that. Ours has no choice but to allow it to happen, and our elites encourage that sort of fragmentation because it makes for easier power-mongering at the top when the commoners are divided against each other.

The Chinese are oppressed by a heavy-handed, paternal government. Americans are oppressed by a government so weak that capital, charisma, and convenience govern our society, not our supposed laws or values. Other developed countries, democratic or not, tend to fall in between those extremes, from the UK & Australia closer to us (weaker gov) to the Canada & EU (stronger govs) to the Nordic and Asian democracies (stronger yet), yet China's is way way way stronger than all of those.
Yeah, I've heard this view a hundred times on Twitter, and I wish it were true.

But sadly, there is no company which doesn't rely, at least at one point or another, on a human being typing an arbitrary command or value into a box.

You're really coming up against P=NP here. If you can build a system which can auto-validate or auto-generate everything, then that system doesn't really need humans to run at all. We just haven't reached that point yet.

Edit: Sorry, I just realised my wording might imply that P does actually equal NP. I have not in fact made that discovery. I meant it loosely to refer to the problem, and to suggest that auto-validating these things is at least not much harder than auto-executing them.
I strongly dislike how we are forced into centralizing our online life into a few big corporations. Therefore, it is somewhat nice to read that even the access cards don’t work at Facebook HQ due to them running everything via the Facebook domains.

Still, let‘s hope that this gets fixed soon for the engineers and users involved
Google, Apple and Amazon, maybe. Twitter actively participates in and profits from "cancel" culture so they may as well be direct political rivals to the West.
Right, which is why you just roll in protection for all manner of config changes by taking pains to ensure there are always whitelists, local users, etc. with secure(ly stored) credentials available for use if something goes wrong; rather than assuming your config changes will be perfect.
> two of the most used messaging apps inn the world fall at the same time

Text messaging works fine. There is no serious disruption of public service.
Computer-Based System Safety Essential Reading List
My first job was working in data centers for telecos and my impression was that everything was one cable trip away from never working again.

Networks were really complex, nothing was documented nor deployed as code, most equipment were untestable black boxes, people who deployed stuff moved on, etc... Just thinking about working with on prem again gives me chills.

But maybe FB knows better than telecos, IDK.
Python 3.10 Is Available
Do you really think that Facebook is going to write one right now?

It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.
You can try https://dnsleaktest.com/ which shows which DNS server is actually used.
How much revenue could they have lost
As someone who used 768p 15" laptop until late 2019, yes, everyone else has to deal with bs huge menus. Oh the number of webpages that would have you scroll horizontally, often with white bar on both sides... The day I upgraded to 1080p I could feel a sense of joy in being able to see the web as it was designed. And now every web developer is beginning to develop on 27" (or less) 4K displays and the target is shifting for less fortunate yet again.
The ducks are fantastic !
Rent The Runway S-1 IPO
“Apple, which uses Chrome as the base for its Safari browser, …”

Hrmmm. That doesn’t sound right.
Creating a Botnet of IPTVs to Rickroll over 10000 High School Students
I wish I could look at this as a joke...haha
Absolutely, and I'd even call it a rite of passage to lock yourself out in some way, having worked in a couple of DCs for three years. Low-level tooling like iLO/iDRAC can sure help out with those, but is often ignored or too heavily abstracted away.
No worries. You would never make it that far. You would not ever take ownership of the domain, so Facebook would never need to "retake ownership".

This is like going to a garage sale and someone tells you that they can broker a deal for the original Mona Lisa for for just $59 plus a 10% commission. But when the broker calls the owner of the Mona Lisa they inform you that its not for sale.

In this case, the registrar is trying to sell its domain brokering services. When you search a domain they either will sell you the domain, or if the domain is not available, they will sell you a service where they try to broker the domain for you. Now of course when they call up Facebook to see if they want to sell the domain, Facebook will just say no and the conversation will be over. You are out $59, which was the flat fee to initiate the brokering.
At some point it's time to start trimming your resume. I mean, unless you're applying for your first real job or something, no one cares about how you worked at McDonalds when you were in high school (I sure didn't, when I was reading resumes... if they're that anal, you probably don't want to work there anyway).

10 years is Long Enough, especially in this field.

I'd recommend labeling the section "Recent work history" or maybe "Relevant work history" and then ending it with "Further work history furnished upon request".

The odds of them asking for "further work history" are very small (unless the company is completely dominated by HR... which, of course, some are).

If none of your jobs before this one were in software, none of them are really relevant, or so I see it.
This is an example of the "special friend" technique harmful platforms make use of (slide 40):

https://www.slideshare.net/danctheduck/gdc-2011-game-of-plat...

"First you sign long-term contracts with certain influential developers"
whatsapp uses phone numbers, so if you can do whatsapp can't you just do sms?
You mistake Facebook with Twitter. Twitter dying will be a boon to humanity
That’s…not what “appeal to authority” means.
The Facebook Status Page is up again
Hinduism/Buddhism/Sikhism has no role for alcohol as far as I am aware.
Are you fucking kidding me?

We even had a site and operation for a long while called:

"NOC MONKEY .DOT ORG"

We called all of ourselves NOC MONKEYS. [[Remote Hands]]

Yeah, that was a term used widely.

I'm 46. I assume you are < #

---

Where were you in 1997 building out the very first XML implementations to replace EDI from AS400s to FTP EDI file retrievals via some of the first Linux FTP servers based in SV?

I was there? Remember LinuxCare?
Maybe FOMO, misplaced trust in authority, or nudge theory?
https://en.m.wikipedia.org/wiki/Nudge_theory

Definitely know the overall effect you're talking about by putting all the setting switches together in a low-context roll, and especially the use of "reward" colors (green!) for flipping a switch on. One of the "life hacks" that gets thrown around for spending less time on your phone is to go into the settings and replace the screen colors with a monochrome color palette.
Introduction to NIS, the Network Information Service (2006)
Why the article claims the change originated at Facebook? Updates to BGP routing are not authenticated. BGP hijacking is a real thing. To the best of my understanding, other well-positioned AS could publish this evil update to BGP routing tables.