skysight-inc/HackerNews huggingface dataset:
One issue that should have been mitigated at least by Fastly; worse if the client has to do it.

They proudly stated this from their own website to their customers:

> "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime."

If that isn't one huge lie, I don't know what is.
Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it.

People need to be blamed, and responsibility for actions taken (without covering asses)
A big missing feature in Caddy for me is an embedded language like Lua for nginx so you can write tiny hooks. The Caddy authors have indicated on HN a while ago that Caddy 2 may have an embedded scripting language but I can't find anything about it in their docs.
"By the account of them and us being completely vaporized"
We use Fastly (and our site is down too) but I asked them about this a couple of years ago.
It is deliberate.
They said it was so they can tell if it is their Varnish service or the customer's Varnish service that is down
Web Browsers should probably retry a different server in DNS if they get a 503 - but they don't.
Fastly is a Content Distribution Network (CDN).

Basically the closer the server serving the webpage is to the end user the faster it is for the end user to see and interact with.

But running servers all over the world 1) isn't efficient 2) costs a lot of money.

So a few companies (fastly, cloud flare, akamai) figured, hey, why don't we build a bunch of small data centers all over the world and then provide a distributed way to serve web traffic from it.

It originally was brought about for services like Netflix, but has expanded greatly.

You still host your servers, but a copy of the webpage/media is given to the CDN to serve to customers.
Alcohol and tobacco are legal in many countries and yet you still get counterfeits and illegal production.

Also drug use is often not down to that user having a fair happy reality and oh so often the product of bigger issues that go untackled and addressing those social injustices would do far more to address crime overall than just legalising drugs.

Now if they legalised drugs and used that tax income to address those social issues, then we would see progress and more so, some fairness restored.
FWIW, Fastly ~8 hours ago (3am UTC) reported another incident: https://status.fastly.com/incidents/1glxxb8sf2zv and deployed a fix—either the fix made it worse or wasn't sufficient to mitigate the problem.
Depends if the legal version ends up really expensive, compare cigarettes which are still smuggled because of the sin taxes.
Perhaps Fastly is simply taking their commitment to reducing CO2 seriously? Three hurrays for the climate!
Not true, there is an exception for income under 80k/year. Over that person can operate via LLC with dividends etc. Most people renounce because it is difficult and frustrating to deal with US tax system.
Worrying that this is impacting so many dev toolchains and services, which will hinder the ability to respond to the issue.
According to the Wikipedia page, it is intentional:

https://en.wikipedia.org/wiki/Guru_Meditation

Or did one of you already edit the Wikipedia page to reflect this discussion on hn?
I gave it about 10 tries, and it seems a very small percentage of transactions do go through.

A decent number of tries is rejected right at the Varnish front door:

< HTTP/2 503
< server: Varnish
< retry-after: 0
< date: Tue, 08 Jun 2021 10:11:41 GMT
< x-varnish: 271470009
< via: 1.1 varnish
< fastly-debug-path: (D cache-bma1666-BMA 1623147101)
< fastly-debug-ttl: (M cache-bma1666-BMA - - -)
< content-length: 450
<
Service Unavailable
Guru Mediation:
Details: cache-bma1666-BMA 1623147101 271470009

Many more reach some backend system that just dumps "connection failure":

< HTTP/2 502
< content-type: text/plain; charset=utf-8
< content-length: 18
<
connection failure

And a tiny few do get through:

< HTTP/2 200
< content-type: text/html; charset=UTF-8
< cache-control: max-age=0, must-revalidate
< date: Tue, 08 Jun 2021 10:11:43 GMT
< via: 1.1 varnish
< vary: accept-encoding
< set-cookie: ...snip...
< server: snooserv
< content-length: 275036
<
<!doctype html><html>...snip...
Isn't it Muphry's Law? Or was that another example :)

https://en.wikipedia.org/wiki/Muphry%27s_law
I can't help but imagine that what you're describing are the criminal gangs we know about; the ones which are well documented. If there are criminal gangs which we don't know about, that aren’t well documented, perhaps they're better at maintaining long term stability.
Quick question if the cdns are down why cant traffic be routed to the web servers the central web servers the company owns ?

I thought cdns had fallback configured ?
Yeah, that's my experience as well. I thought it meant "we have no idea what's going on" though.
Not that the BBC are gloating that they're still up
Between macs, pcs, Androids and chromebooks this is pretty much the worst possible set up you can have for android develoment. Well, expect coding on your watch i guess.

Even an android phone would be better with naive tools like AIDE and tmux.
Not equivalent exactly, but the fediverse or IRC have been fun places for me to chat about some hobbies. gemini is another option. I think people write gemlogs on there and then others write some in response. I only use gemini a little bit, and just as a reader.

Something like IRC for chatting combined with a wiki to manage info related to the hobby could work well.

You could also treat Reddit like reading a book or Wikipedia and choose not to contribute. Go there to learn something, bring the info back to your own space you're building that has people you'd rather be around.
Global CDN Disruption Incident Report for Fastly
Normally you configure your a record to point at the cdn as the cdn is the thing that gives you multiple points of failure (caches all over the world). Hard to have a fallback to that. Running multiple cdns would be extremely expensive. Cdn caches are kept useful by traffic running through them, so hard to have a backup for that too.
Have two different CDN partners, own your own DNS, and then withdraw one of the CDNs if they are down. Suspect that's what Amazon have done.
Related question: is it legal to rehost another site’s links ?

My idea is basically a Reddit-but…

So it would work great with the same exact content as Reddit itself.
They reset the setting, regularly just to piss off people who only want the old frames.
Believe it or not a lot of people in Texas still live a rural lifestyle! There’s a lot more going on in the land outside the major cities than suburbs.
Holy smokes these Wikipedia writers are quick! I'm sometimes impressed by how fast a page on a super recent happening gets populated with all of the currently known details.
Yes I quit social media a couple of years ago and agree with pretty much everything the author says. Social media environments encourage incendiary discussion because it fuels upvotes and general attention. Most people won't take the time to consider a more valid grey opinion because they've jumped to a conclusion and are busy attacking someone, or because it won't get as much attention. The first thing that came back for me after quitting social media was that "slow thinking" brain that tries to empathize with all sides rather than just call out some "evil" and check out intellectually.
Makes me wonder how the engineers will fix this if they can’t visit Stack Overflow :)
No way that would be manipulated and abused by the politicians and the wealthy. I think at some point we have to accept that people are going to game any system (on all sides and angles). Resource competition is the base problem.
The most popular websites are the ones that need to use CDNs. There's about 4 CDNs of note, so when one is down there's a massive knockon.

At work we have fastly, cloudflare, and some others. Fastly has been removed from the pool so we're back up. Fortunatly (or rather thanks to planning) we didn't rely on a cloud service to do that though (say a password stored in cloud based bitwarden)
I'm pretty sure you can serve hundreds if not thousands of users from a single Raspberry Pi
The question I have is why didn’t people hold this opinion before COVID-19?
I wonder why they don't have USB sticks ready to boot from...
also, unrelated but images on hulu.com are not loading and reader.one is showing errors to all reddit links
Me too, but in a way I'm even happier knowing that not everyone does and something else popular exists too.
Content-centric networking had been a central research topic for many years. And many potentially useful systems have been proposed and implemented.

At some point some of them will start to become popular.
apache repos are not working tho
Github is working fine for me in Canada but others aren't. Tried without browser cache too and it works okay.

EDIT: Most sites seem fixed now here in Canada. Tested stackoverflow, reddit, GitHub, PayPal, gov.UK and all worked fine.
Still getting broken assets from the UK.
Not an American here but this could be related to the order in which things pay tax.

If you receive 10K USD as income one month, you would pay tax on that and then be able to buy things

If your business received 10K USD it could buy all sorts of things before declaring a profit and paying tax only on the profit.
Now imagine how many sites would go down if it was CF
Their CSS and JS were down for a few minutes. I was able to login to Amazon but the entire site was in Times New Roman, but was fixed a few minutes later
I imagine they use a few different CDNs for things like this.
> It's neocolonialism when the interest of American companies are backed by the full force of the American government at the expense of the desires of the local people.

I don't mean to defend the US, but if you replacing "American" with "China" in that statement is basically the Chinese policy in a nutshell.
There is some serious damage control going on. Numerous websites are down for me, yet hacker news has LOTS of posts about it as flagged when clicking on the "new"section.
XDA seems like an obvious choice to me for android dev, but maybe you know of it and don't like it.
Extremely disappointed to see Propublica (typically a place of important journalism) resort to clickbait headlines and disingenuous charts like this.

Here’s the TL;DR of main reason why Bezos, Musk, etc don’t pay taxes:

Unrealized capital gains.

This is not some evil nefarious “tax avoidance” scheme. When you create a business, and the entire world starts wanting to buy your products, and the business becomes extremely valuable (sometimes temporarily since the market is arguably in bubble territory)...AND you don’t sell any of that ownership stake, no, you obviously don’t have to pay taxes on that theoretical increase in wealth.

You haven’t sold, thus the gains aren’t even real. Tesla stock could drop by 90% next week and so would Musk’s wealth.

The fact that propublica is including unrealized gains in these charts completely undermines the reputation they once had for me.
Last I looked "age" did not have any sort of recovery utility. It isn't even clear that such a utility is possible (the protocol is poorly documented). A single bit error at the start of the file causes the loss of the entire file. So be careful using it for any sort of thing that might ever require the sort of recovery that bzip2 or lzip provides (gzip has a third party recovery utility).

OpenPGP has excellent recovery properties out of the box BTW...
At least now people will be aware that it functions as a tracking device even when powered off.
I get the feeling Cloudflare is bigger, might just be me.
The use of text for popular protocols is for a reason — computers don’t write programs, people do. And while CPUs prefer binary, it’s easier for programmers to read/write/reason about text. This makes it easier to work with a new protocol.

From a practical perspective, with a binary protocol, it can be difficult to use across different languages or add support for a new language. If you use the simplest possible encoding, you’d send raw struct data. But this doesn’t always work across different OS/arch/versions/etc. if the server is in C, but the client is in Python, reading the binary protocol would require a far more complicated parser.

Obviously a more formal encoding (protobuf, etc) would be preferred, but if you already need to use an encoding mechanism, why not wrap it in a text format? It’s easier to write clients that can read/write text protocols in any language. The reason why text protocols are so popular aren’t because they are necessarily “better” but easier to adopt. This is why the most popular protocols are text based.
I was going to link the appropriate XKCD where organised attackers are panicing as they realise they're dealing with a sysadmin muttering about uptime..

.. but of course XKCD is down too.

e: https://xkcd.com/705/
For sure, similar to other industries all changes come after big troubles like this. But would be interesting to heard about how them (paypal) deal with that
While this is true, I managed to bypass the Facebook login requirement through a workaround and the Quest 2 has apparently recently been rooted. The method has yet to be made public yet but it's only a matter of time.
how is private companies choosing to use a common supplier in a competitive market centralization? monopolies are not centralization either. you need to read a better book.
They will use S3, but they need a CDN in front. Surprised they don't use CloudFront - maybe that's what they've failed over to.
How does it strip it? Test:

Edit: You are right. It got rid of the emoji after Test.
Conor McGregor Net Worth: Latest
The funny part is that it isn't uncommon for sites to depend on both cloudflare and fastly in one way or another, due to buying services from saas companies that also depend on them.
Rhetorically, yes he/she/they do get to do that.

Ethics is a matter of philosophy, which has a bit more going for it than just being composed of raw, uneducated opinion.
Amazon doesn’t enforce dogfooding, their retail site is its own stack and has been only recently been migrating to EC2s
Buy morphine hcl 10mg in USA|usapropeciameds.com
Morphine is a pain medication of the opiate family that is found naturally in a dark brown, resinous from the poppy plant (Papaver somniferum). It can be taken orally or injected; it is also often smoked. It acts directly on the central nervous system (CNS) to increase feelings of pleasure and warm relaxation and reduce pain, and is often abused for this purpose.
Is the fact of looking at those links is like looking at a road accident with insistence instead of just passing by?
Because interacting and switching between cdns can be very complicated and/or costly

It should be planned for, especially by major tech organizations like reddit, or Amazon, etc.

But I won't fault news organizations, who already don't have boatloads of money for not having fail over cdns
None of the ES/NQ/RTY/YM futures contracts took kindly to the outage! This could have had a much wider financial impact. Most seem to have recovered now.
AWS don't report outages until it's undoubtedly them.
As i computer i find your trust in technology amusing.
A Network Error Occurred
To save everyone else hitting the site as well:

As of 10:44UTC, this status page has just updated to say the issue has been identified and a fix is being implemented.
Thanks. That makes sense.

Wouldn’t you build in a failsafe that bypasses Fastly and sends traffic to your own servers in the case of this kind of outage? Or outages are so rare that it’s not worth the trouble?
BBC: Huge parts of internet currently offline
A VLC plugin or somesuch would be nice
I think this may be because we've had much higher uptake as far as I know, so getting down the age ranges has been slower (by which I mean, yes, maybe the US has made it available to all adults, but how many (as a proportion) have taken it up)
In place of a legal notice, here is a blessing
Things seem to have come back online in Australia, although not sure if that's just sites switching over their DNS?
Perfect time for the crypto whales to dump massively and cause an absolute panic.
It took me a while to figure out that we are impacted too by this.
A whole bunch of them have been arrested in or on their way to Dubai, Thailand, Maldives etc.
Fascinating story.
I'm so sorry to hear that has happened to the team behind hyperscan. It's truly a fantastic library and has helped us speed up some of our regex heavy workloads write drastically. I even spent some time writing about hyperscan both internally (at $dayjob) and externally.

Thanks for your work on hyperscan! I think I'll identify a few other libraries that we use and reach out to their developers - it's a good reminder to not take everything for granted.

https://geekmonkey.org/regular-expression-matching-at-scale-...
The devil is in the details. Should one attempt to tax unrealized capital gains, what is a workable way to do so? Would there be massive credits for paper losses? How does one value illiquid shares? Minority shares of private companies?

The capital-gains step-up definitely is worthy of reconsideration. It is unclear to me why an asset's basis should change simply because someone died. If heirs don't sell, they won't owe the tax, but it makes sense that they would at the time of sale.

Buffett has thought long and hard about these questions -- how to address inequities in tax burdens without breaking industry. Don't miss his reply to ProPublica (cited within the article): https://www.documentcloud.org/documents/20798866-buffett-sta...
> Why not replace plane travel with hyperloop?

You can’t replace a reliable mode of transportation with something that doesn’t exist yet. Even when it’s ready, it’d take decades. Political powers are reluctant to commit to something that’s going to take longer than their careers.

Then again, you don’t even need hyperloop. Germany has 250 km/h trains, those are more than enough for many routes.
It should be resolve soon. From fastly status page:

The issue has been identified and a fix is being implemented.
Posted 1 minute ago. Jun 08, 2021 - 10:44 UTC
Correct. The thing is though, so was all the "news" that asserted that the lab leak theory is a conspiracy theory, or that the only possibility is a natural origin, or that we should ban everyone from the internet who suggests lab leak is possible.

Conjecture is conjecture, but "this is conjecture" is not a refutation of said conjecture. And banning all refutations of conjecture such as "this was definitely natural and any suggestion that it is not is a racist conspiracy theory" is also not refutation.
I̶n̶ ̶r̶o̶m̶a̶n̶i̶a̶ ̶e̶v̶e̶r̶y̶t̶h̶i̶n̶g̶ ̶s̶e̶e̶m̶s̶ ̶b̶a̶c̶k̶ ̶t̶o̶ ̶n̶o̶r̶m̶a̶l̶.̶.̶.̶?̶

Edit: nope, just worked for 2-3 requests (10 secs)
Looks like they're currently applying a fix.

https://status.fastly.com/incidents/vpk0ssybt3bj
The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place.

I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.
They already have:

$ host m.media-amazon.com
m.media-amazon.com is an alias for c.media-amazon.com.
c.media-amazon.com has address 99.86.119.84

(which is a Cloudfront IP)
Click the new tab. Lots of posts about sites being down. All flagged.
err, to make representations platform independent?
Why would you even need a reverse proxy for a static page? This isn't about that.
Those of you that work in DevOps, SRE or are CTOs.

What kind of things do you put in place to manage these kind of centralised issues that are beyond your control?
> 6m users

Is the plan to lose money on each user until they reach profitability? How do they plan to make money?
> Immersion doesn't mean you have to pack your bags and move to Europe.

Joke's on you, I've lived in Europe my whole life.
By “dirty (but legal) tricks” you means following the the tax code and legally paying what you owe?
Every other comment about what's down in this thread -- as if we needed dozens of site-by-site accountings of this outage in the first place -- is a bitch about reddit. Why is reddit so important to this crowd? The specific topics I used to read the site for (half a dozen years ago) have all been overrun by "bucket people," there is literally never an answer to any question I find a google link to there, and the site's design is actively user-hostile. Seriously: what's keeping that place afloat? Porn, I suppose.
The issue has been identified and a fix is being implemented.
Posted 3 minutes ago. Jun 08, 2021 - 10:44 UTC
Pretty sure the PiMax "8K" (Misleading term since it's not 8K, it's just 4K per eye) fits the bill. They have had issues with production before but more recent reviews indicate a major improvement.

Do expect distortions on the edges though especially if you set it to display the full FOV.
Wonder if all the caches will have been wiped, causing knock on issues
You would think sites like Github and key government sites would at least have a fall back at the ready. It reasonable to use a CDN like Fastly, but having a single point of failure seems silly if you're the BBC or Gov UK. Although, it does seem BBC managed to get back up and running pretty quick so perhaps they were prepared for this.
And Noita wiki!
I'm working on something similar, so I may be biased but I'd say no. It's something that transforms and integrates with the content in your browser to facilitate learning. I think fair use applies because 1. it's for educational purposes 2. it does not diminish or compete with the value of the original copyright owner since the content is not copied to some separate site, and subscriptions/ads still apply (unless of course they also start a language learning business)

The name "Netflix" though seems troublesome, I agree.
My work's website is down too and the regular sites I use to escape work borderm
UK is ahead of the US https://ourworldindata.org/covid-vaccinations
Do companies really not run test suites / do manual testing before deploying to production?
How Can I Scrape Companies Data from LinkedIn?
Yo dawg, I heard you like CDNs...
For sites of any complexity with any dynamic content having CDN redundancy is akin to being multi-cloud -- it is not worth the effort.

A lot of dynamic sites use Fastly for its programmatic edge control and a near immediate ( ~1s-4s, typically around 2 ) global cache invalidation for any tagged objects with a single call to the tag. That feature alone simplifies backend logic significantly. To make this feature portable to CDNs that do not support it and provide only regular cache invalidation requires a complicated workflow setup which significantly increases the cache bust time, which in turn removes all the advantages of the treat dynamic content as static and cache bust on write approach.
You're right, I should've said *partially* back. At least the CSSs now load, but a few products images are still gone. However it was completely broken here before (literally loading just the main HTML).
"The internet will just route around a local / centralised problem ... like water around an object"

Obligatory LOL ...