skysight-inc/HackerNews huggingface dataset:
Firebase projects are working but can't go to dashboard.
The UK does have control on web advertising, but it's pretty limited: https://www.asa.org.uk/news/regulation-of-online-advertising...
Now that's an interesting mode of failure cascade!
Google is back up
All services back up for me.
> indirectly by preventing competitors who aren't looking for an exit from existing.

This is huge, and definitely hurts in the short term, but could it actually be good in the long term? Let's say the break-even price for Service A is $x/mo/user. Nobody really knows the technical problems underlying A, but startups #1-3 get venture funding and start offering Service A for .75x, for a couple years each, burning a few million in some dentist's or VC's capital each and build a huge staff to figure out the tech. Eventually one of the companies gets acquihired and two of them fail.

Bootstrapped Startup #4 opens shop, offers A to 1.15x, and hires engineers from startups 2 and 3 who know the tech inside and out. Consumers get a great product.
From personal experience and other Ops peoples stories, I'd assume that could be the case for many status pages ;)
Good ole phone calls is my guess
I really hope this is just a temporary issue and no accounts are going to be lost. If that happens, way too many people are going to feel the pain of thinking Google makes backups unnecessary - including me.
If your home automation system requires a network connection to work then something is very wrong.
The SR-71 was apparently a maintenance and repair nightmare, the per hour fully calculated operating cost was quite ridiculous. The NRO obviously doesn't publish any useful unclassified info for radar or visual, infrared spectrum observation satellites newer than about 1965. But if I had to guess, a lot of its mission became obsolete when post-1980s era NRO satellites went into service.
You can't solve world hunger on the supply side. There will always be more hunger afterwards.
SSD vs. HDD: Which Storage Device Should You Choose?
Probably dozens of other people executed comparable requests in the instant you did.
zanzibar is an authorization service, and we are looking auth as in authentication here

So no, not zanzibar
opening it with Incognito mode works too
Yeah I couldn’t get the hours of operation for a store near me on Google maps. Thought it was wifi at first.
The Google Outage thread is probably overwhelming HN. If you log out, HN can serve your request from cache and is much better.

If you're just reading for the next few hrs, just Log out.

(Not an insider, just repeating what's happened last time there was a big article here)
I am not google, but if i were to start looking for idiots to throw under a bus for this outage, you will on top of the list.

besides google won't be exactly calling you because you made that comment here.

other commenters are right that you should not expose your personal number on internet regardless of it was you or not
There is an honest need in here, to surface unknown unknowns within the world's of consumer producta/services. It could be solved by a trade show/business fair kind of thing (maybe we would know we had solved it when B2C looked like B2B)

The thing is, advertising does not efficiently suggest products/services now, so the idea that removing it would hurt customers ability to find what they want is a little melodramatic
Whitewood Under Siege: The front lines of the pallet wars (2013)
Makes sense, at work we have an application running on Google Cloud and everything seems to be working. So the outage is probably not at network or infrastructure level.
Gmail service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.
Pour one out for the SREs...
It seems to me that mass media has always been under the control of a small group of people; I believe that the current narrative that unrestricted, uncensored mass media is dangerous/an existential threat stems from the fact that digital communications systems stand to threaten that bottleneck, if properly deployed.

It strikes me that this is why Facebook et al are so eager to censor; otherwise an attempt will be made to destroy them, as they provide a bypass around centralized mass media narrative control.

Narratives are how the world is practically governed. The ability to propagate a narrative to millions of people is a power on par with a standing army.
This isn't true, but considering every Google outage is a one in a billion rube goldberg domino machine, it could be true. Put this comment in the post mortem!
Just permissions. The "IAM" can be safely dropped. It's exactly what you think it'd be: restrictions and privileges.

"IAM" is basically the name for a specific model of doing it.

Unless something really crazy happened, this user is unlikely to be correct. Accounts are supposed to be firewalled/sandboxed in a way that you can't contagion across to someone else's let alone systemwide.

It's possible (some sweeping script on a powerful connection that smashes just the right things or some exploit to break the sandboxing), just probably not likely - especially unintentionally.

But crazier things have happened
https://books.google.com/books?id=rRp7DkTegMEC&newbks=0&prin...
Some drive cars on race tracks where it is legal. Also some roads have high speed limits like ones in Germany.
The point is that you shouldn't put all your eggs in one basket. All services go down. If you're worried about someone else handling it when it goes down then host your own [1], otherwise you can use something different for each thing you need. Don't rely on Google for everything.

1- https://github.com/awesome-selfhosted/awesome-selfhosted#ema...
let's not make panicked drastic decisions just yet
Yeah, that status page is really frustrating. Didn't anyone consider building it to REALLY check if services were working? not just returning a 200 Status... but actually DOING things?
In Germany there is no speed limit on highways.
Everything is back and working fine now, here.
its not green anymore
Earning free gift cards is easy and fun
> You are quite literally selling tech savvy people their future

It's a bit overkill. You can just do it yourself. Plenty of resources available for free. Students have to put the work in regardless.
drive too.
Ask your friend to build a post about what happened. I can imagine the heat in the online conference, one thing is to get all nodes from one service down; another thing is to get all the servers down at the same time.

Clearly they're using the same infrastructure for everything which IMHO is a huge mistake
I'd wager a guess that you set up some weird 'expression'? coupled with some bug in the IAM service, maybe some stale resources that you were deleting at the same time?

I'd then assume once expression is evaluated the services end up busy looping / proxies throwing out internal errors and taking out capacity.

Still, you shouldn't be able to cause downtime to more then a few servers in the extremely unlikely case I am anywhere close.

PS:
- I haven't used googles IAM so guessing after a few min of reading docs.

- you are incredibly unlikely to have triggered this at google's scale.
HN was having problem showing anything for the past 10 min or so. I was going to ask if HN was down, then I saw this.
This doesn't seem like a very compelling argument to me. They serve different purposes.

Do Habitat for Humanity stores put Home Depot out of business?

Do food banks put grocery stores out of business?

Does offering free water as a drink put bars out of business?

Same product category, very different markets.
That is not the point and never was ! When your hospital is saturated by people that ultimately will survive but need 3 to 4 weeks to recover, you're going to die from anything but covid.

You're in a car crash? To bad hospital is saturated.
You got 4th degree burn in a burning building? To bad hospital is saturated.

Do you realise that is the problem?
Big Tech Turns Its Lobbyists Loose on Europe, Alarming Regulators
If it was you, why the hell on earth would you bother exposing your personal phone number on the internet and asking google to call you on this post? Like, seriously...

Wouldn't you rather call them directly on the hotline...?

That's a phone number from Bulgaria, and it looks like it should be part of the Vivacom GSM network, so I guess it's his personal mobile phone number or a scam.
Although since it seems to be back up now I suppose there is one way we could find out for sure.
Can't remember where I read this but IIRC it's not a healthy for a machine to constantly run at 100%, hence you make an engine with a peak performance of 200mph which constantly runs at a cosy speed of 100mph.
Total world economic output is ~$150M / minute, so billions every minute is off by few orders of magnitude.
"If greenhouse gas emissions are not significantly reduced, part of the MENA [Middle East and North Africa] region risks becoming uninhabitable before the year 2100."

https://en.wikipedia.org/wiki/Climate_change_in_the_Middle_E...
Babashka is dangerous...
Home Assistant rulez with non-cloud sensors. I never by sensors, cameras, switches which require connection to an external host. But I'm sure that most people will do it even with such outages too.
> It’s default on kids accounts on iOS. This sounds more like she let her kid use her own iPad on her own account, not her kids account

Given that iPads are expensive and don’t support multiple accounts, this doesn’t seem so unreasonable.
I fell for it for a few seconds :-)
Similar to the attack on Ukrainian systems, which was also thorough a centralised update system being compromised.

It is that simple.
Clean advertising means improving the user experience and having more accountability, so consumers aren’t irritated and brands don’t fear fraud or misplacement.
It's time to decentralized Gmail, Youtube, and Google Docs, maybe their Search Engine too.
Yeah xD
Supposedly this was 'fixed' in 5.9 with experimental nvidia-drivers (455.45.01)
If this does not improve soon, we're looking at one of the most significant outages in recent internet history, at least from the number of people impacted.
How much do you think Google can learn from this incident's postmortem? Writing down all the internal details probably amounts to a small book.
Microsoft: Guidance on SolarWinds Attacks with Details on Persistence
It is called HN hug of death (or Reddit hug of death). the traffic due to google outage is much more than usual
Indeed. I deleted a gcp project at the same time.

However, it would be fun if it had a UUID clash with a google service :)
> I do not envy Google engineers right now.

A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release.

I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
It is legal to accelerate from 0 to 100Kph very quickly
Time to mention the alternative, a decentralized internet, e.g. https://zeronet.io
Golan Heights Law
It's restored now.
That figure seems way too low, what are your sources on it?
I've been using mailu.io to host my own email server. Makes it real easy to manage yourself.

I haven't had any issues with new domains being marked as spam, but I always make sure the SPF, DKIM and DMARC records are set up.
I'm on a 4k screen. You just convinced me to move from 125% to 150% scaling.
Tracks, private roads, autobahns, dynos, plenty of places to go fast legally.

Not to mention the extra power goes towards faster acceleration, not just top speed. The same force that gets you an extra 10mph top speed also works to get you from from 0-60mph that much faster.
"all green, which does not reflect reality for me (e.g. Gmail is down)"

This isn't true. Which make it more complicated as we try to figure out what systems are down -- since their status board is usually faked.

gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed: - Internal Error
Why do they make keyboards that can type hate crime? Why do they make cameras that can shoot child porn? This is possibly the worst question I've seen in this site.
I used to think about this a lot when I was younger, because it makes very little sense, and as I became older I realized how many more things make very little sense and I eventually came to realize the bug is low-level and there are a lot of rich and powerful people who do not want it to be fixed, because it allows for them to exploit everybody else.
Falsifiability was Popper’s demarcation of science, not his demarcation of tolerance (if he had one).

Note that he specifically says in that passage that he’s not advocating the suppression of all intolerant speech. Thus even if some speech were intolerant, that by itself would be insufficient for suppression.

---

Misinformation is problematic to the extent that it endangers the open society’s very existence, and with it its ability to self-correct.
4:41AM PT, Google services have been restored to my accounts (free & gsuite).

And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads.

I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their services or I am encountering underutilized servers since it is not fixed for everyone.

It is nice to experience (even if it is short lived) the snappiness of Google services if they weren't so multi-tenented.
Jobs was famously against pornography on a personal level. You will find email exchanges back in the day with fans/bloggers/writers where he lists his reasons why porn is not allowed on the App Store.
First the two Gmail accounts I had in Thunderbird logged me out and asked to relogin.

Then Google said my accounts did NOT exist which made me feel very uneasy. Banned? Lost all my data? OMG.

Then I got error "502 That's all we know" after entering my passwords.

Finally I realized it couldn't be just me, so I opened HN and confirmed my suspicions.

It's all up and running now but I was really scared.

I'm really curious what happened because it surely looks like Google has a single point of failure in their authentication mechanism which is just horribly wrong. The company has over two billion users - a situation/configuration like this just shouldn't be even theoretically possible.
Take a look at the headband. It’s going to fall apart in a year with constant use.

Almost every other pair of headphones in this price range offers replaceable headbands, and many of them can be customized with off the shelf products.
Commercial ISPs will do fine - in the coming age of work-from-home there are always going to be those who need/desire more bandwidth than that provided by community ISPs. Most basic access doesn’t require the kind of high bandwidth that commercial providers are supposed to be capable of.

A threat to commercial providers is a very good thing as it might force them to step up their game and provide even better access, if people realize they don’t need continuous access to streaming media to get by. More likely, they’ll just throw money at lawyers and politicians to get any competition shut down, but hope remains.
"Google has been contacted for comment, but one spokesperson said they were unable to access their email."
The speed of cars mostly comes down to how much power they have. The main reason for having more power is not to increase top speed, which is limited by law, but to increase acceleration, which is not.
Most of the things I backed up for myself are either gone forever or irretrievably lost.

Most of the things I backed up with google remain largely accessible, except for an occasion like this.

It's rare that any services I operate solo come back this quickly after there is a downing issue.
Or the more realistic scenario is that the outage happened before you did getIamPolicy which is what caused the garbage data.
Big Tech is practicing their shutdown maneuvers as we get closer to Inauguration Day in the US.
It seems to me anecdotally as a free YouTube user that the number of ads I see on there have been cranked up. As often as once every 3 minutes. I remind myself that I'm not paying anything for their hosting and serving of the video, but it still grinds... particularly so when it's the likes of BBC content which as a UK license payer, has already been paid for and is being monetised on there.
interestingly, from joel
"With all of Microsoft’s muscle, money, and marketing skill, they are just not going to be able to break into auctions or instant messaging, because the network effects there are so strong."
I remember AIM well (used AOL until they became TalkTalk), but I also remember using MSN messenger even more and being much more ubiquitous, granted that could have been generational (I wasn't a teenager until the mid-naughties). Ultimately it seems MSN did win out.
*small sections of some highways
Different failure mode. If the cloud goes down, many more people are affected. If your self-hosted thing goes down, only you are affected. If everybody self-hosted, would the overall downtime be lower? Even if it were, would it be worth the effort of self-hosting?
>No, I'm saying researchers ignored APT's for decades

That isn't even close to being true. In fact, that statement is ridiculous on its face.

Research into detecting, mitigating and countering such attacks go back at least 30 years.

That APT actors have often been successful is more a function of effective execution and poor security practices than lack of knowledge/research around such attacks.
It's what Amazon calls your cloud login account

Identity and Access Management (IAM)
It was obviously not you.
Consider the ratio of growth area to exposed surface area. 14x in this case. It's a vertical farm to maximise this ratio.

Light pipes may be slightly more efficient than PV in energy conversion during daylight hours, but they do not scale horizontally as the farm scales vertically.

For this project in Denmark, the energy comes from abundant wind, not weak and intermittent sun.
For baby pictures yes, for everything else, no
Fortunately email was designed back in a day when mail servers could fail. Don’t worry - proper SMTP servers will retry patiently.
I still like my physical switch on the wall to turn the light on and off very much.. for me it's hard to beat. I can even turn it on/off semi-remotely with e.g. a tennis ball.
Yep. At least some of the teams do.
data supports the idea that our expectations of visual experiences are controlled by a feedback loop between the visual cortex and the medial temporal lobe,
Emails will be delayed but sending SMTP servers have a retry mechanism up to a few days.
> Last thing you want is a conversation with google's lawyer

Google more or less want people find their weaknesses so they can patch and secure them. A person accidentally triggering a global outage is not something that would cause that person to get lawyers on them. Especially not something that only affects his or hers GCP-project.
I'd argue that you can hire employees for intelligence and other assorted skills but without an ambitious founder few startups would succeed.
Yes, we could use some numbers here.
I.e., how much energy would it take to grow one kg of potatoes in artificial lighting. Compared to how much it takes to transport it by ship from Morocco to Copenhagen. Or by an electrical train from Spain.

Production of electricity from wind turbines fluctuates a lot. So does the price. I wonder if there are plants that you can give light when it is windy at any time of the day and leave in the dark when when electricity is expensive.
It works if login with facebook or Apple ID, so GCP itself is fine, but I guess admin cannot login to management
Going back to normal.
Oh man, you're right. Bloated gmail loaded instantly. What's going on? It's loading almost 2x to 3x faster.
I'm experiencing issues with some other services https://killedbygoogle.com/
Looks like the outage was very close to that duration. Over what time period is that?
Capturing bitcoin mining heat does not make it less energy intensive. If a bitcoin is worth X dollars it will always take X dollars worth of energy to produce a bitcoin. This is true even if you increase the efficiency of the miner.
Of course, street-legal automobiles could simply be governed with a top speed of the fastest legal speed limit in a given region.
Is this satire? I'm genuinely confused.
Коле, ти ли си?
World GDP is $80 trillion per year.
Clearly they're using the same infrastructure for everything which IMHO is a huge mistake.

Let's not forget that most Google services have exceptional reliability and things like today's outage are incredibly rare. I'd bet that failures due to interoperability between disparate services would cause a lot more downtime than Google suffers right now if teams ran different infrastructures.
There are some things that were not available 50 years ago. Better robots, better accumulators, flying drones, image recognition, AI able to make complex decisions. Those things might reduce need in humans and humans are not cheap.
How Startups Plan for Billion-Dollar Success
I don't think it's difficult, I think they just lie.

Monitoring is very simple, I even learned this from a document released by the Google devops team many years ago.

Always alert from the end user perspective. So in other words have an external server test login to Gmail. Simple as that.

They manually update that status page to not scare away stockholders.
And now a days, bragging rights. People like speed - period. If 1 model out performs another model at a similar price tag and features / design - people will go with the faster one (usually). Same as any other personal preference - people shop with their eyes first, wallets second. If they like speed, that sells more cars.
TIL. What for?
Well, probably a lot of them are sitting around doing nothing because Google's down right now.
obviously it was an accident
this is what I don't get though. Why mortgage your future earnings to take a MOOC from a random startup employee when you can get the same from a world class professor for free?
Currently, I don't like the comparison to asbestos.

Asbestos was known early to be bad. Early mining towns lives were shorter[0]. Information was slow to communicate and I 100% agree there was a lot of money to keep the dangers from being known by the general public.

But asbestos, has a long and clear history of being bad.

Plastics are no where near as bad and are pretty much ubiquitous.

I think the opioid crisis or even sugar is more closely related to asbestos's style of danger combined with misinformation.

As this whole thread is discussing, the jury is still out, but there seems to be hardly the study that can point to an asbestos level problem.

If we all start developing cancer in about 20 years, I'll be the first to raise my hand admitting I didn't see that coming.

[0] https://en.wikipedia.org/wiki/Asbestos#Discovery_of_toxicity
Do 3) and 4) not contradict each other? 3) states all the distribution power is in the hands of a few companies, and 4) states that no entity can control the flow of information on these newly created distribution channels. Seems like they can't both be true?
How does this incident affect IoT devices? I'd expect many of them to be using Firebase just like HN.
Works well for the WF-1000MX3 .

Notification feature will be handy to remind me charge the damn things when they're getting low (i've got the voice prompts off).
I think it is necessary — even if not sufficient — to end the war on drugs. The only exception I’m willing to make is proper labeling and safe storage conditions. It should ought to remain unlawful to distribute drugs without proper labeling. We will need to come up with labeling standards.