There are 2 problems with this way of reasoning is what you see in the aviation industry.

One, you're comparing this incident with an idealized human. Everybody thinks pilots and drivers are magical beings that don't make mistakes. And then 90 people die because one engine catches fire, pilot turns off the other engine and then banks the aircraft for make a quick emergency landing, obviously the plane loses lift immediately, drilled itself into a bridge, taking a taxi with it into the river below. No survivors, almost a hundred dead [1]. And I get it, the pilot needed to rapidly make a series of decisions under ridiculous levels of stress, that's the real cause. But when things go wrong close to the ground at 150km/h in an object that weighs 50 tons, they go wrong quickly, so you need to respond quickly. Needless to say, nothing prevents reoccurence. Quite simply, a plane will crash if you do this. There's nothing that can technically be done to prevent it. As for cars, over 10.000 people die every year because humans can't be bothered to wait until they sober up to drive [2]. Those are the human pilots that we should measure against.

Frankly, I don't understand how humans are allowed to drive cars or fly planes at all. We pass signals in our brain, they can cross from neuron to neuron in about 10ms. That means that in a second, a signal can affect, at most, a ball with radius 2cm in our brain. To spread out over the brain requires 7-8 seconds minimum in theory and in practice minutes is the more common scenario. That means, it's your spinal column that's driving the car/flying the plane and it gets updates "from upstairs" that are 2-5s old by the time they reach the control loops. Our brain is very good at predicting events so it doesn't look like that's the case, but it is.

Compare self-driving or other autopilots against realistic humans, who make these sorts of mistakes. An average autopilot needs to do better than a decent human driver. It should not need to outperform a magical how-we-imagine-ourselves perfect human driver.

That said, I do agree that we need some basic rules. Tesla's car did NOT stop after crashing into a truck after a lidar mistake (and a serious mistake by the truck driver that it was attempting to compensate for I might add), but not stopping, that's utterly unforgivable. Same here, obviously the car was obviously either driving with the lidar turned off or ignoring it's output. That's like a human driving with their eyes closed.

But certification must be functional. It can't be on the quality of individual components. It can't be on code review. It must be functional. We should have test tracks where autopilots get confronted with dozens of situations, preferably combining 5-6 individual problems at the same time. And then it needs to navigate it safely, under constant decision pressure. And then it needs to end with 5 cars being sacrificed to test their reactions when something heavy drops on them, when they drop off a cliff, when they get mechanically blocked, and when catastrophic mechanical failure occurs ... (so they don't put the pedal to the metal causing 10x the damage to others when they do have an accident).

But it needs to be a functional, practical test. Not the madness we currently have for aviation.

[1] https://www.youtube.com/watch?v=jKNREZ_u8E8&t=31s

[2] https://www.cdc.gov/motorvehiclesafety/impaired_driving/impa...