Money. Time helps.

Creating reliable software-intensive systems is something we know how to do, we have repeatedly done it in the past, and we are doing in the present. Only problems are it ain't cheap in terms of money or time, so to be cost effective, the costs associated with failure or malfunction must be high.

Safety critical systems (where human health can be endangered by system failures and malfunctions) are one such system. They're usually developed under some sort of regulatory oversight (e.g., FAA, FDA). There are a set of minimal government standards that must be met, much of these related to documentation, and much of the documentation relating to risk analysis and quality management. There are also standards administered by industry groups---often, these are "voluntary" in name but for liability reasons practically required (in my experience). There's going to be some sort of (mandated) monitoring of the product after release (which feeds into the quality system) to correct design flaws, defects, etc. You have regulatory affairs people on staff to handle this area. The safety critical system itself is developed under some sort of formal product lifecycle management system. There will be a formal set of requirements. Early in the process, there will be a risk analysis (continually updated) of the design. There will be an a priori development plan, timeline, and cost estimates. As for development models, the waterfall model is alive and well, in part because it coöperates with the demanding documentation requirements; waterfall does seem to be on the decline in favor of more agile models (but not too agile!).

The software itself may be assembler if you're really resource constrained, but more likely, it will be C or C++ with a restrictive coding standard, or real-time Java, or possibly Ada. If you're fancy (and progressive), there might be a system model with automatic code generation. The tooling you adopt (e.g., from Atego) will assist with verification and validation, as well as requirements traceability. Most people, I think, find software work under these conditions to be boring and frustrating---it's certainly limiting. You're maybe generating 100 SLOC a day? The system is almost certainly running on some sort of real-time OS like vxWorks or QNX, and the hardware, software, and any associated libraries are (as much as possible) validated for use for your application. You can, given the right circumstances, use COTS software (a.k.a. SOUP, software of unknown pedigree), but then the validation becomes your responsibility.

The HN crowd hates it, but in my experience on these sorts of projects, degrees and certifications are required and expected except at perhaps the very lowest level of employees. Electrical and mechanical designs are probably stamped by a real engineer; depending on the nature of the software portion (and where it's being developed) the software design might get stamped too, but more likely, it's being supervised by someone with a bunch of industry-relevant certifications.

I've glossed over a ton, and skipped even more.

In my opinion: I think that the most important lesson to learn from safety critical projects (and good engineering practice in general) is that tools are not the major driver of quality: proper engineering processes and quality management are the driver of quality. A "right-sized" product lifecycle management plan, including a "right-sized" quality system are critical and necessary to predictably and reliably produce a quality product. All else being equal, I'd predict that a C application developed under a lightweight PLM and QMS will be superior to one developed under Agile and Rust, or Scrum and Haskell, or whatever the du jour fad might be.