I think the problem is vastly more subtle and fundamental than the author of this piece (and ProPublica in general, given their previous coverage of the same story) is giving credit.

Software is in a dangerous place right now[1]. Half[2]-way between stupid and smart. Recent advances from fields like Machine Learning have moved the software out of world of the purely deterministic. Stochastic methods have given us software that can live in the gray that is reality; this has made it mighty. Particular subdomains that were once entirely the purview of human workers are rapidly moving towards automation. These algorithms have gotten good enough at what they do, accurate enough at what they do, that they are approaching the semantic-work-outsourcing limit that is 'trustworthiness'; You can treat them more like agents than tools, and trust that they do their job sufficiently well that you, the consumer of their work, need not worry about the details; you get to take the executive role of dealing only with the abstractions they provide. "Just tell me yes or no if we should do this". You implicitly trust that the system will "Do The Right Thing"; of course it will, it's got a fantastic resume with some great recommendations[3].

The problem is, the algs are also still dumb. Very dumb. They cannot model themselves. They cannot introspect. And, perhaps most crucially, they cannot interface with their newly-promoted executives in the lingua franca to explain why they're dumb. When you want to figure out why the intern decided that it would be a good idea to name all their variables some permutation of the words 'herp' and 'derp', you march over to their cube and ask them. A conversation occurs. Different perspectives are exchanged via a common protocol[4]. New knowledge is acquired. A mutual understanding is reached. When you want to figure out why your Facial Recognition Software isn't acknowledging Black People[5]... you go get a Masters in Statistics, Distributed System, Probability Theory, with a minor in Anthropology and Demographics. Then you spend a month reading code and running experiments. The software, briefly perceived as a trustable agent that knows how to do its job, suddenly becomes a tool again, because you can't just ask it why it screwed up so badly. And not just any tool; an incredibly complicated, fragile, and opaque tool, with a million different knobs and dials and a Gordian nest of pipework and conduits that would make even the bravest chaotician sweat a little bit. Even when you've open sourced the data and implementation and the deployment architecture and the napkins you've been scribbling hyperparameters on, the box is still pretty damn black.

And so, while auditability or accountability is important, it's only a small (and very, very, very hard) piece of the societal changes that might be needed. Changes in ethics (who goes to jail when a UAV confuses a hospital for a barracks? What does a smart gun do if its wielder pulls the trigger while pointing at a civilian?). Changes in focus (Is it correct to make such deterministic choices about the world? Maybe Hume was right and the predictability of human behavior is slightly harder than we currently state it to be[6]. Maybe accuracy is bounded lower than we like, and so justification should be the primary target). Changes in education (and a reduction in the magical thinking about computers. I hope some distant ancestors of mine finally see the day that computers are as boring and obvious as hammers).

I worry about focusing so much on accountability because as it stands now, a highly autidable system as defined by the article still needs deep domain knowledge to even begin theorizing about. All of these things above and more will probably need to shift as well, whether deliberately or not, in order to accommodate these smart-dumb tool-agent hybrids we have now, systems that we're building right now, that are just powerful enough to be dangerous.

---

[1] Every generation has said these exact words about every technological advance across every field for as long as we've been finding ways of rendering human beings obsolete. I'm sure there's some philologically reconstructible PIE for the phrase "those scientists are playing god". We've discovered it before, and I have no doubt we'll discover again solutions for these things, but that doesn't render the conversation in the interim any less meaningful.

[2] 'Half' is probably optimistic. Consider it an upper bound.

[3] Studied at MNIST... Tuned by Hinton and Ng... Deployed on three customer engagements with great KPIs... Very impressive stuff, Mr. Convolutional!

[4] I mean Natural Language here, but I suppose baseball bats and guttural screams may be other possible channels in this particular scenario.

[5] Or why your self-driving smart car crashed into a cement barrier

[6] I was trying to construct some pun about "the son not rising to the mistakes of the father tomorrow". It was pretty bad.