> We tend to think of machines, in particular smart machines, as somehow cold, calculating and unbiased. We believe that self-driving cars will have no preference during life or death decisions between the driver and a random pedestrian. We trust that smart systems performing credit assessments will ignore everything except the genuinely impactful metrics, such as income and FICO scores. And we understand that learning systems will always converge on ground truth because unbiased algorithms drive them.

I absolutely hate this myth, and I suspect it's just an excuse used by technologists (consciously or not) to wash their hands clean of taking social responsibility.

There is no such thing as innocent technology, because it is made by and for humans, and with a purpose. Which means that before we even built anything we have already left the domain of "is" and entered "ought". Technology "encodes" values by being an answer to how we attempt to implement some intended purpose, and thus becomes entwined with the questions of responsibility that we all struggle with.

Here's a great anthology of writings on that topic: Inside the Politics of Technology[0], mostly written from a post-phenomenological[1] perspective.

Excerpt from the introduction:

> It was eventually proven that the pilot acted as capably as possible, so he was not to blame. But that only partly settled the question. After this first clarification, however important it was to the pilot, the accident could still be ascribed either to human error, e.g., false instructions from the control tower,
poor maintenance, or lack of knowledge about the migration cory patterns of birds, or, alternatively, to technical deficiencies, e.g., fuel problems or engine failure.

> In the end, a year after the crash, an official research report established the “real” cause of the crash: the snapping-off of a 10-cm metal pin that regulated the position of a fin in one of the 13 cogs in the rotor of the F-16. This set off a chain reaction demolishing the cogs one by one, ending up in a complete breakdown of the engine.

> But even then the problem of humanity versus technology was not solved. Who could be blamed for this technical defect: service engineers, the Air Force, or the manufacturer? The latter was ultimately left holding the bag. But still then, was it a production fault or a designer’s error? Again,different actors and different technicalities are involved.

> Apparently, a definitive dividing line between technical and human causes cannot be drawn. However technical the cause of the crash appeared to be, human beings always come along with the technicalities – and vice versa. Purely technical causes are just as illusory as purely human faults.

The idea of machines being unbiased and innocent is not just naive, but outright damaging.

[0] Free PDF: https://oapen.org/download?type=document&docid=340207

[1] A friend of mine explained post-phenomenology as follows: "My understanding is that for Verbeek and others like him in general, postphenomenology continues the core traditions of phenomenology but with a deliberate emphasis on social science methods and empiricism over philosophical arguments - a positivist Heideggerian phenomenology"