> In the same way that science enters into the making of pink or blue paint. It does not say which color is the prettiest, just how to manufacture them and the characteristics they have.

Unless we're committed to the idea that moral claims have the same sort of weight as preference claims about colors, I don't think this analogy holds water. Science can't tell us how to make value-independent morality in the way that in can tell us to make value-independent paint, because moral systems are themselves fundamentally normative.

> Similarly, science can tell how morality systems appear, are transmitted, what their consequences and eventual contradictions are: You think we should have equality of chances? Here are the likely outcomes given our current society. You think we should have equality of outcomes? Here are the factors to adjust for.

It can probably tell us how norms appear, are transmitted, &c. You're burying the more serious question here: why am I posing this thing (equality of opportunity) in this way ("good") to begin with?

> We are now rushing laying the theoretical groundworks for a practical theory of ethics because we are the one who will be blame for what our models learn and because, quite frankly, philosophers have not done their jobs there.

I take issue with this: practical ethics is one of the shining achievements of 20th and 21st century philosophy (along with advances in the phil. of math and science that have enabled this conversation). IRBs, harm reduction policies, the UN Declaration of Human Rights, Effective Altruism, and ethical consumerism are all direct products of practice-oriented moral philosophy. The fact of the matter is that computer scientists and engineers, including ones in my own subfield, simply haven't done their homework.

> The whole point of science is to build a predictive model of the world that is coherent with observations. If you are doing science, you care about reality matching your model.

According to who? That's not Quine's science, or Popper's, or Carnap's. It's certainly not Newton's or Aristotle's. It's not immediately obvious to me what "science" even is beyond the vaguest senses (it's certainly epistemic, and just about everybody agrees that it involves knowledge of real things).

And, to be clear, this isn't solipsism: I am not doubting the existence of science, or its ability to produce truth. I'm just pointing out that we haven't established our scope yet.

> Specifying a moral system in terms of objective metrics is indeed a first step. Why has there been so little work on this?

There's actually been a great deal of work on this: Bentham and Mill both sought to structure their utilitarian systems on formal (and quantifiable!) evaluations of pleasure and pain. Singer follows suit, albeit with fuzzier orderings rather than quantities. Kant's metaethics are essentially a souped-up law of non-contradiction. Reasoning about natural rights follows a similar pattern.

> Do as in math: state what is true, given some explicitly given assumptions. "If we consider only positive integers, then a^3+b^3=c^3 has no solution". "If we consider only numbers in Z2 then 1+1=0"

This would involve a predication of any metaethical system, e.g. "if we accept Mill's utilitarian calculus, then the right thing to do in situation X is Y." But here's the problem: we don't all accept Mill's utilitarian calculus, and those of us who don't aren't likely to be persuaded in the same way that considering only the natural numbers in a mathematical problem might. When it comes down to actually acting, we're back where we started: why ought I accept Mill's utilitarian calculus?