I'd be willing to bet some money on it being simply the result of some employees saying, "AI ethics is super important. We feel strongly there should be people working on AI ethics to make sure it's not racist or sexist".

Oh, no doubt. But that scenario would be rooted in this larger context of how we generally relate to this topic.

Creating a dedicated ethics team was a dumb idea because it can only possibly end like this

I agree, which is why I say "it's probably stupidly conceived."

I find social stuff fascinating and I've had the following college classes: Intro to Psychology; Social Psychology; Negotiation and Conflict Management. I also raised two fairly Aspie-ish sons and I've had to explain social things to them in very exacting terms, which forced me to really up my game on being precise in order for it to be useful to them. My sons and I still look for things that meet our rigorous standards for "How do you know that's actually accurate?" because social phenomenon are inherently hard to test: People behave differently when they know "This is a test. This is only a test." than they do in real life (which is why I think UBI experiments are useless: These people know the study will end and the money will stop rolling in, so it doesn't tell you how people would really behave if we really instituted a UBI).

Anyway, I find social stuff fascinating and people routinely interpret my social observations as being ethical or moral judgements and I get this reputation for being a real judgy bitch, basically, even though I"m a lot less judgy than most people. But when push comes to shove, "ethics" is generally about determining which behaviors protect one or both of two things: individuals from group abuse or groups from individual abuse ("tragedy of the commons" type stuff et al).

So when I talk about what works or what doesn't, in some sense I am talking about ethics, but I'm usually not trying to be some kind of moralizing priss, which is how that gets interpreted. I'm usually trying to say something like "Objectively speaking, you can't be both heavy and light at the same time. That fundamentally doesn't parse." and then someone is all up in arms about "Don't talk about my momma!!!! grrrrr!" type stuff.

So I feel strongly that what they really were probably wanting to study is "How do we do AI well in ways that interact with this social stuff like racism?" and out popped the word ethics because that's how people think of that. And it was a terrible idea on the face of it because of the inherent assumption of judgment rather than a more objective framing about studying AI and social phenomenon and how does that work? When you start using AI to do things like parse faces, what are the practical pain points there that we need to be aware of and try to address?

But I think most people can't readily separate that out in that way, which is part of why I am so often in hot water for saying "Well, actually, I think that simply doesn't work because...." And when I say that about social phenomenon, I am often de facto talking about "human ethics" whether I want to be or not and it's very often a giant pile of manure that I've just stepped into.