I know what you mean, and I agree that the term's construction can easily lend itself to unproductive and divisive rhetoric, though I think it's a bit more complicated.

I do think "toxic masculinity" is very different from a phrase like "violent blacks" because it specifies a behavioral trait/set of cultural mores (masculinity) over which people have a significant degree of control versus an intrinsic trait (like maleness or blackness) over which people have no control whatsoever. There's also not much of a history of men being violently repressed or disenfranchised purely for being men, so the potential antagonistic aspects of a phrase like "toxic masculinity" are not instantly amplified by association/context in the same way as your other examples. Also, "toxic" is not tied to a specific harmful stereotype; this leaves one open to dissect which aspects of stereotypical masculinity are toxic, rather than concluding a priori that men are "violent" or "neurotic" like in the other two examples. I think "toxic feminity" could similarly, in good faith, be used constructively in ways that "neurotic femininity" cannot.

That said, the fact that a good number of people use "toxic masculinity" in a vague and unproductively confrontational way has given it a bad connotation to many; this sort of usage is certainly enabled by the brevity of the term. I think a lot of people (I suspect the majority) don't make the distinctions I made in my previous comment, and it undermines their efforts to use the term or concept of "toxic masculinity" in a truly helpful way. I've seen many examples of this: right-aligned people trying to discredit feminists as misandrist; feminists unempathetically erasing these distinctions; angry people in Twitter comment sections complaining about toxic masculinity; people who actually know better lazily using it as jargon on public forums where it can be misinterpreted (I'm sometimes guilty of this); and so on.

The term is also problematic in its omitted connotations; one of these omitted connotations is the interconnectedness of traditional feminine gender roles with toxic masculine traits. Many people ignore how these roles interact with and reinforce each other.

For example, where I live (NYC), it's unclear what type of masculinity is desirable to straight women. Most straight women I know are having a hard time finding guys they want to date because they've been trained to be attracted to some traits of toxic masculinity. But since they live in NYC in 2019, they also have learned recently to be wary of those same traits. Most of them are aware of this weird cognitive dissonance, but they can't figure out how to resolve it. Many of the straight men I know genuinely want to avoid toxic stereotypical behavior, and they're concerned about things like coming on strong/harassing women, but because many of the women they want to date still expect men to do more of the pursuing, it's ambiguous how precisely they should behave to make their dates feel desired but not cornered.

There are many less benign examples; I've seen plenty of women who mock boys and men for being too emotional, too prissy, too pretty, or too physically weak. It's not just men who define manhood in an overly-restrictive way. And these same women have to deal with the negative aspects of feminine gender conditioning. It's a cyclical thing, and I think the only healthy way I've seen to deal with it is to think of everybody as both perpetrator and victim. You can use "toxic masculinity" as a shorthand for the component of this approach related to masculine stereotypes, or you can use it in a divisive way that is unhelpful.

The fundamental issue is that "toxic masculinity" is non-technical jargon; it has a highly compressed meaning, but without a technical framework to limit its interpretation, it's also prone to heavy semantic drift through vagueness, malice, laziness, or stupidity. I think that someone choosing an alternative to "toxic masculinity" will have to deal with jargon's fundamental tradeoff between brevity and specificity. I won't be surprised if such an improved term (or at least and attempt at such) comes about at some point, as often happens with non-technical jargon, but picking such a term effectively is a deep problem rooted in the weakness and strength of natural language's inherent vagueness.