Not quite: I think you've conflated some senses.

• The third-person plural 'they' is uncontroversial.

• The third-person singular 'they' for an unknown (but not general) individual, while valid English in nearly all dialects, went through a period of being proscribed for no apparent reason.¹

• The third-person singular 'they' for a specific individual, of unknown gender, was proscribed and uncommon. Evidence of its historical use is a lot rarer than the unknown-individual usage.

• For a known individual of known binary gender, you normally use the pronoun corresponding to their gender. (As you observed, neutral 'they' is becoming popular as an alternative.)

• For a known individual of known non-binary gender, it gets trickier. It's hard to separate language from culture, and English culture has more-or-less² only had two genders throughout the EMod–Modern English period: denoted by 'he' and 'she', respectively.³ To describe a non-binary individual who's sufficiently far from either of those categories is impossible, unless you fall back on the closest available construction: once 'it', currently 'they'.

To use 'they' to refer to somebody whose gender is known is a relatively recent construction – but English has been steadily losing its gender for the past few centuries. A few decades ago, to people in rural areas of England, a hedge was 'she', not 'it'. Now, we have sewists, and a woman's hair can be 'blond'. It breaks grammar no more than any other option would – and certainly less than the loss of 'thou' did:

> Again, the corrupt and unsound form of speaking in the plural number to a single person, you to one, instead of thou, contrary to the pure, plain, and single language of truth, thou to one, and you to more than one, which had always been used by God to men, and men to God, as well as one to another, from the oldest record of time till corrupt men, for corrupt ends, in later and corrupt times, to flatter, fawn, and work upon the corrupt nature in men, brought in that false and senseless way of speaking you to one, which has since corrupted the modern languages, and hath greatly debased the spirits and depraved the manners of men;—this evil custom I had been as forward in as others, and this I was now called out of and required to cease from.

The History of Thomas Elwood, via https://en.wikisource.org/wiki/Page:The_varieties_of_religio...

Plenty of languages have a gender-neutral form of address: one that can be used for anyone. It serves a purpose, it fits a pattern, people are using it, and it doesn't even require extra logic in my natural language parser: I see no reason to call this sense ungrammatical, especially not when the others are accepted.

---

¹: For a specific instance of the general person, 'one' and 'you' are used, with 'one' currently out of fashion: I personally prescribe 'one', but often find myself using 'you' anyway.

²: Upper-class English culture, anyway. Great Britain's got a dozen ethnic groups on it, more ways of speaking than you can shake a stick at, can't even make up its mind what a 'country' is, and don't get me started on the trade/invade/cold-war dynamic with what seems like the entirety of Western Europe. And then you've got religion on top of that: are we with the Pope? Are we against the Pope? Do we even care? Which prayer books are we using? Are we running out into the woods when the moon is full and yelling 'Diana' into the night? Is the priesthood male, or some 'third sex' – and if so, what (if anything) does that have to do with 'eunuchs'? And then there's historiography on top of that, because culture is affected by people's beliefs about what is and isn't traditional… No, it's much easier to stick with what the wealthy and powerful's letters and diaries and books say, than to try to work that whole mess out with basically no sources available.

³: This isn't strictly true: I've seen writing that used þorn ſimultaneously with 'it' for Hermaphroditus. Currently, 'it' seems to be exclusively for objects, dehumanising when used for people… except infants, where it's an acceptable gender-neutral personal, for some reason.