What's contentious and not scientifically accepted is his claim that differences in gender are due to inherent and fixed differences in the brain.   You simply can't tell from studying adults whether their differences are due to biology or environment.  

Damore's arguing that women are not suited to tech because of differences in their brains is just a way of saying that his female coworkers are biologically inferior without coming right out and saying it.  And at the same time, any woman who does prove herself highly capable is by definition some kind of freak of nature.  

Women have pretty much had it with men telling us what we aren't able to do.  Historically, men have always been proven wrong.

Google isn't a debating club, and their HR people know more about managing HR than than he does.  They also know the guy's attitudes are poisonous to a functional working environment.  They don't owe him anything, except possibly severance, and even that's not clear.