It is indeed troubling that Clinton could be driven out by voters who don't necessarily deplore his conduct.
The voters don't necessarily oppose Clinton's conduct.