  Let $S$ be a Shimura variety and let $h$ be a Weil height function on $S$. We
conjecture that the heights of special points in $S$ are discriminant
negligible. Assuming this conjecture to be true, we prove that the sizes of the
Galois orbits of special points grow as a fixed power of their discriminant (an
invariant we will define in the text). In the case of Shimura varieties of
abelian type, the height bound holds by the recently proved averaged Colmez
formula, and our theorem gives a new proof of Tsimerman's Galois lower bound in
this case. The main novelty is that our approach avoids the use of
Masser-W\"ustholz isogeny estimates, replacing them by a point-counting
argument, and establishes lower bounds for Galois orbits conditional on height
bounds for \emph{arbitrary} Shimura varieties. In particular, following the
Pila-Zannier strategy (and Gao's work in the mixed case) this implies that the
Andre-Oort conjecture for an arbitrary (mixed) Shimura variety follows from the
corresponding conjecture on heights of special points.
