  When noninteracting fermions are confined in a $D$-dimensional region of
volume $O(L^D)$ and subjected to a continuous (or piecewise continuous)
potential $V$ which decays sufficiently fast with distance, in the
thermodynamic limit, the ground state energy of the system does not depend on
$V$, but only on the imposed boundary conditions. Here, we review this almost
unknown result from several perspectives and derive a proof for radially
symmetric potentials valid in $D=2$ and $D=3$ dimensions, generalizing an
existing approach to the case $D=1$. We find that this universality property
holds under a quite mild condition on $V$ and extends to thermal states.
Moreover, it leads to an interesting analogy between Anderson's orthogonality
catastrophe and first-order quantum phase transitions.
