  We study magnetohydrodynamic stability of neutron star core matter composed
of neutrons, protons and leptons threaded by a magnetar-strength magnetic field
$10^{14}$--$10^{17}$ G, where quantum electrodynamical effects and Landau
quantization of fermions are important. Stability is determined using the
Friedman--Schutz formalism for the canonical energy of fluid perturbations,
which we calculate for a magnetizable fluid with $H\neq B$. Using this and the
Euler--Heisenberg--Fermi--Dirac Lagrangian for a strongly magnetized fluid of
Landau-quantized charged fermions, we calculate the local stability criteria
for a fluid slab as a stand-in for a segment of a neutron star core, accounting
for magnetic and composition gradient buoyancy. The slab is threaded by a field
orthogonal to the gravitational field, the Cartesian analogy to a toroidal
field. We find that, for sufficiently strong fields $B\gtrsim10^{15}$ G, the
magnetized fluid is unstable to a magnetosonic-type instability with growth
times of order $10^{-3}$ s. The instability is triggered by sharp changes in
the second-order field derivative of the Euler--Heisenberg--Fermi--Dirac
Lagrangian which occur where additional Landau levels start being populated.
These sharp changes are divergent at zero temperature, but are finite for
nonzero temperature, so realistic neutron star core temperatures $5\times10^7$
K$<T<5\times10^8$ K are used. We conjecture that this mechanism could promote
the formation of magnetic domains as predicted by Blandford and Hernquist
(1982) and Suh and Mathews (2010).
