  Fe, Mg, and O are among the most abundant elements in terrestrial planets.
While the behavior of the Fe-O, Mg-O, and Fe-Mg binary systems under pressure
have been investigated, there are still very few studies of the Fe-Mg-O ternary
system at relevant Earth's core and super-Earth's mantle pressures. Here, we
use the adaptive genetic algorithm (AGA) to study ternary Fe$_x$Mg$_y$O$_z$
phases in a wide range of stoichiometries at 200 GPa and 350 GPa. We discovered
three dynamically stable phases with stoichiometries FeMg$_2$O$_4$,
Fe$_2$MgO$_4$, and FeMg$_3$O$_4$ with lower enthalpy than any known combination
of Fe-Mg-O high-pressure compounds at 350 GPa. With the discovery of these
phases, we construct the Fe-Mg-O ternary convex hull. We further clarify the
composition- and pressure-dependence of structural motifs with the analysis of
the AGA-found stable and metastable structures. Analysis of binary and ternary
stable phases suggest that O, Mg, or both could stabilize a BCC iron alloy at
inner core pressures.
