  We present a systematic study of higher-order exchange interactions beyond
the pair-wise Heisenberg exchange in transition-metal trilayers based on
density functional theory calculations. We show that these terms can play an
important role in magnetic trilayers composed of a single hexagonal Fe or Co
atomic layer sandwiched between $4d$ and $5d$ transition-metal layers. We study
the dependence of the biquadratic and the three-site and four-site four spin
interaction on the band filling of the $4d$ and $5d$ layers as well as the
stacking sequence, i.e. fcc vs. hcp stacking. Our calculations reveal
relatively small higher-order interactions for Co based trilayers. For Fe based
trilayers with a Rh or Ir layer the higher-order terms can be on the same order
of magnitude as pair-wise Heisenberg exchange. The trends obtained for
freestanding trilayers are used to understand the higher-order interactions in
ultrathin film systems on surfaces that are experimentally accessible. It is
shown that hcp Rh/Fe/Ir(111) and hcp-Rh/Fe/Rh(111) exhibit the largest values
for the biquadratic and the three-site four spin interaction of all systems
under study. We further demonstrate that the three-site four spin interaction
is responsible for the experimentally observed change of the magnetic ground
state of Rh/Fe/Ir(111) from a spin spiral (single-Q) for fcc-Rh to a 2Q state
for hcp-Rh. We find similar trends for Rh/Fe/Rh(111), i.e. replacing the Ir
surface by the isoelectronic Rh surface. For Rh/Co/Ir(111), we obtain a
negative value for the four-site four spin interaction which will lead to a
reduced stability of magnetic skyrmions which are metastable in this film at
zero magnetic field. In contrast, for Pd/Fe/Ir(111), the four-site four spin
interaction is positive which leads to an enhanced stability of skyrmions.
