  We study the Hall response of topologically-trivial mobile impurities (Fermi
polarons) interacting weakly with majority fermions forming a Chern-insulator
background. This setting involves a rich interplay between the genuine
many-body character of the polaron problem and the topological nature of the
surrounding cloud. When the majority fermions are accelerated by an external
field, a transverse impurity current can be induced. To quantify this polaronic
Hall effect, we compute the drag transconductivity, employing controlled
diagrammatic perturbation theory in the impurity-fermion interaction. We show
that the impurity Hall drag is not simply proportional to the Chern number
characterizing the topological transport of the insulator on its own - it also
depends continuously on particle-hole breaking terms, to which the Chern number
is insensitive. However, when the insulator is tuned across a topological phase
transition, a sharp jump of the impurity Hall drag results, for which we derive
an analytical expression. We describe how the Hall drag and jump can be
extracted from a circular dichroic measurement of impurity excitation rates,
particularly suited for ultracold gas experiments.
