  The most promising avenue for detecting primordial gravitational waves from
cosmic inflation is through measurements of degree-scale CMB $B$-mode
polarisation. This approach must face the challenge posed by gravitational
lensing of the CMB, which obscures the signal of interest. Fortunately, the
lensing effects can be partially removed by combining high-resolution $E$-mode
measurements with an estimate of the projected matter distribution. For
near-future experiments, the best estimate of the latter will arise from
co-adding internal reconstructions (derived from the CMB itself) with external
tracers of the large-scale structure such as the cosmic infrared background
(CIB). In this work, we characterise how foregrounds impact the delensing
procedure when CIB intensity, $I$, is used as the matter tracer. We find that
higher-point functions of the CIB and Galactic dust such as $\langle BEI
\rangle_{c}$ and $\langle EIEI \rangle_{c}$ can, in principle, bias the power
spectrum of delensed $B$-modes. After estimating the dust residuals in
currently-available CIB maps and upcoming, foreground-cleaned Simons
Observatory CMB data, we find, using non-Gaussian dust simulations, that the
bias to any primordial signal is small compared to statistical errors for
ground-based experiments, but might be significant for space-based experiments
probing very large angular scales. However, mitigation techniques based on
multi-frequency cleaning appear to be very effective. We also show, by means of
an analytic model, that the bias arising from the higher-point functions of the
CIB itself ought to be negligible.
