  We study the polaritonic bandstructure of two-dimensional atomic lattices
coupled to a single excitation of a surface plasmon polariton mode. We show the
possibility of realizing topological gaps with different Chern numbers by
having resonant atomic transitions to excited states with different angular
momentum. We employ a computational method based on the recently proposed
Dirichlet-to-Neumann (DtN) map technique which accurately models non-Markovian
dynamics as well as interactions involving higher-order electric and magnetic
multipole transitions. We design topologically robust edge states which are
used to achieve unidirectional emission and non-reciprocal transmission of
single photons. We also point out the challenges in realizing bands with higher
Chern numbers in such systems.
