  Understanding and mitigating loss channels due to two-level systems (TLS) is
one of the main corner stones in the quest of realizing long photon lifetimes
in superconducting quantum circuits. Typically, the TLS to which a circuit
couples are modelled as a large bath without any coherence. Here we demonstrate
that the coherence of TLS has to be considered to accurately describe the
ring-down dynamics of a coaxial quarter-waver resonator with an internal
quality factor of $0.5\times10^9$ at the single-photon level. The transient
analysis reveals an effective non-markovian dynamics of the combined TLS and
cavity system, which we can accurately fit by introducing a comprehensive TLS
model. The fit returns relaxation times around $T_1=0.8\,\mathrm{\mu s}$ for a
total of $N\approx2\times10^{8}$ TLS with power-law distributed coupling
strengths. Despite the short-lived TLS excitations, we observe long-term
effects on the cavity decay due to coherent elastic scattering between the
resonator field and the TLS. The presented method is applicable to various
systems and allows for a simple characterization of TLS properties.
