  Most distributed-memory bulk-synchronous parallel programs in HPC assume that
compute resources are available continuously and homogeneously across the
allocated set of compute nodes. However, long one-off delays on individual
processes can cause global disturbances, so-called idle waves, by rippling
through the system. This process is mainly governed by the communication
topology of the underlying parallel code. This paper makes significant
contributions to the understanding of idle wave dynamics. We study the
propagation mechanisms of idle waves across the ranks of MPI-parallel programs.
We present a validated analytic model for their propagation velocity with
respect to communication parameters and topology, with a special emphasis on
sparse communication patterns. We study the interaction of idle waves with MPI
collectives and show that, depending on the implementation, a collective may be
transparent to the wave. Finally we analyze two mechanisms of idle wave decay:
topological decay, which is rooted in differences in communication
characteristics among parts of the system, and noise-induced decay, which is
caused by system or application noise. We show that noise-induced decay is
largely independent of noise characteristics but depends only on the overall
noise power. An analytic expression for idle wave decay rate with respect to
noise power is derived. For model validation we use microbenchmarks and stencil
algorithms on three different supercomputing platforms.
