  Assessing goodness of fit to a given distribution plays an important role in
computational statistics. The Probability integral transformation (PIT) can be
used to convert the question of whether a given sample originates from a
reference distribution into a problem of testing for uniformity. We present new
simulation and optimization based methods to obtain simultaneous confidence
bands for the whole empirical cumulative distribution function (ECDF) of the
PIT values under the assumption of uniformity. Simultaneous confidence bands
correspond to such confidence intervals at each point that jointly satisfy a
desired coverage. These methods can also be applied in cases where the
reference distribution is represented only by a finite sample. The confidence
bands provide an intuitive ECDF-based graphical test for uniformity, which also
provides useful information on the quality of the discrepancy. We further
extend the simulation and optimization methods to determine simultaneous
confidence bands for testing whether multiple samples come from the same
underlying distribution. This multiple sample comparison test is especially
useful in Markov chain Monte Carlo convergence diagnostics. We provide
numerical experiments to assess the properties of the tests using both
simulated and real world data and give recommendations on their practical
application in computational statistics workflows.
