  Performance variability is an important measure for a reliable high
performance computing (HPC) system. Performance variability is affected by
complicated interactions between numerous factors, such as CPU frequency, the
number of input/output (IO) threads, and the IO scheduler. In this paper, we
focus on HPC IO variability. The prediction of HPC variability is a challenging
problem in the engineering of HPC systems and there is little statistical work
on this problem to date. Although there are many methods available in the
computer experiment literature, the applicability of existing methods to HPC
performance variability needs investigation, especially, when the objective is
to predict performance variability both in interpolation and extrapolation
settings. A data analytic framework is developed to model data collected from
large-scale experiments. Various promising methods are used to build predictive
models for the variability of HPC systems. We evaluate the performance of the
methods by measuring prediction accuracy at previously unseen system
configurations. We also discuss a methodology for optimizing system
configurations that uses the estimated variability map. The findings from
method comparisons and developed tool sets in this paper yield new insights
into existing statistical methods and can be beneficial for the practice of HPC
variability management. This paper has supplementary materials online.
