  The design and testing of supervised machine learning models combine two
fundamental distributions: (1) the training data distribution (2) the testing
data distribution. Although these two distributions are identical and
identifiable when the data set is infinite; they are imperfectly known (and
possibly distinct) when the data is finite (and possibly corrupted) and this
uncertainty must be taken into account for robust Uncertainty Quantification
(UQ). We present a general decision-theoretic bootstrapping solution to this
problem: (1) partition the available data into a training subset and a UQ
subset (2) take $m$ subsampled subsets of the training set and train $m$ models
(3) partition the UQ set into $n$ sorted subsets and take a random fraction of
them to define $n$ corresponding empirical distributions $\mu_{j}$ (4) consider
the adversarial game where Player I selects a model $i\in\left\{
1,\ldots,m\right\} $, Player II selects the UQ distribution $\mu_{j}$ and
Player I receives a loss defined by evaluating the model $i$ against data
points sampled from $\mu_{j}$ (5) identify optimal mixed strategies
(probability distributions over models and UQ distributions) for both players.
These randomized optimal mixed strategies provide optimal model mixtures and UQ
estimates given the adversarial uncertainty of the training and testing
distributions represented by the game. The proposed approach provides (1) some
degree of robustness to distributional shift in both the distribution of
training data and that of the testing data (2) conditional probability
distributions on the output space forming aleatory representations of the
uncertainty on the output as a function of the input variable.
