This paper introduces a general framework for explicitly constructing universal deep neural models with inputs from a complete, separable, and locally-compact metric space \mathcal{X outputs in the Wasserstein-1  P_1( \mathcal{Y . We find that any model built using the proposed framework is dense in the space C(\mathcal{X continuous functions from \mathcal{X to \mathcal{P}_1( \mathcal{Y . We identify two methods in which the curse of dimensionality can be broken.The first approach constructs subsets of C(\mathcal{X ,\mathcal{P}_1( \mathcal{Y can be efficiently approximated . In the second approach ^d, on which any function in C(\mathbb{R}^d} , given any fixed f \in C(\mathcal{X \mathcal{P}_1( \mathcal{Y)) , we build non-trivial subsets of \mathcal{X} on which f }  can be efficiently approximated. The results are applied to three open problems lying at the interface of applied probability and computational learning theory. We find that the proposed models can approximate any regular conditional distribution of a \mathcal{Y-valued random element Y depending on an \mathcal{X}-valued random element X, with arbitrarily high probability. The proposed models are also shown to be capable of generically expressing the aleatoric uncertainty present in most randomized machine learning models. The proposed }  framework is used to derive an affirmative answer to the open conjecture of Bishop (1994); namely: mixture density networks are generic regular conditional distributions. Numerical experiments are performed in the contexts of extreme learning machines, randomized DNNs , and heteroscedastic regression  .
We introduce a general framework for approximating regular conditional distributions (RCDs). Our approximations of these RCDs are implemented by a new class of geometric deep learning models with inputs in \mathbb{R outputs in the Wasserstein-1 space P_1( \mathbb{R . We find that the models built using our framework can approximate any continuous functions from \mathbb{R to \mathcal{P}_1( \mathbb{R . We identify two methods for avoiding the " curse of dimensionality "; i.e.: the number of parameters determining the approximating neural network depends only polynomially on the involved dimension and the approximation error. The first solution describes functions in C(\mathbb{R ,\mathcal{P}_1( \mathbb{R can be efficiently approximated on any compact subset of \mathbb{R the second approach describes sets in \mathbb{R^d, on which any function in C(\mathbb{R}^d} , \mathcal{P}_1( )) , we build non-trivial subsets of \mathcal{X} on which f } \mathbb{R can be efficiently approximated. -valued random element Y depending on an \mathcal{X}-valued random element X, with arbitrarily high probability. The proposed models are also shown to be capable of generically expressing the aleatoric uncertainty present in most randomized machine learning models. The proposed } Our framework is used to obtain an affirmative answer to the open conjecture of Bishop (1994); namely: mixture density networks are universal regular conditional distributions. The predictive performance of the proposed models is evaluated against comparable learning models on various probabilistic predictions tasks in the context of ELMs, model uncertainty , and heteroscedastic regression . All the results are obtained for more general input and output spaces and thus apply to geometric deep learning contexts .
R
This paper introduces
We introduce
R
explicitly constructing universal deep neural
approximating regular conditional distributions (RCDs). Our approximations of these RCDs are implemented by a new class of geometric deep learning
R
from a complete, separable, and locally-compact metric space \mathcal{X
in \mathbb{R
A
space
R
\mathcal{Y
\mathbb{R
R
any model built using the proposed framework is dense in the space C(\mathcal{X
the models built using our framework can approximate any
R
\mathcal{X
\mathbb{R
R
\mathcal{Y
\mathbb{R
R
in which the
for avoiding the "
R
can be broken.The first approach constructs subsets of C(\mathcal{X
"; i.e.: the number of parameters determining the approximating neural network depends only polynomially on the involved dimension and the approximation error. The first solution describes functions in C(\mathbb{R
R
\mathcal{Y
\mathbb{R
R
. In
on any compact subset of \mathbb{R
A
describes sets in \mathbb{R
D
given any fixed f \in C(\mathcal{X
D
\mathcal{Y
A
\mathbb{R
D
The results are applied to three open problems lying at the interface of applied probability and computational learning theory. We find that the proposed models can approximate any regular conditional distribution of a \mathcal{Y
A
Our
R
derive
obtain
R
generic
universal
R
Numerical experiments are performed in the contexts of extreme learning machines, randomized DNNs
The predictive performance of the proposed models is evaluated against comparable learning models on various probabilistic predictions tasks in the context of ELMs, model uncertainty
A
. All the results are obtained for more general input and output spaces and thus apply to geometric deep learning contexts