Paper:
We propose a method to learn stochastic activation functions for use in probabilistic neural networks. First, we develop a framework to embed stochastic activation functions based on Gaussian processes in probabilistic neural networks. Second, we analytically derive expressions for the propagation of means and covariances in such a network, thus allowing for an efficient implementation and training without the need for sampling. Third, we show how to apply variational Bayesian inference to regularize and efficiently train this model. The resulting model can deal with uncertain inputs and implicitly provides an estimate of the confidence of its predictions. Like a conventional neural network it can scale to datasets of arbitrary size and be extended with convolutional and recurrent connections, if desired.

Review:
In Bayesian neural networks, a deterministic or parametric activation is typically used. In this work, activation functions are considered random functions with a GP prior and are inferred from data.
- Unnecessary complexity
The presentation of the paper is unnecessarily complex. It seems that authors spend extra space creating problems and then solving them. Although some of the derivations in Section 3.2.2 are a bit involved, most of the derivations up to that point (which is already in page 6) follow preexisting literature.
For instance, eq. (3) proposes one model for p(F|X). Eq. (8) proposes a different model for p(F|X), which is an approximation to the previous one. Instead, the second model could have been proposed directly, with the appropriate citation from the literature, since it isn't new. Eq. (13) is introduced as a "solution" to a non-existent problem, because the virtual observations are drawn from the same prior as the real ones, so it is not that we are "coming up" with a convenient GP prior that turns out to produce a computationally tractable solution, we are just using the prior on the observations consistently.
In general, the authors seem to use "approximately equal" and "equal" interchangeably, which is incorrect. There should be a single definition for p(F|X). And there should be a single definition for L_pred. The expression for L_pred given in eq. (20) (exact) and eq. (41) (approximate) do not match and yet both are connected with an equality (or proportionality), which they shouldn't.
Q(A) is sometimes taken to mean the true posterior (i.e., eq. (31)), sometimes a Gaussian approximation (i.e., eq (32) inside the integral), and both are used interchangeably.
- Incorrect references to the literature
Page 3: "using virtual observations (originally proposed by Quiñonero-Candela & Rasmussen (2005) for sparse approximations of GPs)"
The authors are citing as the origin of virtual observations a survey paper on the topic. Of course, that survey paper correctly attributes the origin to [1].
Page 4: "we apply the technique of variational inference Wainwright et al. (2008)".
How can variational inference be attributed to (again) a survey paper on the topic from 2008, when for instance [2] appeared in 2003?
- Correctness of the approach
Can the authors guarantee that the variational bound that they are introducing (as defined in eqs. (19) and (41)) is actually a variational bound? It seems to me that the approximations made to Q(A) to propagate the uncertainty are breaking the bounding guarantee. If it is no longer a lower bound, what is the rationale behind maximizing it?
The mathematical basis for this paper is actually introduced in [3] and a single-layer version of the current model is developed in [4]. However, in [4] the authors manage to avoid the additional Q(A) approximation that breaks the variational bound. The authors should contrast their approach with [4] and discuss if and why that additional central limit theorem application is necessary.
- No experiments
The use of a non-parametric definition for the activation function should be contrasted with the use of a parametric one. With enough data, both might produce similar results. And the parameter sharing in the parametric one might actually be beneficial. With no experiments at all showing the benefit of this proposal, this paper cannot be considered complete.
- Minor errors:
Eq. (4), for consistency, should use the identity matrix for the covariance matrix definition.
Eq. (10) uses subscript d where it should be using subscript n
Eq. (17) includes p(X^L|F^L) in the definition of Q(...), but it shouldn't. That was particularly misleading, since if we take eq. (17) to be correct (which I did at first), then p(X^L|F^L) cancels out and should not appear in eq. (20).
Eq. (23) uses Q(F|A) to mean the same as P(F|A) as far as I understand. Then why use Q?
- References
[1] Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs.
[2] Beal, M.J. Variational Algorithms for Approximate Bayesian Inference.
[3] M.K. Titsias and N.D. Lawrence. Bayesian Gaussian process latent variable model. 
[4] M. Lázaro-Gredilla. Bayesian warped Gaussian processes.