  Bayesian neural networks are theoretically well-understood only in the
infinite-width limit, where Gaussian priors over network weights yield Gaussian
priors over network outputs. Recent work has suggested that finite Bayesian
networks may outperform their infinite counterparts, but their non-Gaussian
output priors have been characterized only though perturbative approaches.
Here, we derive exact solutions for the output priors for individual input
examples of a class of finite fully-connected feedforward Bayesian neural
networks. For deep linear networks, the prior has a simple expression in terms
of the Meijer $G$-function. The prior of a finite ReLU network is a mixture of
the priors of linear networks of smaller widths, corresponding to different
numbers of active units in each layer. Our results unify previous descriptions
of finite network priors in terms of their tail decay and large-width behavior.
