Neural SDEs combine many of the best qualities of both RNNs and SDEs , and as such are a natural choice for modelling many types of temporal dynamics. They offer memory efficiency , high-capacity function approximation, and strong priors on model space. Neural SDEs may be trained as VAEs or as GANs; in either case it is necessary to backpropagate through the SDE solve. In particular this may be done by constructing a backwards-in-time SDE whose solution is the desired parameter gradients. However, this has previously suffered from severe speed and accuracy issues, due to high computational complexity, numerical errors in the SDE solve, and the cost of reconstructing Brownian motion . Here, we make several technical innovations to overcome these issues  . First, we introduce the reversible Heun method: a new SDE solver that is algebraically reversible -- which reduces \textit{  numerical gradient errors to almost zero, improving several test metrics by substantial margins over state-of-the-art . Moreover it requires half as many function evaluations as comparable solvers, giving up to a 1.98\times speedup. Next , we introduce the Brownian interval. This is a newand computationally efficientway of exactly sampling and reconstructing Brownian motion; this is in contrast to previous reconstruction techniques that are both approximate and relatively slow. This gives \textit{ \textit{  up to a 10.6\times speed improvement over previous techniques . After that , when specifically training Neural SDEs as GANs (Kidger et al. 2021), we demonstrate how SDE-GANs may be trained through careful weight clipping and choice of activation function. This reduces computational cost (giving up to a 1.87\times speedup) , and removes the truncation errors of the double adjoint required for gradient penalty, substantially improving several test metrics. Altogether these techniques offer substantial improvements over the state-of-the-art .
Neural SDEs combine many of the best qualities of both RNNs and SDEs : memory efficient training , high-capacity function approximation, and strong priors on model space. This makes them a natural choice for modelling many types of temporal dynamics. Training a Neural SDE (either as a VAE or as a GAN) requires backpropagating through an SDE solve. This may be done by solving a backwards-in-time SDE whose solution is the desired parameter gradients. However, this has previously suffered from severe speed and accuracy issues, due to high computational cost and numerical truncation errors . Here, we overcome these issues through several technical innovations . First, we introduce the reversible Heun method. This is a new SDE solver that is \textit{algebraically reversible : eliminating numerical gradient errors , and the first such solver of which we are aware . Moreover it requires half as many function evaluations as comparable solvers, giving up to a 1.98\times speedup. Second , we introduce the \textit{Brownian Interval : a new, fast, memory efficient, and exact way of sampling\textit{and reconstructing Brownian motion. With this we obtain up to a 10.6\times speed improvement over previous techniques , which in contrast are both approximate and relatively slow. Third , when specifically training Neural SDEs as GANs (Kidger et al. 2021), we demonstrate how SDE-GANs may be trained through careful weight clipping and choice of activation function. This reduces computational cost (giving up to a 1.87\times speedup) and removes the numerical truncation errors associated with gradient penalty. Altogether, we outperform the state-of-the-art by substantial margins, with respect to training speed, and with respect to classification, prediction, and MMD test metrics. We have contributed implementations of all of our techniques to the torchsde library to help facilitate their adoption .
R
, and as such are a natural choice for modelling many types of temporal dynamics. They offer memory efficiency
: memory efficient training
R
Neural SDEs may be trained as VAEs or as GANs; in either case it is necessary to backpropagate through the
This makes them a natural choice for modelling many types of temporal dynamics. Training a Neural SDE (either as a VAE or as a GAN) requires backpropagating through an
R
In particular this
This
R
constructing
solving
R
complexity, numerical errors in the SDE solve, and the cost of reconstructing Brownian motion
cost and numerical truncation errors
D
make several technical innovations to
A
through several technical innovations
R
reversible Heun method:
reversible Heun method
A
. This is
D
algebraically reversible -- which reduces
A
algebraically reversible
A
: eliminating
R
to almost zero, improving several test metrics by substantial margins over state-of-the-art
, and the first such solver of which we are aware
R
Next
Second
D
Brownian interval. This is a newand computationally efficientway of exactly sampling and reconstructing Brownian motion; this is in contrast to previous reconstruction techniques that are both approximate and relatively slow. This gives
A
Brownian Interval
A
: a new, fast, memory efficient, and exact way of sampling
A
and reconstructing
A
Brownian motion. With this we obtain
R
. After that
, which in contrast are both approximate and relatively slow. Third
D
,
R
truncation errors of the double adjoint required for gradient penalty, substantially improving several
numerical truncation errors associated with gradient penalty. Altogether, we outperform the state-of-the-art by substantial margins, with respect to training speed, and with respect to classification, prediction, and MMD
R
Altogether these techniques offer substantial improvements over the state-of-the-art
We have contributed implementations of all of our techniques to the torchsde library to help facilitate their adoption