  A conventional approach to train neural ordinary differential equations
(ODEs) is to fix an ODE solver and then learn the neural network's weights to
optimize a target loss function. However, such an approach is tailored for a
specific discretization method and its properties, which may not be optimal for
the selected application and yield the overfitting to the given solver. In our
paper, we investigate how the variability in solvers' space can improve neural
ODEs performance. We consider a family of Runge-Kutta methods that are
parameterized by no more than two scalar variables. Based on the solvers'
properties, we propose an approach to decrease neural ODEs overfitting to the
pre-defined solver, along with a criterion to evaluate such behaviour.
Moreover, we show that the right choice of solver parameterization can
significantly affect neural ODEs models in terms of robustness to adversarial
attacks. Recently it was shown that neural ODEs demonstrate superiority over
conventional CNNs in terms of robustness. Our work demonstrates that the model
robustness can be further improved by optimizing solver choice for a given
task. The source code to reproduce our experiments is available at
https://github.com/juliagusak/neural-ode-metasolver.
