  Model Predictive Controllers (MPC) require a good model for the controlled
process. In this paper I infer inductive biases about a physical system. I use
these biases to derive a new neural network architecture that can model this
real system that has noise and inertia. The main inductive biases exploited
here are: the delayed impact of some inputs on the system and the separability
between the temporal component and how the inputs interact to produce the
output of a system. The inputs are independently delayed using shifted
convolutional kernels. Feature interactions are modelled using a fully
connected network that does not have access to temporal information. The
available data and the problem setup allow the usage of Self Supervised
Learning in order to train the models. The baseline architecture is an
Attention based Reccurent network adapted to work with MPC like inputs. The
proposed networks are faster, better at exploiting larger data volumes and are
almost as good as baseline networks in terms of prediction performance. The
proposed architecture family called Delay can be used in a real scenario to
control systems with delayed responses with respect to its controls or inputs.
Ablation studies show that the presence of delay kernels are vital to obtain
any learning in proposed architecture. Code and some experimental data are
available online.
