Prior work has focused on evaluating the ability of neural networks to reason about novel combinations from known components, an intrinsic property of human cognition. In this work, we aim to study systematic generalization in predicting future state trajectories of a dynamical system, conditioned on past states' trajectory (dependent variables), past and future actions (control variables). In our context, systematic generalization implies that a good model should perform well on all new combinations of future actions after being trained on all of them, but only on a limited set of their combinations. For models to generalize out-of-distribution to unseen action combinations , they should reason about the states and their dependency relation with the applied actions. We conduct a rigorous study of useful inductive biases that learn to predict the trajectories up to large horizons well, and capture true dependency relations between the states and the controls through our synthetic setup, and simulated data from electric motors .
Systematic generalization aims to evaluate reasoning about novel combinations from known components, an intrinsic property of human cognition. In this work, we study systematic generalization of NNs in forecasting future time series of dependent variables in a dynamical system, conditioned on past time series of dependent variables, and past and future control variables. We focus on systematic generalization wherein the NN-based forecasting model should perform well on previously unseen combinations or regimes of control variables after being trained on a limited set of the possible regimes. For NNs to depict such out-of-distribution generalization , they should be able to disentangle the various dependencies between control variables and dependent variables. We hypothesize that a modular NN architecture guided by the readily-available knowledge of independence of control variables as a potentially useful inductive bias to this end. Through extensive empirical evaluation on a toy dataset and a simulated electric motor dataset, we show that our proposed modular NN architecture serves as a simple yet highly effective inductive bias that enabling better forecasting of the dependent variables up to large horizons in contrast to standard NNs, and indeed capture the true dependency relations between the dependent and the control variables .
R
Prior work has focused on evaluating the ability of neural networks to reason
Systematic generalization aims to evaluate reasoning
D
aim to
R
in predicting future state trajectories of
of NNs in forecasting future time series of dependent variables in
R
states' trajectory (dependent variables),
time series of dependent variables, and
R
actions (control variables). In our context, systematic generalization implies that a good
control variables. We focus on systematic generalization wherein the NN-based forecasting
R
all new combinations of future actions
previously unseen combinations or regimes of control variables
D
all of them, but only on
R
their combinations. For models to generalize
the possible regimes. For NNs to depict such
R
to unseen action combinations
generalization
R
reason about the states and their dependency relation with the applied actions. We conduct a rigorous study of useful inductive biases that learn to predict the trajectories
be able to disentangle the various dependencies between control variables and dependent variables. We hypothesize that a modular NN architecture guided by the readily-available knowledge of independence of control variables as a potentially useful inductive bias to this end. Through extensive empirical evaluation on a toy dataset and a simulated electric motor dataset, we show that our proposed modular NN architecture serves as a simple yet highly effective inductive bias that enabling better forecasting of the dependent variables
R
well, and capture
in contrast to standard NNs, and indeed capture the
R
states and the controls through our synthetic setup, and simulated data from electric motors
dependent and the control variables