Paper:
Interactive Control of Diverse Complex Characters with Neural Networks
We present a method for training recurrent neural networks to act as near-optimal feedback controllers. It is able to generate stable and realistic behaviors for a range of dynamical systems and tasks -swimming, flying, biped and quadruped walking with different body morphologies. It does not require motion capture or task-specific features or state machines. The controller is a neural network, having a large number of feed-forward units that learn elaborate state-action mappings, and a small number of recurrent units that implement memory states beyond the physical system state. The action generated by the network is defined as velocity. Thus the network is not learning a control policy, but rather the dynamics under an implicit policy. Essential features of the method include interleaving supervised learning with trajectory optimization, injecting noise during training, training for unexpected changes in the task specification, and using the trajectory optimizer to obtain optimal feedback gains in addition to optimal actions. Figure 1: Illustration of the dynamical systems and tasks we have been able to control using the same method and architecture. See the video accompanying the submission.

Review:
In this submission, authors propose an algorithm for real-time control of 3D models, where a neural network (NN) is used to generate animations, after having been trained offline to reproduce the output of Contact-Invariant Optimization (CIO). Importantly, CIO and NN training are not independent: they are performed in an alternating fashion, the output of one being used in the criterion optimized by the other. Another core element of the method is the injection of noise, through data augmentation by small perturbations as well as additive noise in the NN's hidden layers. Experiments show that such a network is able to produce realistic and stable control policies on a range of very different character models.This is a very interesting approach, considering that real-time control of arbitrary models is a challenging task. Although the proposed method may initially appear relatively straightforward ("learn a NN to predict the output of CIO"), the additional steps to make it work (joint training and noise injection) are significant new contributions, and shown in experiments to help get much better generalization.Those experimental results would be stronger, however, if comparisons were made with a larger dataset. Since the base training trajectories were generated with only "between 100 to 200 trials, each with 5 branched segments", it is not surprising for the "no noise" variant to overfit. The "no joint" variant might also benefit from more diverse training trajectories -- even if that is less obvious.Overall the comparative evaluation is the weak point of the paper (acknowledged in the introduction: "A systematic comparison of these more direct methods with the present trajectory-optimization-based methods remains to be done"). The only comparison to a competing method is a quick one with MPC at the end of the paper (and there is no reference for this method by the way).For the most part, the maths are clearly explained and motivated, although someone willing to replicate these results will definitely need to read up additional references, since some points are only addressed superficially (especially the CIO step -- see Eq. 6 whose notations are not defined). Providing code would certainly be much appreciated by the community. There is one part I was not able to properly understand: section 3.2 about generating optimal actions for noisy inputs. Notations got really confusing for me at this point, and it was not clear to me what was being done (in particular the time index "t" was dropped, but is it still implied for some of the quantities?).It would have also been useful to get some time measurements for computations during real-time control, since that could be a major bottleneck (and also knowing how these computations scale with the complexity of the 3D model). In an application like a video game, there is not much CPU available to animate a single character on screen.Additional minor remarks: - "biped location": locomotion? - "It is our aim to unity these disparate approaches": unite? unify? - Am I right that the target is only about x,y,z coordinates, and there is no constraint on the angle the character is facing when reaching it? If yes, would it be easy to add an extra target angle? (which seems important in practice) - I would not call "noise injection" the step where additional data is generated from noisy inputs, because the target is re-computed to match the modified inputs. To me, this is simply "data augmentation", and it has nothing to do with the idea behind denoising autoencoders. - Are you really using the initial (random) network weights theta in the very first iteration, to compute the first X's in Alg. 1? Or are you starting with X's computed without the regularization term? - What is the justification for using the same hyperparameter eta in the two steps of Alg. 1? - "it is non-trivial to adapt them to asynchronous and stochastic setting we have": missing "the"? - Acronym LQG is used one line before it is defined. - "While we can use Linear Quadratic Gaussian (LQG) pass": missing "a"? - l. 231 the star in the argmin is in the wrong place, and is \bar{s} missing in \tilde{C}? - "the minimizer of a quadratic around optimal trajectory": missing a word? - "sX and aX are subsets of \phiX": why is that guaranteed? - "a single pass over the data to reduce the objective in (4)": it would rather be the objective from Alg. 1 since only theta is optimized - The asynchronous joint optimization scheme described in Section 6 is very different from the alternating scheme from Alg. 1. It would be worth at least mentioning this earlier. - l. 315, f should be f^{des}? - "In the current work, the dynamics constraints are enforced softly and thus may include some root forces in simulation.": I did not understand this sentence - The "no noise" variant in experiments could be split between "no data augmentation" and "no hidden noise injection". - A question I would find interesting to investigate is whether alpha can be decreased all the way to zero during optimization (for CIO only). In other words, is it working mostly because of a "curriculum learning" effect (easier task initially), or is it always useful to keep the trajectories learned by CIO close to what the network can learn? - Caption of Table 1a says 4 layers but I believe this is 3