Paper:
Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning
The physical design of a robot and the policy that controls its motion are inherently coupled. However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs. In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision. Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller. Throughout training, we refine the robot distribution to maximize the expected reward. This results in an assignment to the robot parameters and neural network policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs.

Review:
This is a well written paper, very nice work.
It makes progress on the problem of co-optimization of the physical parameters of a design
and its control system.  While it is not the first to explore this kind of direction,
the method is efficient for what it does; it shows that at least for some systems, 
the physical parameters can be optimized without optimizing the controller for each 
individual configuration. Instead, they require that the same controller works over an evolving
distribution of the agents.  This is a simple-but-solid insight that makes it possible
to make real progress on a difficult problem.
Pros:  simple idea with impact;  the problem being tackled is a difficult one
Cons:  not many;  real systems have constraints between physical dimensions and the forces/torques they can exert
       Some additional related work to consider citing.  The resulting solutions are not necessarily natural configurations, 
   given the use of torques instead of musculotendon-modeling.  But the current system is a great start.
The introduction could also promote that over an evolutionary time-frame, the body and
control system (reflexes, muscle capabilities, etc.) presumably co-evolved.
The following papers all optimize over both the motion control and the physical configuration of the agents.
They all use derivative free optimization, and thus do not require detailed supervision or precise models
of the dynamics.
- Geijtenbeek, T., van de Panne, M., & van der Stappen, A. F. (2013). Flexible muscle-based locomotion
  for bipedal creatures. ACM Transactions on Graphics (TOG), 32(6), 206.
  (muscle routing parameters, including insertion and attachment points) are optimized along with the control).
- Sims, K. (1994, July). Evolving virtual creatures. In Proceedings of the 21st annual conference on
  Computer graphics and interactive techniques (pp. 15-22). ACM.
  (a combination of morphology, and control are co-optimized)
- Agrawal, S., Shen, S., & van de Panne, M. (2014). Diverse Motions and Character Shapes for Simulated
  Skills. IEEE transactions on visualization and computer graphics, 20(10), 1345-1355.
  (diversity in control and diversity in body morphology are explored for fixed tasks)
re: heavier feet requiring stronger ankles
This commment is worth revisiting.  Stronger ankles are more generally correlated with 
a heavier body rather than heavy feet, given that a key role of the ankle is to be able
to provide a "push" to the body at the end of a stride, and perhaps less for "lifting the foot".
I am surprised that the optimization does not converge to more degenerate solutions
given that the capability to generate forces and torques is independent of the actual
link masses, whereas in nature, larger muscles (and therefore larger masses) would correlate
with the ability to generate larger forces and torques.  The work of Sims takes these kinds of 
constraints loosely into account (see end of sec 3.3).
It would be interesting to compare to a baseline where the control systems are allowed to adapt to the individual design parameters.
I suspect that the reward function that penalizes torques in a uniform fashion across all joints would
favor body configurations that more evenly distribute the motion effort across all joints, in an effort
to avoid large torques. 
Are the four mixture components over the robot parameters updated independently of each other
when the parameter-exploring policy gradients updates are applied?  It would be interesting
to know a bit more about how the mean and variances of these modes behave over time during
the optimization, i.e., do multiple modes end up converging to the same mean? What does the
evolution of the variances look like for the various modes?