Paper:
In this paper we investigate an artificial agent's ability to perform task-focused tool synthesis via imagination. Our motivation is to explore the richness of information captured by the latent space of an object-centric generative model -and how to exploit it. In particular, our approach employs activation maximisation of a task-based performance predictor to optimise the latent variable of a structured latent-space model in order to generate tool geometries appropriate for the task at hand. We evaluate our model using a novel dataset of synthetic reaching tasks inspired by the cognitive sciences and behavioural ecology. In doing so we examine the model's ability to imagine tools for increasingly complex scenario types, beyond those seen during training. Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way: the agents often specifically modify aspects of the tools which relate to meaningful (yet implicitly learned) concepts such as a tool's length, width and configuration. Our results therefore suggest, that task relevant object affordances are implicitly encoded as directions in a structured latent space shaped by experience.

Review:
This paper proposes an architecture for synthesizing tools to be used in a reaching task. Specifically, during training the agent jointly learns to segment an image of a set of three tools (via the MONet architecture) and to classify whether one the tools will solve the given scene. At test time, one of the three tools is selected based on which seems most feasible, and then gradient descent is used to modify the latent representation of the tool in order to synthesize a new tool to (hopefully) solve the scene. The paper demonstrates that this approach can achieve ok performance on familiar scenes with familiar tools, but that it fails to generalize when exposed to unfamiliar scenes or unfamiliar tools. The paper reports a combination of the quantitative results showing that optimizing the latent space can lead to successful synthesis in some cases, and qualitative results showing that the synthesized tools change along interpretable dimensions such as length, width, etc. The combination of these results suggest that the model has learned something about which tool dimensions are important for being able to solve the types of reaching tasks given in the paper.
While I think this paper tackles a very interesting, important, and challenging problem, I unfortunately feel it is not ready for publication at ICLR and thus recommend rejection. Specifically, (1) neither the particular task, results, or model are not very compelling, (2) there are no comparisons to meaningful alternatives, and (3) overall I am not quite sure what conclusions I should draw from the paper. However, given the coolness of the problem of tool synthesis, I definitely encourage the authors to continue working on this line of work!
1. The task, results, and model are not very compelling. Any of these three things alone would not necessarily be a problem, but given that all three are true the paper comes across as a bit underwhelming.
 
- First, while the task can be construed as a tool synthesis task, it doesn’t come across to me as very ecologically valid. In fact, the task seems to be more like a navigation task than a tool synthesis task: what’s required is simply to draw an unbroken line from one part of the scene to another, rather than actually generate a tool that has to be manipulated in an interesting way. Navigation has been studied extensively, while synthesis of tools that can be manipulated has not, which makes this task both not very novel and disappointing in comparison to what more ecologically-valid tool synthesis would look like. For example, consider a variation of the task where you would have to start the tool at the red region and move it to the green region. Many of the tools used here would become invalid since you wouldn’t actually be able to fit them through the gaps (e.g. Figure 2E).
 
- Second, given that the “synthesis” task is more like a navigation task, the results are somewhat disappointing. When provided with a feasible solution, the model actually gets *worse* even in some of the in-sample scenes that it has seen during training (e.g. scene types C and D) which suggests that it hasn’t actually learned a good generative model of tools. Generalization performance is pretty bad across the board and is only slightly better than random, which undermines the claim in the abstract that “Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way”. While it’s clear there is successful synthesis in some cases, I am not sure that the results support the claim that the synthesis is “targeted” or “deliberate” given how poor the overall performance is.
 
- Third, the model/architecture is a relatively straightforward combination of existing components and is highly specialized to the particular task. As mentioned above, this wouldn’t necessarily be a problem if the task were more interesting (i.e. not just a navigation task) and if the results were better. I do think it is cool to see this use of MONet but I’m skeptical that the particular method of optimizing in the latent space is doing anything meaningful. While there is prior work that has optimized the latent space to achieve certain tasks (as is cited in the paper), there is also a large body of work on adversarial examples which demonstrate that optimizing in the latent space is also fraught with difficulty. I also suspect this is the reason why the results are not particularly good.
 
2. While I do appreciate the comparisons that are in the paper (to a “Random” version of TasMON that moves in a random direction in the latent space, and to “FroMON” agent which is not allowed to backpropagate gradients from the classification loss into MONet), these comparisons are not particularly meaningful. The difference between FroMON performance and TasMON tool imagination performance (I didn’t test tool utility) across tasks is not statistically significant (z(520, 544)=-0.8588, p=.38978), so I don’t think it is valid to claim that “a task-aware latent space can still provide benefits.” The Random baseline is a pretty weak baseline and it would be more interesting to compare to an alternative plausible architecture (for example, which doesn’t use a structured latent space, or which doesn’t have a perceptual frontend and operates directly on a symbolic representation of the tools/scene).
 
3. Overall, I am not quite sure what I am supposed to get out of the paper. Is it that “task relevant object affordances are implicitly encoded as directions in a structured latent space shaped by experience”? If so, then the results do not support this claim and so I am not sure what to take away. Is it that the latent space encodes information about what makes a tool feasible? If so, then this is a bit of a weak argument---of *course* it must encode this information if it is able to do the classification task at all. Is it that tool synthesis is a challenging problem? If so, then the lack of strong or canonical baselines makes it hard to evaluate whether this is true (and the navigation-only synthesis task also undermines this a bit).
Some additional suggestions:
 
It would be good to include a discussion of other recent work on tool use such as Allen et al. (2019) and Baker et al. (2019), as well as on other related synthesis tasks such as Ha (2018) or Ganin et al. (2018).
 
The introduction states that “tool selection and manufacture – especially once demonstrated – is a significantly easier task than tool innovation”. While this may be true, it is a bit misleading in the context of the paper as the agent is doing something more like tool selection and modification rather than tool innovation (and actually the in-sample scenes are more like “manufacture”, which the agent doesn’t always even do well on).
 
It would be helpful to more clearly explain scene types. Here is some suggested phrasings: in-sample = familiar scenes with familiar tools, interpolation = novel scenes with familiar tools, extrapolation = novel scenes with novel tools.
 
I was originally confused how psi’ knew where to actually place the tool and at what orientation, and whether the background part of the rendering process shown in Figure 1. I realized after reading the supplemental that this is not done by the agent itself but by separate code that tries to find the orientation and position of the tool. This should be explained more clearly in the main text.
 
In Table 1 it would be helpful to indicate which scene types are which (in-sample, interpolation, extrapolation).
 
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2019). The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving. arXiv preprint arXiv:1907.09620.
 
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528.
 
Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S. M., & Vinyals, O. (2018). Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118.
 
Ha, D. (2018). Reinforcement learning for improving agent design. arXiv preprint arXiv:1810.03779.