Paper: A computational approach to visual ecology with deep reinforcement learning (ICLR-2024)

Reviewer comment (Evaluation):
The results are not exactly earth-shattering. More difficult tasks seem to benefit from more complex architecture and additional inputs. If anything, the surprising result is the *low* impact of network size on performance (Figure 2).

Author response:
We do think that the devil is in the details here – we find it interesting the way in which architecture and hyperparmeters interact with the task in different ways. For example, the feedforward network is generally insensitive to hyperparameters, whereas the RNN benefits from larger hyperparameters only on the most visually difficult tasks.

Reviewer comment (Reproducibility):
Although the architecture is reasonably well-described, the training itself is not, with little detail except for a mention of PPO. E.g. what is the reward function, exactly?

Author response:
Although we stated as much in the text, we realize that we explained it poorly: the reward is the current satiety of the agent. We will improve our explanation of the details of the reward and training.

Reviewer comment (Writing):
What's an "ELU" nonlinearity?

Author response:
An “exponential linear unit”. It is similar to a RELU yet everywhere differentiable.

Reviewer comment (Experiments):
My ternary concern follows that, in this specific work, the end-to-end training may be unnecessary, increasing the training time but, potentially, not providing new insights. In the specific case of foraging, most of the works successfully operate on simple state representations, reducing the computational time from months on a GPU cluster to seconds on a laptop. I am familiar with only one work using 3D simulation for a foraging task but, likewise, I didn’t find their case well-argued. Either way, if there is indeed a benefit of using an end-to-end model in the current framework, it would make sense to highlight this benefit via a baseline analysis where the blocks of the same model are trained separately.

Author response:
*My secondary concern is that I do not agree that end-to-end processing is necessarily beneficial for learning the optimal representations in submodules of the neural network. Specifically, for the longest time, it’s been an ongoing debate with no clear outcome. The examples include audio processing (where SOTA was switching between using mel-spectrograms, then deep-learning representations such as the ones in wav2vec, then again considering mel-spectrograms), vision processing (e.g. using feature-based pre-alignment before training the triplet loss in facial recognition), and control (where control submodules are trained end-to-end, e.g. in quadruped robots with manipulators, but vision models are separately pretrained). Related to this point, I think that the aforementioned cases are relevant and need to be discussed in the paper.*

*My ternary concern follows that, in this specific work, the end-to-end training may be unnecessary, increasing the training time but, potentially, not providing new insights. In the specific case of foraging, most of the works successfully operate on simple state representations, reducing the computational time from months on a GPU cluster to seconds on a laptop. I am familiar with only one work using 3D simulation for a foraging task but, likewise, I didn’t find their case well-argued. Either way, if there is indeed a benefit of using an end-to-end model in the current framework, it would make sense to highlight this benefit via a baseline analysis where the blocks of the same model are trained separately.*

This is a very important point, and one we should have better addressed in the paper. Firstly, we must reiterate that the goal of our simulations was not simply to find the agent that achieves the highest performance, but rather to understand the conditions under which various behaviours and representations emerge in an agent. 

The actual RL problem is easy by design, and we do not doubt that it could be solved by a clever combination of pretrained vision model and simpler RL training. In fact, our existing results show this: in the apples task, where there are only two visually distinct objects that are entirely colour separated, agents can more or less maximize their performance after a couple hours of training. Using an appropriately pre-trained vision model should afford similarly fast learning on the other tasks, because it is only the visual statistics that are changing from task to task. 

Nevertheless, we believe end-to-end training is critical to maintaining the ecological validity of our approach for at least two reasons: (i) We don’t want to train the agent on anything it might not experience in its ecological niche, and (ii) we want to ensure that the agent only learns what is relevant for the task, and ignores anything else. Pretraining the vision model to e.g. reconstruct a large image dataset would violate these principles. 

We have a large set of experiments that we cut from the paper due to space constraints: we generated a set of “distractors” tasks, where the environment was filled with e.g. apples and cifar images, where the apples images impacted satiety and cifar did not, and vise versa. What we found was that in terms of how the agent learned and its ability to identify food, the distractors had no effect on the agent. In other words, an agent that only gets nourished by apples does not learn to differentiate cifar images from each other because they are irrelevant. This notion of survival relevance is one of our key interests in our framework, and would not be captured by modular training where the objects of the vision model and policy were separated. 

If you would feel that these results might significantly improve the paper, we could rework the paper to include them.

Reviewer comment (Theory):
Paper overall seems geared to a slightly less technical audience than ICLR. For instance, describing each conv layer as a different region of the brain (Section 2.1) seems like a weird thing to do, at least in the AI community. Why is the 4th conv layer analogized to the lateral geniculate nucleus, for example? What is the basis for creating these mappings between conv layers and mammalian brain regions?

Author response:
Thank you for pointing this out, as we did not make this sufficiently clear in our writing. The goal of this paper is not to provide technical innovations of deep RL, but rather to understand how visual ecology shapes animal vision. The specification of our vision model is based on established models of early mammalian vision in the computational neuroscience literature (see Lindsey et al (2018) in ICLR, Ocko et al (2018) in NeurIPS, and Maheswaranathan et al (2023) in Neuron) - we cited these papers in our text but did not make the connection between our model and existing models explicit. 

Models with the structure of our vision model are highly effective at modelling the early visual system. In a simplified ANN model of the early visual system, the LGN can be understood as a 4th layer of neurons after the three layers of the retina (see Lindsey et al (2018)).

Reviewer comment (Novelty):
Some results are presented as interesting by the authors, but to me seem relatively straightforward and expected. For instance, including the input satiety into the architecture should obviously improve performance.

Author response:
It is of course not surprising that providing additional information to the agent could lead it to better performance, yet we believe interesting findings emerged from studying this architecture. Firstly, adding this signal improved agent performance by unlocking qualitatively distinct behaviours, for example waiting to be sufficiently hungry before eating food. Secondly, even though we showed that RNN-based architectures could estimate their own satiety, they did not exhibit this behavioural strategy. In this context, it seems worth reporting the degree to which this strategy improved agent performance.

Reviewer comment (Novelty):
Related work section should be much more substantial. Have prior papers studied “visual ecology” or done analysis on neural vision architectures when given an environment the agent must survive/do tasks in? It is hard to evaluate the contributions of this paper without a solid summary of previous work in this area.

Author response:
This is a fair point, and we apologize for insufficiently summarizing the existing literature. To the best of our knowledge this study is the first of its kind that studies neural network models of visual circuits in simulated ecologies where the agent must survive. We will improve the related work section to better situate our paper.

Reviewer comment (Theory):
Based on Figure 4C, it seems like satiety decreases at a constant rate, no matter what action the agent is taking. This seems undesirable, as animals expend different amounts of energy depending on different actions. For instance, being stationary should be much less draining on satiety than moving.

Author response:
We agree that this would be an interesting manipulation. Of course, there are a large variety of innovations one could build into the model, and the one you mention is only one of them. One that we explored was whether an animal has access to its satiety state. Others include multiple agents, competing behavioral goals and so on. Any of these would be interesting to study in the future.

Reviewer comment (Experiments):
Environment lacks interesting actions beyond moving in different directions or staying put. Additional simple actions may make the environment much more interesting, such as rotating field of view without changing (x,y) position, and needing to perform some manipulative action (such as triggering a set of discrete actions, such as “pulling vegetables out of the ground,” before being able to consume them). These additional actions can also have different satiety costs to them.

Author response:
In fact, it is possible to the agent to stay put and rotate its visual field, and trained agents do indeed exhibit this behaviour. We will explain more clearly in a revised text. Regarding your second point, the goal of this study is not to identify the most difficult task we can get an agent to perform, but to understand how visual ecology shapes the architecture of animal vision. If we simply make the task as difficult as possible, then we won’t be able to distinguish between poor performance due to the vision model/brain of the agent, and poor performance due to the difficulty of the task. Adding these task innovations would only be reasonable if we have a clear idea of the conditions under which an agent can solve the simplest task, or how “pulling vegetables out of the ground” creates a new constraint on the visual system that may influence the learned representation.

Reviewer comment (Experiments):
Environment does not accurately model effects of satiety on actions. Decreased satiety should have a harmful effect on the agent being able to take actions. A satiety=1 agent should not be as effective at moving around as a satiety=100 agent. Under this situation, it would be cool to analyze the satiety value at which the agent is most effective at finding food, since it would tend to be more stationary at higher satieties and tend to be less effective at moving around at lower satieties.

Author response:
Again, we very much appreciate the intuition behind this suggestion, and adding innovations like this would be appropriate for future work. However, we see no obvious reason why adding this additional complexity is necessary for the quality of our results, and otherwise keeping the environment as simple as possible is an advantage of our current study.

Reviewer comment (Experiments):
First abstract claim: “The complexity of the vision model required for survival on this task scaled with the variety and visual complexity of the food in the environment.” This may sound trivial, but authors should define model complexity. Is model complexity only dependent on parameter count? That seems inadequate, since a huge, deep feed-forward network with the same parameter count as a CNN or an RNN should still not be as “complex” due to the inductive biases encoded in the latter two networks. Some of the author’s experimental choices for looking at complexity, such as focusing a lot on number of channels, is probably misguided, since increasing number of channels after a point is known to saturate network performance, as Figures 2D-F show. Things like residual connections, kernel dimension/size, and number of conv layers will probably make for more interesting graphs.

Author response:
This is all quite fair. Firstly, we do mostly use “complexity” as a shorthand for parameter count. We will clarify this in the text. 

Secondly, there are biologically motivated arguments for varying some of the architectural features as you suggest, and we would like to explore them. Nevertheless, we focused on channel count for the reasons mentioned above (which again we did not explain sufficiently well in the text). We would also point out that performance does not saturate based on the number of channels on the CIFAR-10 task for the RNN network, which was one of the effects we were happy to isolate.

Reviewer comment (Presentation):
Third abstract claim: “Different network architectures learn distinct representations of environment and task, leading to different behavioral strategies.” Agent behavior was better investigated in the results, but not necessarily the “distinct representations” part of this claim, though there was Figure 4C which showed the different value functions. One suggestion here is that it would be better to revise Figure 4B and 4C, for instance, to show the sensitivity of all 4 methods of \( \hat{V} \) on the same image observation at the same location in the environment, so that readers can see the “distinct representations” in image space as well as in value space.

Author response:
Again, this is a fair point, and it is something we considered, but there are challenges to doing this rigorously. The main issue is that recurrent networks would require the recent history of images to be loaded as well, but then the history would be based on the policy of one agent architecture, i.e. the network with or without satiety. We know that networks with/without satiety behave quite differently, and so we can’t rule out that e.g. feeding an RNN that includes input satiety (IS) with an image history generated by an RNN without input satiety would lead the IS-RNN to generate uncharacteristic values of \hat{V}. Nevertheless, if this seems like a secondary concern to the reviewer, we could easily add this to a revised version of the manuscript – perhaps to the supplement at least.

Reviewer comment (Theory):
My main issue is the comparison of a simple CNN to an animal vision system, i.e. "We modeled the CNN after the early mammalian visual system: the base layers were grouped sequentially into the photoreceptor (PR), bipolar (BP), retinal ganglion cell (RGC), lateral geniculate nucleus (LGN), and primary visual cortex (V1) layers.” As far as I understand, it’s just a different number of channels and kernel sizes? Naming the different layers in a network after biological brain regions does not directly make them more biologically realistic.

Author response:
This is a fair point, and we could have explained better. The specification of the vision model of our RL agent is based on established models of early mammalian vision in the computational neuroscience literature (see Lindsey et al (2018) in ICLR, Ocko et al (2018) in NeurIPS, and Maheswaranathan et al (2023) in Neuron) - we cited these papers in our text but did not make the connection between our model and existing models explicit. Of course, such models are highly simplified, but the patterns of convergence and divergence of channels are modelled after what is happening in the visual system. In fact, such models are highly effective at modelling the activity of biological neurons in the early visual system (Maheswaranathan et al (2023)). So, while simple, the model is useful to our goal of uncovering the principles by which the environment and behavioral goals shape neural representations.

Reviewer comment (Experiments):
Environments in nature are much more complex than the ones proposed in this paper. To study visual ecology, it seems our agent environments need to be more complex as well.

Author response:
We agree that the world is indeed much more complex than the simple environment we use. However, we are convinced that this type of environment is the simplest possible that teaches us something interesting on how the statistics of the environment and behavioral goals interact to drive the evolution of neural representations, and choosing the simplest possible model to understand a phenomenon is at the heart of science. For example, if we make the environment or the task more complex, it becomes much more difficult to distinguish performance differences that were due to the difficulty of learning the task, versus differences that were due to the limitations of the vision system. For the purposes of this first study, we therefore found it important to keep the former as simple as possible, towards the goal of understanding the latter. In future work we indeed hope to study how agents learn and behave in more complex environments.

Reviewer comment (Novelty):
There is a lot of related work in the evolutionary community, which isn’t mentioned, where survival is the only reward mechanism.

Author response:
This is a fair point. We will  include citations from the evolutionary community to better motivate this part of our framework. We would certainly appreciate any recommendations you might have.

Reviewer comment (Experiments):
In conclusion, I would argue that a computational framework on visual ecology has to go beyond one experiment/domain with a slight variation on a deep RL setup.

Author response:
As explained above, the goal of our work is to extract the principles by which an agent’s visual system is shaped by the environment and its behavioral affordances. To the best of our knowledge, this is not something that deep RL has been used for. Furthermore, our computational framework is capable of supporting a wide variety of experiments and domains, and a wide variety of RL setups. Also, given (i) the amount of compute it takes to run deep RL simulations, and (ii) the unbounded number of experiments that might constitute all possible visual ecologies, we would argue that it is important to rigourously understand a base case as well as possible, against which more complicated cases may then be compared.

Reviewer comment (Experiments):
How would a model perform that is not "based on" the mammalian visual system?

Author response:
This is an interesting question, although we wouldn’t quite know how to start answering it, because CNNs are based on the mammalian visual system. Perhaps something based on transformers, although we suspect a thorough characterization would be beyond the scope of one paper.

Reviewer comment (Theory):
How realistic is the vision model when compared to animal vision?

Author response:
There are two answers to your question: One the one hand, the model is more realistic than it might appear - please see our response above. On the other hand, it is quite simplistic, but for good reasons, as also explained above. The question should be: is our model able to answer interesting questions: And we think it does.

Reviewer comment (Novelty):
In nature, adaptation is a result of evolution and lifetime learning. Wouldn’t it be important for this research to combine both adaptation mechanisms in some way? In particular, the evolution of plastic neural networks with Hebbian learning rules could be relevant here.

Author response:
Thank you for bringing this up. Currently, the two types of learning are somewhat conflated in our setup and we will make this more clear in the Discussion. In the future, we might explore an evolutionary approach to modifying the architecture of the network. With regards to Hebbian Learning, we are constrained by what can be efficiently trained in the context of a Deep RL system, and so for the purposes of this study we relied on conventional architectures that should in principle perform well.