Paper: A Newborn Embodied Turing Test for Comparing Object Segmentation Across Animals and Machines (ICLR-2024)

Reviewer comment (Presentation):
The presentation of this work still needs improvement, as the figures are not in vector format. For example, Figure 3 lacks clear labels for the x and y axes, making it difficult to understand the meaning of the figure.

Author response:
**The Reviewer notes that the figures need improvement.**

As suggested, we improved the quality of the figures and added clear axis titles.

Reviewer comment (Presentation):
The presentation of this work still needs improvement, as the figures are not in vector format. For example, Figure 3 lacks clear labels for the x and y axes, making it difficult to understand the meaning of the figure.

Author response:
**The Reviewer notes that the figures need improvement.**

As suggested, we improved the quality of the figures and added clear axis titles.

Reviewer comment (Experiments):
Perhaps the biggest weakness is that the vision models here are trained in a way that is not very indicative of the state-of-the-art. Most vision models that is used in the community are usually a combination of self-supervised training and supervision with a lot of exemplars. While this might diminish how relevant this work is for the community, it does not take away from the significance of the work since the goal of the work is to study models when they get only as much visual experience as newborn chicks (i.e newborn chicks are not getting a ton of supervision).

Author response:
**The Reviewer argues that the biggest weakness of the paper is that we did not use state-of-the-art systems in object segmentation. The reviewer notes that this contradicts the goals of the benchmark (which requires matching the training data across animals and machines), but nevertheless convincingly argues that this would help integrate the paper into the ICLR community.**

We agree and implemented the suggestion in two ways:

First, we inserted a variety of powerful pre-trained visual encoders into the agents, including DINOv2 (Oquab et al., 2023), Ego4D (Radosavoic et al., 2022), and Segment Anything (Kirillov et al., 2023). These pre-trained models perform well on standard computer vision benchmarks for object segmentation; nevertheless, artificial agents equipped with these pretrained encoders still failed to solve the one-shot learning task solved by newborn chicks. As shown in Fig. S3, agents with pre-trained encoders learned background-dependent object representations.

Second, we pre-trained the visual encoders using simulated first-person images from inside the controlled-rearing chambers, mimicking the visual experiences of the chicks (Fig. S2). The encoders were thus pre-trained in the same environment in which we tested the artificial agent. We then inserted those frozen pre-trained encoders into the artificial agents, allowing the policy network to learn from a stable visual encoder. We reasoned that training the encoder and policy networks sequentially, rather than simultaneously, might help the agents solve the task, since the frozen encoders would provide more robust visual experiences for learning the embodied imprinting task. However, like all the other agents we tested, the agents failed to solve the task, largely learning background-dependent object representations.

Reviewer comment (Experiments):
PPO agents are trained for a lot of episodes (original PPO paper (Schulman, 2017) did 1M episodes) usually while this work only does 1000 episodes. This is probably okay since this setting is much simpler. But I would still train one model for much more steps to rule out strange behaviors (like grokking (Power, 2022)).

Author response:
**The Reviewer notes that since we only trained for 1K episodes, it is possible that more powerful generalization abilities might emerge with more training.**

Great suggestion. To test this possibility, we trained the artificial agents with 10X more episodes (10,000 episodes). As shown in Fig. 3G, the agents still learned background-dependent object representations, akin to agents trained for 1,000 episodes. We also emphasize in the revision that we chose the 1,000 training episodes after observing that the artificial agents successfully learned to imprint in 1,000 episodes (as shown in Fig. S5).

Reviewer comment (Writing):
The paper might benefit from an expanded section on the differences between birds and mammal brains, especially since the audience is likely to be more from the computational side. While I do appreciate the effort in section 1.1, it might be helpful to expand it a bit more.

Author response:
**The Reviewer suggests adding an expanded section comparing bird and mammalian brains.**

Great suggestion. We added a more detailed discussion of the differences between avian and mammalian brains to the Supplementary Methods (Section A.1).

Reviewer comment (Presentation):
Why does Figure 1 make it seem like this is a cyclic process? I don't see why it is put on a circle - step 1 does not follow step 6?

Author response:
**The Reviewer wonders why we visualized the NETT as a circular process.**

In the Figure 1 caption, we now explain that we see this NETT as the foundation for a closed-loop scientific paradigm, in which researchers build ML models that learn like newborn animals, and then use those models to generate predictions for new experiments with newborn animals to validate and refine those models.

Reviewer comment (Presentation):
Section 2.1: Was the second display blank (all black) or just a background?

Author response:
**The Reviewer suggests clarifying the stimuli presented on the 2nd display walls and Fig. 3 labels.**

We revised the manuscript accordingly to indicate that the 2nd display wall was a white screen. We also improved the quality of the figures and clarified what “F” and “N” stand for in the figures.

Reviewer comment (Writing):
Section 2.2: Do the RL agents need to be called "artificial chicks"? I think calling them "RL agents" or something is less confusing.

Author response:
**The Reviewer suggests using the term “artificial agents” rather than “artificial chicks.”**

We agree and have changed the manuscript accordingly.

Reviewer comment (Presentation):
All the figures need major revamp. The text is barely readable and they need to be of higher resolution. Figure 2 needs to say what "F" and "N" stand for. Figure 2 could also be made such that the backgrounds and foregrounds are easily distinguishable.

Author response:
**The Reviewer suggests clarifying the stimuli presented on the 2nd display walls and Fig. 3 labels.**

We revised the manuscript accordingly to indicate that the 2nd display wall was a white screen. We also improved the quality of the figures and clarified what “F” and “N” stand for in the figures.

Reviewer comment (Presentation):
Figures in the paper are very low resolution (and possibly heavily compressed with lossy compression). This is particularly noticeable for figure 3, which is nearly illegible.

Author response:
**The Reviewer notes the low resolution figures.**

We improved the resolution quality of the figures.

Reviewer comment (Experiments):
The number of trials used for training artificial chicks seems rather small at 1000. Similarly, the episode length of 1000 seems long given how small the environment is. Why were these numbers chosen? Are they similar to the number of times the biological chicks turn towards their object and how long they stay facing it?

Author response:
**The Reviewer notes that since we only trained for 1K episodes, it is possible that more powerful generalization abilities might emerge with more training.**

Great suggestion. To test this possibility, we trained the artificial agents with 10X more episodes (10,000 episodes). As shown in Fig. 3G, the agents still learned background-dependent object representations, akin to agents trained for 1,000 episodes. We also emphasize in the revision that we chose the 1,000 training episodes after observing that (1) the artificial agents successfully learned to imprint in 1,000 episodes, and (2) the artificial agents developed a robust form of object recognition in 1,000 episodes.

Reviewer comment (Experiments):
Did the authors perform the biological chick experiments themselves or are those directly from Woods & Woods 2021? (I was unable to get access to this paper to check myself.) If the authors performed this experiment, has it been checked by an Institutional Review Board (IRB) or similar?

Author response:
**The Reviewer wonders whether we performed the chick experiments ourselves to check that necessary animal welfare/ethics procedures were followed.**

The chick experiments were conducted by Wood & Wood (2021) and not conducted for this paper. We have revised the text to make this clearer for readers. Our paper extends this prior work by turning the behavioral results into a NETT benchmark. The original paper (Wood & Wood, 2021) specifies that the work was approved by an IACUC (pg. 2411).

Reviewer comment (Experiments):
Some "high-water" marks for artificial chicks would be useful. If one was trained on the test set, how would it do? Ideally it should perfectly solve the task. If one was trained with a singular object but many different backgrounds, how would it do?

Author response:
**The Reviewer suggests adding “high-water” marks for the artificial chicks, by training and testing them on the test set.**

Great idea. While we did not have time to add “high-water” makers to this revision, we plan to add them to the camera-ready version if the paper is accepted.

Reviewer comment (Novelty):
The technical contribution of the paper is limited. Specifically, the proposed ‘artificial chick’ is a PPO agent trained with ‘imprinting reward’, which encourages the agent to move close to an object. It is not surprising that this simple baseline with a heuristic reward fails to solve the one-shot object segmentation task.

Author response:
**The Reviewer was concerned about the heuristic (imprinting) reward.**

We agree with this concern. Our aim is to offer a challenging benchmark that requires deep RL algorithms to be trained and tested in the same environment as real animals in controlled-rearing experiments. To succeed at this task, researchers may need to use ML systems that use different rewards, like curiosity. In our revision, we emphasize that we designed the benchmark to allow researchers to use their own custom-designed intrinsic rewards to drive the agent’s behavior (i.e., researchers do not need to use our imprinting reward). This makes the NETT more flexible: researchers can plug different encoders, policy networks, and/or intrinsic rewards into the artificial agents. We hope this NETT will be useful for discovering reward structures that produce robust object segmentation in embodied ML systems.

That said, we informally tried a number of different rewards and found that this imprinting reward produced behavior that best matched the imprinting behavior of newborn chicks. However, we agree with the reviewer that newborn animals learn through intrinsic motivations like curiosity, so building ML systems that learn like brains will ultimately require “turning off” our imprinting reward and using a more biologically plausible intrinsic reward.

Reviewer comment (Experiments):
The reviewer found main claims in the paper are not well-supported by the experiments. Specifically, the authors claimed that ‘’... none of the algorithms learned background-invariant object representations that could generalize across novel backgrounds and viewpoints”. However, the ML algorithm the authors used in the paper are just PPO with different architectures and a heuristic reward. The reviewer thinks those simple methods well-represent the state-of-the-art one-shot object segmentation methods. There are many works on one-shot segmentations [1, 2]. Incorporating those methods in the ‘artificial chick’ might make the experiments more convincing.

Author response:
**The Reviewer argues that the claims are not well-supported by the experiments because we used a PPO agent trained on a heuristic (imprinting) reward. The Reviewer suggests adding state-of-the-art object segmentation models into the artificial agents to make the experiments more convincing.**

This is a great idea. We implemented this suggestion in three ways:

First, we took three top-performing object segmentation models—DINOv2 (Oquab et al., 2023), Ego4D (Radosavoic et al., 2022), and Segment Anything (Kirillov et al., 2023)—and inserted them into the artificial agents. The pre-trained encoders were trained on millions of images across one thousand object categories, far exceeding the sparse visual experiences of the chicks. Nevertheless, artificial agents equipped with these pre-trained encoders still failed to solve the one-shot learning task solved by newborn chicks. As shown in Fig. S3, agents with pre-trained encoders learned background-dependent object representations.

Second, to allow the policy network to learn from a stable visual encoder, we pre-trained visual encoders using simulated first-person images from inside the controlled-rearing chambers (Fig. S2). The encoders were thus pre-trained in the same environment in which we trained the artificial agents, mimicking the visual experiences of the chicks. We reasoned that training the encoder and policy networks sequentially, rather than simultaneously, might help the agents solve the task, since the frozen encoders would provide more stable visual features for learning the embodied object recognition task (as demonstrated by Parisi et al., 2022). However, like all the other agents we tested, the agents failed to solve the task, largely learning background-dependent object representations.

Third, we emphasize that we used both PPO and LSTM PPO policy networks in the paper.

In future work, we will focus on inserting more powerful policy networks into the artificial agents. As the Reviewer suggests, it is possible that PPO and LSTM PPO policy networks are insufficient to solve this task. We hope researchers use our benchmark to find policy networks that can match the rapid and robust learning of chicks.