Paper: Investigating Feature Alignment Under An Infant-Inspired Visual Distribution Shift (ICLR-2024)

Reviewer comment (Experiments):
The dataset itself has quality issues. The toy dataset shows the hands. The image datasets show (most likely) statistically correlated backgrounds. As a result, the whole investigation is not so much about the respective objects, as the paper indicates -- even if one sees some overlap in the classification performance.

Author response:
**Quality Issues In Datasets:** 
We disagree that the presence of hands in Toybox and of statistically correlated backgrounds in the instance-rich dataset weaken our results. Hands are pervasive in an infant’s view during active object exploration, while category instances seen in the real world also likely have statistically correlated backgrounds.

Reviewer comment (Writing):
Section 4 – the notation is not fully consistent – kernel K and k appear to be the same. (x,y) are sometimes two image instances and sometimes denote image and class of image (in the set notation). It also remains unclear in which layer of the ResNet structural alignment is applied – a small figure would be very useful here. Moreover, Section 4 gives much more detail than experiment 4, while Experiment 2 and 3 each have their own sections.

Author response:
**Structural Alignment:** 
We apply the structural alignment on the outputs of penultimate layer of the network before the classification layer.

Reviewer comment (Experiments):
One key question that is not answered in the paper is - how the ratio of different data distributions affect the generalization ability of the network. Varying size of datasets could impact results significantly, I’d appreciate more discussion on this front.

Author response:
The size of the datasets does play an important role in the generalization ability of the network. In some preliminary experiments, we found that reducing the size of any of the domains leads to a reduction in performance for test images for that domain. We want to explore this further in future work.

Reviewer comment (Evaluation):
The work is motivated by infant visual learning but there is no direct comparison with data collected from infants or human behavioral data across time to back up the cognitive claims.

Author response:
Our work is motivated by developmental psychology literature and the two datasets we used contain some properties that have been documented in this literature. Thanks for your comment on better relating to this line of literature in motivating our work and supporting the claims about the important properties of the two datasets.

Reviewer comment (Experiments):
I do think a big blind spot of the paper is the lack of consideration for unsupervised or self-supervised learning. In an infant learning scenario, we know that the supervisory signals are very sparse. It has long been suspected that self-supervision/unsupervised learning is more plausible for learning visual representations, and self-supervised representations align well with those of humans (e.g. Zhuang et al. 2021). In that context, I would like to see how self-supervised learning leads/doesn't lead to aligned representations across these two scenarios.

Author response:
Yes, we agree that supervisory signals are sparse in infants’ learning environments and that it is likely that infants self-supervise their learning processes. We also agree that there are many important and interesting questions pertaining to this area. However, that is not the focus of this paper. In this submission, we ask how different kinds of learning signals lead to different kinds of learning outcomes as measured by classification accuracy and alignment between the different datasets. We assume the presence of perfect supervisory knowledge, as given by class labels for each image. Even in this case, the effect of different learning signals on the learning outcomes is not well understood. In this paper, we focus on this aspect.