Paper:
Toddler-Inspired Visual Object Learning
Real-world learning systems have practical limitations on the quality and quantity of the training datasets that they can collect and consider. How should a system go about choosing a subset of the possible training examples that still allows for learning accurate, generalizable models? To help address this question, we draw inspiration from a highly efficient practical learning system: the human child. Using head-mounted cameras, eye gaze trackers, and a model of foveated vision, we collected first-person (egocentric) images that represent a highly accurate approximation of the "training data" that toddlers' visual systems collect in everyday, naturalistic learning contexts. We used state-of-the-art computer vision learning models (convolutional neural networks) to help characterize the structure of these data, and found that child data produce significantly better object models than egocentric data experienced by adults in exactly the same environment. By using the CNNs as a modeling tool to investigate the properties of the child data that may enable this rapid learning, we found that child data exhibit a unique combination of quality and diversity, with not only many similar large, high-quality object views but also a greater number and diversity of rare views. This novel methodology of analyzing the visual "training data" used by children may not only reveal insights to improve machine learning, but also may suggest new experimental tools to better understand infant learning in developmental psychology.

Review:
This paper explores the effect of data distributions on the performance of object classification systems, by specifically looking at data produced by toddlers. Specifically, the paper hypothesizes that the views of objects gathered by head-mounted cameras on toddlers have a combination of (1) high-quality, close-up views and (2) a large number and diversity of rare views (in contrast to views produced by adults in the same setting). The paper suggests that such high-quality datasets (such as those produced by infant wearing a head-mounted camera) may help with the problem of data efficiency.  Overall, I think this is an excellent paper and I think it would be great to have it presented at NIPS. It’s generally a good idea to look at how much different data distributions affect the performance of our algorithms, and it seems like an important piece of the puzzle to understand and quantify the data distributions actually perceived by human learners in comparison to our machine learniners. The choice of examining toddler data in particular seems quite unique and I think is very interesting. In general the experiments are quite nice, though the paper could be strengthened by the addition of a few more (described below).  Quality ---------  Generally the quality of the paper is quite high, but I think a few more experiments could be performed and a few more details given regarding the datasets and results.  I would appreciate it if the training accuracy could be reported as well as the test accuracy. In particular, it would be interesting to know whether the adult data was also more difficult to train on than the child data, or whether the difference only appeared at test time. If the latter is true, that would indeed be a very interesting finding.  I would like to see the analyses in Figure 3 performed for at least one standard computer vision dataset, like ImageNet, in order to better understand whether it is closer to the parent distribution or the child distribution. The paper suggests it would be closer to the parent distribution since the photos are taken by adults, but this is an empirical question.  An additional experiment would be to control for object size and test for the difference in diversity, by cropping the images so that the object takes approximately the same amount of space in the image in both the child and adult datasets. Then, any difference in performance on the test set would be due to diversity in view, rather than size. If there isn’t a difference in performance, then it suggests the main factor between adults and children is size, rather than diversity in viewpoint. This seems like it would be an important variable to isolate.  I think there is one issue in the experiments regarding object size, which has to do with the size of the objects in the test set. Specifically, if the objects in the test set are large, then it could be that by increasing the object size (decreasing the FOV around the gaze center) the training set is being brought closer to the test distribution. In that case, the argument becomes a bit weaker, in that it is more about the child data being closer to the test distribution than the training data. I think the paper should report the histogram of object sizes in the test set and analyze this more closely to see if this is indeed what is happening.  The similar/diverse analysis in Section 4.3 felt a bit weaker to me, because it seems to me that the important distinction isn’t just “similar images” vs. “diverse images” but “similar *canonical* images” vs. “diverse images”. However, Figure 6b suggests that the “similar” images aren’t necessarily canonical views. It’s not at all surprising to me that low diversity of non-canonical views leads to worse performance. A far more interesting comparison (with non-obvious results, at least to me) would be to see if low diversity of canonical views also leads to worse performance. A suggestion for how to achieve this would be to choose a small set of images which the authors feel are canonical views as a seed for their procedure for choosing the similar/diverse images, so that the similar images are all clustered around the canonical views instead.  Some more minor points:  - I would like to know what the overall accuracy of YOLO was, for all frames within a look. “At least one” doesn’t convey much, when the look durations were on average around 50 frames. - Thank you for reporting averages over 10 runs of the network, and error bars too! - Figure 6a might look a bit nicer if a different embedding was used, like t-SNE. Also in Figure 6a, it took me a bit to realize the black points were all the images---I would just clarify this in the caption. - The choice of adding in the blurring according to foveation is interesting, but also makes it a bit harder to compare this dataset to standard image datasets which do not have this information. Since the paper argues it’s not about improving performance so much as exploring the choice of data, this seems like an odd decision to me. Can the authors justify why they included the acuity simulation further, and perhaps also report results without that (maybe in an appendix)?  Clarity --------  The paper is very well written and easy to read and understand. I only had a few points of confusion:  In the second paragraph of the introduction, the paper states that “more quantity naturally leads to better overall quality”. This confused me until I read the rest of the paragraph, and I think the reason is that the paper uses a different definition of “high-quality” than is typically used in the machine learning literature. Usually, “high-quality” in ML means that there is very little noise/error in the labels, that objects are in a canonical view, etc. After reading the rest of the paragraph I understand what the paper means, but I would suggest this paragraph be rephrased just to avoid confusion over the definition of this term in the first place.  Line 78: “a few recent studies have applied deep learning models to data collected by humans” Most real-world machine learning datasets were collected by humans, so I think it would be more accurate to say “nearly all deep learning methods are applied to data collected by humans”. But given the citations, I think this sentence was perhaps meant to convey something different, and more about the intersection of psychological research/data with machine learning?  Originality -------------  While egocentric datasets have been previously explored in computer vision (e.g. [1, 2]), I do not believe toddler data has been examined in the same way, nor have there been comparisons between adult and toddler data in the context of machine learning performance (of course this has been extensively explored in the psychological literature). While the paper does not introduce a new model, it does perform a nice set of experiments that better characterize the dimensions that we care about in high-quality data for computer vision.  Significance ----------------  Overall, I think this paper could be quite useful to the machine learning and computer vision communities for thinking about the distributions of our datasets from a psychological perspective.  One question that I have regarding significance is whether the authors plan to open source the dataset? I think it would be a valuable contribution to the computer vision community and would increase the significance of the paper to release the dataset. However, I realize this might not be possible due to privacy concerns (but perhaps the dataset could be anonymized by blurring out faces, in that case?)  [1] http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html [2] http://imagelab.ing.unimore.it/WESAX/related_datasets.asp  Edit after author response: I have read the author response and am quite satisfied with it; I am particularly happy that they have (1) included the results without the acuity blur, (2) included more information about the distribution of ImageNet images, and (3) included information about the size of objects in the test distribution (and I find it very interesting that the results are as good as they are when the size of the objects in the test distribution is 0.75, compared with the size of 0.129 in the children's distribution). I think the community at NIPS would find this paper and its results very interesting and have thus decided to increase my score by one.