Paper:
Visual Concept Learning: Combining Machine Vision and Bayesian Generalization on Concept Hierarchies
Learning a visual concept from a small number of positive examples is a significant challenge for machine learning algorithms. Current methods typically fail to find the appropriate level of generalization in a concept hierarchy for a given set of visual examples. Recent work in cognitive science on Bayesian models of generalization addresses this challenge, but prior results assumed that objects were perfectly recognized. We present an algorithm for learning visual concepts directly from images, using probabilistic predictions generated by visual classifiers as the input to a Bayesian generalization model. As no existing challenge data tests this paradigm, we collect and make available a new, large-scale dataset for visual concept learning using the ImageNet hierarchy as the source of possible concepts, with human annotators to provide ground truth labels as to whether a new image is an instance of each concept using a paradigm similar to that used in experiments studying word learning in children. We compare the performance of our system to several baseline algorithms, and show a significant advantage results from combining visual classifiers with the ability to identify an appropriate level of abstraction using Bayesian generalization.

Review:
This paper studies an interesting problem of visual 
      concept learning, particularly paying attention to determine the degree to 
      which a concept should be generalized. For this purpose, the work first 
      focuses on how to generate a large-scale dataset of concepts at various 
      levels of abstraction from ILSVRC and collect human judgements on AMT, and 
      then presents Bayesian concept learning algorithm to handle perceptual 
      uncertainty. The problem of learning concepts of various levels of 
      abstraction is interesting. However, this reviewer finds that the 
      algorithm presentation is unclear and lacks intuitive explanations. In 
      addition, the experimental evaluation is somewhat weak because it doesn’t 
      compare with strong baselines. More detailed comments are listed 
      as follows: 1. In Eq. (4) and (5), is I(.) an indicator function? 
      Please clarity. 2. In Eq. (4), I(x_i \subseteq h) implies that $h$ 
      is a set and $x_i$ is also a set. This is confusing to this reviewer. Why 
      is $x_i$ a set? In contrast, in Eq. (5), I(\hat{y}_i \in h) shows that 
      $\hat{y}_i$ is an item. Please clarify. 3. In Eq. (5), it seems 
      that the summation is only effective for the first term, i.e. 
      A_{j\hat{y}_i}. The remaining part is irrelevant to index $j$. Therefore 
      the summation of A_{j\hat{y}_i} becomes one column vector. Is this what 
      you want to get? I cannot understand the motivation of this equation. 
      4. It seems there is a typo in the last line of page 5. “… the 
      true leaf node is $j$ given the classifier output being $j$”. Please check 
      if you do want to put two $j$. 5. In line 243, page 5, the 
      sentence “for example, as the number of examples that are all Dalmatians 
      increases, it becomes increasingly likely that the concept is just 
      Dalmatians and not dogs in general even though both are logically 
      possible, …” is true. But what does this mean to the hypothesis size? The 
      hypothesis is quite confusing in the paper. It seems that the authors 
      define the hypothesis size as the number of samples belonging to a 
      hypothesis. How can this be defined in real world, given that the number 
      of potential images belonging to any concept can be infinitely large? Is 
      the number of samples in ImageNet a reasonable way to define the 
      hypothesis size? 6. Given so many unclear places in the algorithm 
      presentation, it is quite hard to understand the proposed algorithm and 
      know why it can really work, although the algorithm presentation is just 
      of high level. 7. The baseline methods are not sufficiently strong 
      and convincing. For example: a) For the naïve vision approach, only 
      GIST feature is used, whereas the proposed algorithm uses a overcomplete 
      feature set of 160K dimensions. This is unfair to compare the two 
      algorithms. b) The paper mentions the latest work of deep neural 
      network [10, 9] in Section 4.3. Why not choose DNN or its variant as a 
      baseline method? Simply because it requires more training data? If DNN 
      could output the proposed algorithm given that the training data can be 
      easily collected from ImageNet, I would doubt the value of this work.