Paper: Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World (ICLR-2024)

Reviewer comment (Writing):
The presentation and writing could be clearer, making some parts of the paper difficult to understand, especially around the creation of the dataset in Sec. 2 and the precise problem setting. For example, it was not immediately clear that the labels of the two sets (positive, negative) are given and do not need to be inferred. The query image can belong to either, but the name "positive" suggests that this is the GT set of the query image, which is not the case. I am adding more clarifying questions in the questions section below.

Author response:
Thanks for highlighting this point, we have revised our main paper for greater clarity. For your specific question here, the labels of the support set are given and without anything else. Each Bongard Problem includes both a positive and a negative query image.

Reviewer comment (Experiments):
The problem setting is imbalanced. The positive set corresponds to a single concept while the negative set does not, but instead contains a subset of the complete concept. While this is not necessarily an issue and can be a design choice, there is no justification why this choice has been made. For instance, why not have both sets correspond to a single concept where the contrasting sets are close in semantics to make it a hard problem? Similarly to how different splits are evaluated in Table 2, it would have helped to show the performance of positive query images vs. negative query images in order to understand if this imbalance makes positive/negative queries easier/harder.

Author response:
Our problem setting adheres strictly to the original Bongard Problem (a classic puzzle of computer vision and pattern recognition: https://en.wikipedia.org/wiki/Bongard_problem), which include several features:

- The support set comprises only images of positive and negative with their labels (just indicates whether it belongs to the positive or negative set).
- Images from the positive set should depict the same concept, while those belong to the negative set should not depict this concept and the concept depicted by the negative images can be varied -- this could lead to an "imbalance" on the number of concepts but yes, this is a designed choice made by the original Bongard problem. 

But Bongard-OpenWorld also introduces some new features:

- First of all, both the positive and negative images includes distractor concepts (as we've demonstrated in Figure 1 of the original paper) -- that is, images from the positive set are in fact could also depict many more concepts besides the ground truth, alleviating the "imbalance" issue. 

- Secondly, as you can see from the example problems we showed in the Appendix H, when we picking the negative images, we also seek to make them share some common concepts, this is relevant to the solution you mentioned above and indeed help create a hard problem.

- Finally, we even make positive and negative images could overlap on the concepts they depict. For example, given the ground truth concept is "red tie", both positie and negative concepts can depict "dog", resulting "dog wearing a red tie" (positive) and "dog wearing a blue shirt" (negative).

Again, we thank you for raising this issue, and we believe our design in Bongard-OpenWorld not only respect the original Bongard problem setting, but also mitigate the possible drawbacks, including imblancing and easy problems.

Reviewer comment (Experiments):
While a lot of models have been evaluated on the proposed benchmark, a natural baseline is missing: computing the image similarity between the query image and the two sets. For instance, one can use any pre-trained image encoder (CLIP, Dino, etc.) or image-to-image retrieval method and use the mean similarly of the image embeddings per set to make a prediction. Using captioning models and LLMs seems to introduce complexity while at the same time discarding fine-grained image information by only relying on text to make the decision.

Author response:
Thanks for your insight. We provide the results of the natural baseline you mentioned below. In fact, this baseline could be very similar to ProtoNet (which we already benchmarked), but ProtoNet further fine-tunes the visual features on the training set.

This is how we implement it: we calculates the mean similarity between a query image embedding (produced by CLIP) and both positive and negative images embeddings (also produced by CLIP) in the support set, assigning the label of the higher mean as its prediction.

As for the result, its notable underperformance compared to coin-flipping (50% chance of success) stems from adversarial query selection (section 2.2). In selecting query images, we deliberately increase the similarity (measured by CLIP embedding distance) between positive queries and negative images in order to make the problem harder. Such design adversarally prevent this natural baseline with CLIP image embeddings from performing well.

Please note that, even with other backbone (ex. DINO), we believe the problem should still exist, as this natural baseline can be closely resemble to ProtoNet, which also fine-tunes the backbone. In our experiment, we already demonstrate its inability on Bongard-OpenWorld, even with non-CLIP backbone.

| method | image representation | aux. task? | short concept | long concept | CS∗ concept | non-CS∗ concept | avg. |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| natural baseline | OpenCLIP | &#10007; | 12.8 | 7.7 | 12.7 | 9.7 | 10.5 |
| ProtoNet | OpenCLIP | &#10004; | 59.2 | 57.7 | 51.8 | 61.0 | 58.5 |

Reviewer comment (Experiments):
With around 1K tasks it is a rather small dataset. Hence, focusing on the "zero-shot" setting without involving training might be the better use case.

Author response:
Thanks for your insight. Our evaluation includes VLM, LLM, and neuro-symbolic approaches, and all of them exclusively utilize the test set, primarily on zero-shot settings. However, we have also explored diverse few-shot learning approaches to ensure a more comprehensive evaluation in our benchmark.

Reviewer comment (Novelty):
While it is true that the benchmark contains a large variety of concepts, positioning it heavily as an "open-world" and "open-vocabulary" task could be a bit misleading as the core problem is to identify whether an image came from set A or set B. The optional task of naming the concept is most fitting for "open-world", but it serves a minor role in the paper.

Author response:
The core of "open-world" and "open-vocabulary" lies in its thinking process of solving the Bongard Problems, instead of merely examining its output. That is, to solve Bongard problem, namely, predicting whether an image came from set A or set B, the model has to understand and reason these images in an open-world or open vocabulary fashion. Therefore, we believe the position we choose is appropriate.

Reviewer comment (Experiments):
Have you thought about not providing the labels "positive" and "negative" for the two sets to the methods? Why have you chosen this setup?

Author response:
As we mentioned above, not proving the positive or negative labels could make the setting largely deviate from the original Bongard problem setting, which we choose to respect when creating this dataset. Also, we don't see a clear advantages of doing such. That include the "imbalancing" issue in your comment -- our response above has shown that our dataset does not appear to have a strong imbalance issue.

Reviewer comment (Writing):
In Sec 2.1: What is a grid in this context? What is grid sampling? How do you define "concept tuples"? Neither the main paper, nor the supplementary clarifies this sufficiently.

Author response:
We apologize for this oversight and have updated our main paper. Regarding your specific inquiries here, 'grid' denotes the pool we sample concept from. Instead of sampling from the whole CC-3M concept set, we first split it into seveal small pools, or as we name it, "grids", with size of 300 each. Then we perform top-k sampling within each grid to obtain concepts. We find this balances the need for sampling top concepts and sample diversity.

Furthermore, 'concept tuples' refer to candidates for the visual concepts $\mathcal{C}$, where each concept conprises a tuple of words, ex. "<red, tie>".

Reviewer comment (Experiments):
How do you ensure that the dataset does not contain duplicate concepts? I assume this is the case because in Table 1, it is reported that the dataset has 1.01K concepts and 1.01K tasks.

Author response:
Each Bongard Problem in our proposed benchmark has undergone extensive manual review to ensure full compliance with standard settings and is unique, even if filtered during automated collection. Thanks for your attention to this matter.

Reviewer comment (Reproducibility):
What are the exact instructions the annotators were given? For instance, when "annotators are instructed to write visual concepts by following a predefined set of categories illustrated in Table 7" and when "they are also asked to combine these challenging concepts with those mined from CC-3M" (Sec. 2.1).

Author response:
Here are the guides to our annotators:

- General concept annotation guide:
    1. Positive images all share one concrete concept, denoted as $\mathcal{C}$. This concept should be engaging and not overly simplistic (such as basic objects, 'dog' or 'cat'), but rather somewhat abstract (for instance, 'running' or 'reflection').
    2. Each negative image must depict a concept that must have some added distractors with $\mathcal{C}$.
    3. Each negative image must depict a concept that partially overlaps with $\mathcal{C}$.

- Commonsense annotation guide:
    
    Follow the definition of commonsense, manually revise the automatically collected candidate concepts (from CC-3M) to make them more "commonsense"-alike.

    One definition of commonsense from Wikipedia: "Commonsense knowledge includes the basic facts about events (including actions) and their effects, facts about knowledge and how it is obtained, facts about beliefs and desires. It also includes the basic facts about material objects and their properties." The following are some examples:

    1. HOI: discerning subtle interaction between humans and objects, e.g. playing basketball vs. playing volleyball.
    2. Taste / Nutrition / Food: identifying ingredients in food, e.g. high-calorie vs. high-sugar.
    3. Color / Material / Shape: differentiating similar materials with various appearances, e.g. rough wood vs. painted wood.
    4. Functionality / Status / Affordance: recognizing variants that have the same function, e.g. cuttable.
    5. And / Or / Not: making logical judgments about something, e.g. no something.
    6. Factual Knowledge: possessing factual knowledge, e.g. buildings in the capital of the United States.
    7. Meta Class: induction of metaclass, e.g. rescue vehicle (including ambulance, police car, fire engine, etc), Felidae (including cat, lion, puma, etc).
    8. Relationship: perceiving spatial relationships, e.g. "near", which may solely be "near on pixel distance" but located in the foreground and background respectively results in "far on physical distance".
    9. Unusual observations: e.g. light refraction, gaze direction.

As you can see, the annotators are guided to revise & reorganize the original CC-3M concepts into commonsense concepts, with explicit guidance on what can be viewed as commonsense concepts. This aligns with the descriptions in the main paper. We've included this guide in the Appendix C to make it more clear.

Reviewer comment (Reproducibility):
Images are collected by using an online search based on the concepts. What is the license of the images collected? Do the authors have the rights to distribute the images?

Author response:
We do not set further restrictions on searching therefore the license could be varied; by default, we only distribute URLs instead of the original image (we do have cache of the original images as backup); thanks for pointing this out, we are contacting the copyright owner about the distribution issue.

Reviewer comment (Experiments):
In Sec. 2.2. you write: "the annotators are then asked to provide two sets of candidates for positive and negative queries". How many images are collected here for possible query images? Why choose only one positive and negative image as query in the end?

Author response:
Sorry for not making it clear. We start with 7 query images for both positive and negative and rule it down to 1. The choice of only using 1 query is to align with the original Bongard problem, which we respect in Bongard-OpenWorld.

Reviewer comment (Experiments):
Does defining the concepts of category 0 (from CC3M) undergo any crowd-sourcing or is it fully automated?

Author response:
All the data collection processes in our proposed benchmark combine automated preparation of candidates with meticulous manual review to ensure complete compliance with our established criteria.

Reviewer comment (Evaluation):
Is there performance difference between positive and negative queries?

Author response:
We thank you for your suggestion!  Here are some additional results singling out the accuracy on positive and negative queries. As you can see, there might be slight difference for a single model, but we did not observe a pattern or preference towards positive or negative queries globally. Therefore Bongard-OpenWorld does not appear to exhibit a strong imbalance issue.

| method | image representation | aux. task? | positive query | negative query | avg. |
| :---: | :---: | :---: | :---: | :---: | :---: |
| Meta-Baseline | OpenCLIP | &#10004; | 59.5 | 53.0 | 56.3 |
| MetaOptNet | OpenCLIP | &#10004; | 57.9 | 57.0 | 57.5 |
| ProtoNet | OpenCLIP | &#10004; | 58.2 | 58.7 | 58.5 |
| SNAIL | OpenCLIP | &#10004; | 61.2 | 66.8 | 64.0 |
| GPT-4 | InstructBLIP | &#10004; | 67.4 | 60.2 | 63.8 |

Reviewer comment (Presentation):
In Figure 2c, both x and y-axis should be labeled. What is the scale/size of the number of concepts (x-axis)? What is the unit of the numbers on the y-axis?

Author response:
In Figure 2c, x represents an integer sequence ranging from 0 to the total count of unique word (approximately 1.6K) that could appear in the concepts of Bongard-OpenWorld. Meanwhile, y denotes the ratio of each word across all concepts (1.01K), 'people' as the most frequent word, with its ratio being approximately eight in a thousand.

Reviewer comment (Evaluation):
What is meant by "we report the overall accuracy of all models". Does Table 2 report test set accuracy or accuracy over the whole dataset, i.e., including training samples?

Author response:
All accuracy presented in Table 2 were derived from the test set only.

Reviewer comment (Theory):
Why are concepts from CC3M considered non-commonsense?

Author response:
Thanks for pointing this out, per our observation (as we mentioned above, each Bongard problems will be manually screened), most concepts mined from CC-3M and used by Bongard-OpenWorld indeed do not exhibit commonsense characteristic as we showed in the previous response. But indeed, we've removed this claim, as the commonsense category is labelled after the construction of Bongard problems, which means both problems from human annotated commonsense concepts or the mined CC-3M concepts can be labelled as commonsense problems.

Reviewer comment (Reproducibility):
How is ChatGPT finetuned (Table 2)? Does this use the finetuning API of OpenAI? More details would help make this more reproducible.

Author response:
We used the fine tuning API of OpenAI and the training set of our proposed benchmark. To ensure complete reproducibility, in our abstract, we have clearly stated that all the code implementation details and the dataset have been made public under this anonymous github repo: https://github.com/Bongard-OpenWorld/Bongard-OpenWorld.

Reviewer comment (Presentation):
Table 2 includes methods that use training data to update NN weights and others that do not update weights ("zero-shot" setting). It would be much clearer if the table indicates which models use training data.

Author response:
We have updated Table 2, thanks for your reminder.

Reviewer comment (Writing):
The following phrase appears 3 times in the manuscript. I suggest to to reduce this repetition and rephrase it according to the context. "We even designed a neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard problems".

Author response:
We sincerely apologize for our oversight and have rewritten these sentences. Your meticulous attention is greatly appreciated.

Reviewer comment (Presentation):
Sec. 3.2 (at the end) promises captioning metrics, but they do not appear in the main paper, only in the supplementary.

Author response:
Owing to space limits, we have relocated this to the Appendix and accordingly restructured the article. These metrics are conventional for caption evaluation; references for these are provided in Appendix C.

Reviewer comment (Experiments):
In the experiments, the authors primarily focus on conducting investigations using real-world datasets, particularly their self-constructed dataset. However, given the Bongard Problem, it raises concerns about the generalizability of the conclusions/findings obtained from real-world datasets to mathematical datasets.

Author response:
Could you please provide a few examples of mathematical datasets? This would allow us to elaborate further. The primary focus of our benchmark is on real-world reasoning, or reasoning in a more realistic setting -- real-world images, open-vocabulary, or open-world concepts. Mathematical datasets, which we suppose primarily adopt synthetic images and close-world concepts, are not the main focus of this work.

Reviewer comment (Experiments):
The experimental results seem to ignore the traditional models, and it remains a concern.

Author response:
Our experimental results cover a broad spectrum of approaches, ranging from traditional few-shot learning methods, combinations of Large Language Models and Visual Language Models, as well as a neuro-symbol method, and offering a comprehensive analysis. Could you please specify the traditional model you are referring to?  If you give a concrete reference, we would like to try.

Reviewer comment (Novelty):
Some related work is missed. [A] (see references below) studies a setting very related to the proposed benchmark (though they didn’t use the terminology Bongard problems). They also created tasks using natural (‘real-world’) images from different datasets, from using computer vision datasets (rather than scraping the web).

Author response:
We appreciate your suggestion and have incorporated this reference with discussion into our main paper. It is pertinent to note that the primary focus visual concepts of this work comprise various compositional attributes. Our proposed benchmark is characterized by its free-form nature, allowing a wide range of concepts with diverse elements.

Reviewer comment (Experiments):
It would be great to add additional few-shot learning baselines to cover families of approaches that are excluded from the current analysis like approaches that perform FiLM-conditioning e.g. [B, C] (see references below) and approaches that train the backbone with gradient descent within each task, like MAML and Proto-MAML (the latter is a version proposed in the Meta-Dataset paper which is cited by this work).

Author response:
We are grateful for your suggestions. We are currently implementing the FiLM-conditioning approach, this process will take a while. Regarding the MAML and Proto-MAML approaches, we encountered issues where its training process would collapse, leading to NaN. We believed similar issues have been observed in Bongard-HOI(http://arxiv.org/abs/2205.13803), i.e. some FSL approaches cannot produce meaningful results due to collapsing. Our hypothesis is having even fewer training data (in Bongard-OpenWorld) might have worsen this issue. We hence the exclusion of these results from our report.

Reviewer comment (Writing):
The paper has some clarity issues, perhaps owing to the fact that the authors tried to ‘squeeze’ a lot of content in the required number of pages. It’s hard to fully understand the different methods by reading only the main paper. I found the neuro-symbolic method proposed especially hard to understand (even after looking at the algorithm in the appendix). Please include some higher-level motivation and the intuition for the particular updates that it entails.

Author response:
We have further clarified approaches in the updated paper. For your convinience, here we offer some higher-level motivation and the intuition for the neuro-symbolic method:

- In contrast to canonical FSL and large model based approaches, this approach begins with not just converting images into captions (ex. a dog is running on the grass), but further into concepts (ex. <dog, running, on grass>). This is done by employing GPT-4, the prompt is detailed in Appendix D. Our goal here is to mitigate the substantial noise present in image captions (both in single-round and multi-round approaches).
- With these concepts, we apply logical operations like AND, OR, and NOT, aiming at mimicking the human logical thinking process when tacking Bongard problems. Specifically (please note that this is just high-level illustration and some details for handling corner cases are omitted. Again, the [code](https://github.com/Bongard-OpenWorld/Bongard-OpenWorld) includes all the implementation details):
    1. Compute the intersection of the concepts of all positive images, this gives us the initial hypothesis on the true concept of this Bongard problem. If there is no intersection (possibly due to flaw in perception), a majority vote is adopted -- concepts that present in more than 4 positive images will be treated as the "intersection".
    2. For each negative image, we substract its concept from the "intersection" obtained in the previous step, this is to mimick how human rule out the concepts that is depicted by both the positives and the negatives, and finally reach to the concept that is uniquely depicted by the positives.
    3. We compare the updated "intersection" with the concept of the query image. If the "intersection" is a subset of the concepts depicted by the query, the query will be categorized as positive, otherwise deemed negative.
    4. The "intersection" is also the ultimate induced concept of this Bongard problem, exclusively depicted by the positives.
    
- As a result, such logical operations over concepts can deterministically produce binary predictions on the query image and induced visual concepts depicted by the positives.

Negatives contribute candidates that aid positives in updating their missed atomic concepts (potentially distractors). Conversely, positives provide candidates that help negatives refine their concepts (potentially overlaps).

Reviewer comment (Presentation):
In Table 2, it’s hard to tell which methods / rows correspond to which of the families of approaches (e.g. a, b, c, or d in Figure 3) – and e.g. which are single-round or multi-round. Perhaps a good way of indicating this is by adding an extra column in that table.

Author response:
We appreciate your insightful suggestion and have updated Table 2.

Reviewer comment (Experiments):
It would be great to conduct ablation analyses for design choices made in creating the benchmark, like the adversarial query selection that picks the positive query to not be too close to the prototype of the positive class.

Author response:
We have added an illustration (see Figure 11 in Appendix G) to visually demonstrate the ablation analysis of design choices. Without the adversarial query selection, and relying solely on randomization, the query images become easily distinguishable (all images and concepts are not changed).

For instance, in subfigure (a), the positive query and positive images are not similar in appearance, only sharing the color of "red". However, negative images also feature "red" and "tie" (partially overlap), but none depict "red tie" (exclusively depicted by the positives), necessitating complex reasoning for correct determination. If only by appearance, the negative query could be easily misclassified as positive due to its resemblance to P.1, P.3, and P.4 (from left to right), and even P.2, resulting in it is very difficult to disentangle the problem.

Conversely, in subfigure (b), without adversarial query selection, simplifies this matter significantly. The positive query in (b) closely resembles P.1, P.2, P.3, and even P.4, facilitating a straightforward determination. Similarly, the negative query in (b) bears a high resemblance to N.1, N.2, and N.3 in appearance.

Reviewer comment (Experiments):
It would be great to conduct an analysis of the effect of the ‘shot’ on these problems. Intuitively, the more positive and negative images the network sees, the easier it is to infer what is the positive class and correctly label query images. Given the negative results in the paper with the current number of shots (6 positives and 6 negatives), in terms of the very large gap from human performance, have the authors considered increasing the number of shots? Understanding how performance of different methods differs as the number of shots increases would be insightful.

Author response:
We have updated the results to include 2, 3, 4, and 5-shot in support set on SNAIL, in contrast to the initially reported 6-shot approach. All of them employ identical settings and parameters, utilizing OpenCLIP pretrained image representation along with a training caption head. It indeed aligns with intuition. One thing to note: although reducing the number of shots could lead to more significant challenge, but it also deviate from the original setting of the Bongard problems, which we respect in Bongard-OpenWorld.

| x-shot | short concept | long concept | CS* concept | non-CS* concept | avg. |
| :---: | :---: | :---: | :---: | :---: | :---: |
| 2 | 60.6 | 56.6 | 52.7 | 61.0 | 58.8 |
| 3 | 62.4 | 58.8 | 57.3 | 62.1 | 60.8 |
| 4 | 61.9 | 61.6 | 62.7 | 61.4 | 61.8 |
| 5 | 64.7 | 63.2 | 62.7 | 64.5 | 64.0 |
| 6 | 66.1 | 61.5 | 63.6 | 64.1 | 64.0 |

Reviewer comment (Evaluation):
It would also strengthen the paper to tie in the findings of this work with findings in related works. E.g. in the Bongard-HOI benchmark that the authors claim is the most similar, do they have similar findings e.g. in terms of which methods perform better?

Author response:
We thank you for your suggestion! We've added discussions to the Appendix E (under "Results compared to relavent benchmark, ex. Bongard-HOI"). For your convinience, here are some highlights: Compared to Bongard-HOI, there are both shared observations and differences. 

1) shared observations: canonical FSL (ex. SNAIL, MetaBaseine) do not work well on both datasets, regardless what representation backbone is being used.

2) differences: 

    - the most effective methods differ (Bongard-OpenWorld is SNAIL vs. Bongard-HOI is Meta-Baseline);
    - Bongard-HOI is more sensitive to open-vocabulary pretrained representations (its concept vocabulary is limited); Bongard-OpenWorld is not as it is designed to be open vocabulary or open-world;
    - Bongard-HOI does not have results of LLM-based methods.

    The first and last observations are obvious. Here we would like to elaborate a bit on the second one. We offered some additional results of benchmarking CLIP (shown below), a relatively more free-form and open-world visual backbone on both Bongard-HOI (a prior work, focus on compositional generalization on closed-world concepts) and Bongard-Openworld. As you can see, with a much stronger backbone like OpenCLIP, ProtoNet can achieve 71.3% acc on Bongard-HOI (way higher than the previous Bongard-HOI SOTA 58.3% using ImageNet-1k pretrained backbone), but on Bongard-OpenWorld it can only reach 58.3%. In fact, Bongard-HOI discourages the use of such an open-world pretrained backbone like CLIP because it might undermine the challenges posed by the benchmark (compositional generalization will not be an issue anymore, as all concepts in training/generalization test have been saw by CLIP). Rather, Bongard-OpenWorld allows and indeed encourages all types of backbone and models by introducing free-form open-world concepts. And as the additional result suggest, we're successful -- even the powerful CLIP backbone is still far from beating our benchmark.

    | benchmark | method | image representation | aux. task? | avg. |
    | :---: | :---: | :---: | :---: | :---: |
    | Bongard-OpenWorld | ProtoNet | ImageNet-1k | &#10007; | 56.0 |
    | Bongard-OpenWorld | ProtoNet | OpenCLIP | &#10007; | 58.3 |
    | Bongard-HOI | ProtoNet | ImageNet-1k | &#10007; | 58.3 |
    | Bongard-HOI | ProtoNet | OpenCLIP | &#10007; | 71.3 |

Reviewer comment (Presentation):
In the caption of Figure 1, highlight ‘hard negatives’ in orange, like ‘distractors’ are highlighted in green, to match the (captions of the) images shown in that figure.

Author response:
Thanks for your meticulous observation. We have clarified it in our main paper, Figure 1 is intended exclusively to depict the Bongard-OpenWorld Problems for a better view. Actually, each Bongard-OpenWorld Problem comprises six positive and six negative images, along with a corresponding query image separately. Highlight and typos have all been corrected.

Reviewer comment (Writing):
Typo: “was not fine-tuning” → “was not fine-tuned” (in Table 2’s caption).

Author response:
Thanks for your meticulous observation. We have clarified it in our main paper, Figure 1 is intended exclusively to depict the Bongard-OpenWorld Problems for a better view. Actually, each Bongard-OpenWorld Problem comprises six positive and six negative images, along with a corresponding query image separately. Highlight and typos have all been corrected.