Paper: Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association (ICLR-2024)

Reviewer comment (Experiments):
Lacking experiments using different key tools, e.g., LLM, detector/grounder, image generator, etc, which may affect the data generation a lot.

Author response:
Thank you for your suggestion! Reporting bias (as its property suggested) is often overlooked in existing models and datasets. As a result, it is non-trivel to reveal and fix it, hence requires multiple language/visual tools, making it hard to conduct large-scale experiments on a mass of combination of different foundation models. Hence, we use **state-of-the-art foundation models** to construct our framework, so that the generation quality could be as good as possible at the moment. We will leave the exploration of the effect of different foundation models in the future work to find the best practice for deployment.

Reviewer comment (Evaluation):
More solid analyses should be given to probe the relation between the data and the bias and performance of different tasks.

Author response:
Thank you for your thoughtful feedback. As we mentioned above, our experiments involves extensive data processing and model training, which can be quite time-consuming. Due to the time limit, it may not be feasible for us to add more experimental analysis by the end of the discussion period. However, we will continue to enhance the analysis and discussion sections in the future.

Reviewer comment (Presentation):
Prompt: The font color is hard to read.

Author response:
We have updated the prompt in Appendix C with carefully designed style for better readability.

Reviewer comment (Theory):
I suggest the authors also discuss the problems in the causal view, like counterfactual samples.

Author response:
Thank you for your thoughtful feedback. As we mentioned above, our experiments involves extensive data processing and model training, which can be quite time-consuming. Due to the time limit, it may not be feasible for us to add more experimental analysis by the end of the discussion period. However, we will continue to enhance the analysis and discussion sections in the future.

Reviewer comment (Evaluation):
Any user study of the image generation quality, prompt quality, and rationality of the V-l relations?

Author response:
Thank you for the suggestion! We add a subsection for user study in Section 4.4 in the updated paper. Please refer to the paper for details.

In summary, we did human evaluation for the foundation models outputs. We design five questions for the intermediate output:
1. Correct object: ''In step 1 and 2, does the framework generate reasonable objects?''
2. Implicit object: ''In step 1 and 2, do the valid objects contain the ones that are not explicitly mentioned in the original caption?''
3. Correct attribute: ''In step 3, does the framework add description for the correct given attributes?''
4. Valid negative: ''In step 4, can the pair of attributive objects serve as hard negative example to each other?''
5. Valid inpainting: ''In step 7, does the model inpatint the required object in the masked area?''

It shows that we got >90\% correctness for _correct object_ and _valid negative_, around 60\% for _correct attribute_ and _valid inpainting_. And around 60\% augmented examples contains objects that are not explicitly mentioned in the original caption

Reviewer comment (Reproducibility):
CLIP-ft: fine-tuning CLIP on the augmented data? Missing details.

Author response:
CLIP-ft refers to fine-tuning on the source dataset sampled from CC3M but not on the augmented data. CLIP-ft is the baseline variant we compare with the proposed method BiAug. Specifically, we first sample dataset with different sizes from CC3M, i.e. the _source dataset_, then we apply the proposed method on the source dataset to obtain the _augmented dataset_. CLIP-ft is CLIP trained on the source one and BiAug is CLIP trained on the augmented one. They are comparable because the datasets are from the same source.

We have modified the description in Section 5.1 to avoid misunderstanding.

Reviewer comment (Experiments):
It appears that the current method focuses on data augmentation for vision-language (VL) datasets by altering one attribute of one object at a time. It would be beneficial if the author could elaborate on whether they have considered modifying multiple attributes and multiple objects simultaneously.

Author response:
```
Q: It would be beneficial if the author could elaborate on whether they have considered modifying multiple attributes and multiple objects simultaneously.
```
A: This is a good point, especially we're processing a large scale of data. In fact, one attribute of one object at a time is just the example for presentation. We actually modify the all the four attributes for all valid objects to augment the captions, i.e. four attributes X multiple objects a time, please refer to Appendix C, prompt 2 for details.

However, for the image inpaining, we do one attribute-object pair at a time, because modifying multiple objects could corrupt the image, and more importantly, we would not be able to construct the hard negative pairs to highlight the attributive difference for one object.

Reviewer comment (Experiments):
In addition to conventional attributes like color, shape, and material, has the author explored more detailed attributes, as in [1], provided by Large Language Models (LLM)?

Author response:
```
Q: In addition to conventional attributes like color, shape, and material, has the author explored more detailed attributes
```
A: This is also a very good point, we did explore different attributes but found it would be hard to cover every attribute in the large-scale experiments. To address the diversity of attributes, we add an **_other_** attribute to leverage the LLM’s generalization capabilities to infer relevant attributes from provided captions, highlighting the most relevant commonsense knowledge besides color, shape and material. Please refer to footnote 2 in the 4th page, and the prompt 2 in Appendix C.
By doing this, LLM is able to generate attributive description dynamically based on the given caption, beyond the preset conventional attributes.

Reviewer comment (Experiments):
It is advisable for the authors to conduct a broader range of experiments and provide more visualization results, particularly in the context of Attribute Recognition Games (ARG) experiments.

Author response:
```
Q: Add ARG experiments.
```
A: Thanks for the suggestions! We're trying to add broader experiments by the end of discussion period if possible as the time limit. Could you also provide the reference or link to the ARG benchmark you mentioned? We'd also like to kindly note that we're focusing on the reporting bias issue (which is overlooked previously) in VL dataset, ARO is the most relevant benchmark to verify our ideas.

Reviewer comment (Presentation):
The meaning of the abscissa in Fig. 5 is not immediately clear. Further clarification or labeling may be needed for readers to grasp the content effectively. Furthermore, Fig. 5's subcaption would benefit from further adjustment.

Author response:
```
Q: The meaning of the abscissa in Fig. 5 is not immediately clear. Further clarification or labeling may be needed for readers to grasp the content effectively. Furthermore, Fig. 5's subcaption would benefit from further adjustment.
```
A: Thanks very much for the suggestion!
The x-axis refers to the size of the source dataset used for training the model, e.g., 40K, 100K, 200K and 300K, for demonstrating the trend of model's performance with increasing data used for fine-tuning.
We have added description for the x-axis in the caption of Figure 4 (Figure 5 in the original paper), and in Section 4.1 for better readability. As for the formatting in the figure, we'll carefully adjust them in the final version.

Reviewer comment (Evaluation):
The paper would benefit from a more extensive discussion of its limitations and potential directions for future research, offering a more comprehensive evaluation of the current method's scope and boundaries.

Author response:
```
Q: The paper would benefit from a more extensive discussion of its limitations and potential directions for future research, offering a more comprehensive evaluation of the current method's scope and boundaries.
```
A: Thank you for the suggestion! We agree that discussion would be very helpful. We cannot add another section due to the page limit. Please refer to Appendix A for our discussion about limitation and future directions.

Reviewer comment (Evaluation):
This paper utilizes multiple foundation models as black-box tools. It lacks of intuitive measurement of the quality of foundation models' output. For example, is there a numerical evaluation of the box quality generated in step 2 in Figure 2? In step 1 how is the quality of the "guessed" potential objects?

Author response:
## Measurement of the quality of foundation models outputs
That's a very good suggestion. So we add a subsection for that in Section 4.4 in the updated paper. Please refer to the paper for details.

In summary, we did human evaluation for the foundation models outputs, as there is no benchmarks that are exactly designed for our case. To check whether these foundation models follow our instruction to work, we design five questions for the intermediate output:
1. Correct object: ''In step 1 and 2, does the framework generate reasonable objects?''
2. Implicit object: ''In step 1 and 2, do the valid objects contain the ones that are not explictly mentioned in the original caption?''
3. Correct attribute: ''In step 3, does the framework add description for the correct given attributes?''
4. Valid negative: ''In step 4, can the pair of attributive objects serve as hard negative example to each other?''
5. Valid inpainting: ''In step 7, does the model inpatint the required object in the masked area?''

It shows that we got >90\% correctness for _correct object_ and _valid negative_, around 60\% for _correct attribute_ and _valid inpainting_. And around 60\% augmented examples contains objects that are not explicitly mentioned in the original caption

As for your questions,
```
is there a numerical evaluation of the box quality generated in step 2 in Figure 2? 
```
A: Grounding DINO achieves an impressive 60.7 mAP (refer to Table 2, [1]) in zero-shot object detection on the MSCOCO benchmark, without any prior training on this specific benchmark. This performance underscores its robust capability to reliably identify valid objects, which is essential for the effectiveness of our method.

```
In step 1 how is the quality of the "guessed" potential objects?
```
A: As the results shown Table 2 in the updated paper, the quality of the guessed potential objects are very good with a correctness >90\%.


[1] Liu, Shilong et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.” ArXiv abs/2303.05499 (2023): n. pag.

Reviewer comment (Evaluation):
This paper claims also to achieve improvement in general text-image retrieval benchmarks. However, in figure 5 table (c) and (d), the performance of BiAug and Clip-ft is rather close on the 40k subset, and with more examples, BiAug didn't show clear improvement compared with either the BiAug or CLIP-ft baseline on the 40k subset. These results can't really show a solid improvement on the two datasets.

Author response:
## Concern about the improvement on general VL retrieval task
The Filcker30K-order and COCO-order in Figure 4(Figure 5 in older verion) are not the results for general text-image retrieval task, instead it's the results on a very challenging ARO benchmark[2]. 
We have modified the paper to avoid misunderstanding.
In ARO, the original Filcker and COCO captions are modified by changing partial word order, which are very similar to the original caption. This benchmark requires the VL model to pick the correct caption from a group of _hard negative_ examples. Our results show that BiAug has a more stable trend in distinguish _hard negative_ examples.

The claim that BiAug also acheives improvement on general image-text retrieval task is supported by the results in Table 3. Please refer to the paper for details.

[2] Yuksekgonul, Mert et al. “When and why vision-language models behave like bags-of-words, and what to do about it?” ArXiv abs/2210.01936 (2022): n. pag.

Reviewer comment (Experiments):
This work only compares with two baselines: CLIP and CLIP-ft. It is unclear whether BiAug can also benefit other VL models such as NegCLIP (Yuksekgonul et al., 2022) which has been trained with explicitly constructed negative samples.

Author response:
## Comparison with NegCLIP
We're still reproducing the NegCLIP to make the comparison. We're afraid that it cannot be finished by the end of the discussion period as it requires a lot of time to preprocess the 300K data and train the models. We will update the results as soon as possible.

However, we want to note that BiAug and NegCLIP are not actually comparable because our focus is to mitigate the reporting bias issue by decoupling the object-attribute association for **both visual and langauge modalities**, while NegCLIP just simply shuffle the caption, without making any changes to the objects or attributes in the caption and the image.

Reviewer comment (Theory):
I find the framework overly complicated and built on fragile assumptions. For example, why can we assume that the grounding object detector can find objects that a different color or shape of it is still a valid object? How can we make sure that the detected grounding object is big enough for the in-painting model to process this? Also see my questions below. I believe this overall complexity hinders the generalization of the results.

Author response:
## Concern about the relative complicated framework
Reporting bias (as its property suggested) is often overlooked in existing models and datasets. As a result, it is non-trivel to reveal and fix it, such that our method consists of different important steps. For your specific concerns, we clarify them point by point as follows,

```
Q: “why can we assume that the grounding object detector can find objects that a different color or shape of it is still a valid object?”
```
A: It is important to emphasize that our system utilizes the state-of-the-art grounding detector, Grounding DINO [1], for detecting valid objects. This advanced detector is particularly efficient in recognizing objects based on textual input alone, even in a zero-shot context. Notably, Grounding DINO achieves an impressive 60.7 mAP (refer to Table 2, [1]) in zero-shot object detection on the MSCOCO benchmark, without any prior training on this specific benchmark. This performance underscores its robust capability to reliably identify valid objects, which is essential for the effectiveness of our method. Though GroundingDINO is generalizable, we agree that it is not perfect and could give mistakes. However, in our qualitative/human inspections in Section4.4, we found that GroundingDINO is generally reliable. And we expect that more advanced grounding models will be developed in the community and can be easily integrated to BiAug.

```
Q: “How can we make sure that the detected grounding object is big enough for the in-painting model to process this?”
```
A: We remove the detected boxes that are too large or too small to ensure they are valid for the following procedure. Please refer to the filtering strategies introduced in Section 4.1.

```
Q: “How the framework described in Figure 2 could be reliable for large scale experiments.”
```
A: We apply multiple filtering strategies, as introduced in Section 4.1, to remove low-quality grounding objects/augmented captions/generated images. Filtering strategies makes the examples we finally used for training are reliable as possible. We agree that we still cannot avoid noisy examples, even BiAug is utilizing state-of-the-art language and visual foundation models. The experimental results on general visual-language retrieval task shown in Table 3, also demonstrate that the dataset augmented by BiAug is reliable enough for training a VL model, as the performance keeps increasing with finetuning on the augmented dataset.

[1] Liu, Shilong et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.” ArXiv abs/2303.05499 (2023): n. pag.

Reviewer comment (Writing):
Overall I did not find the paper easy to read. This is partly due to the fact that the methodology is very complex with lots of pieces. However, I think the authors need to spend more time thinking about the structure and also definitely re-writing the abstract.

Author response:
```
Q: However, I think the authors need to spend more time thinking about the structure and also definitely re-writing the abstract.
```
Thank you for your insightful questions and suggestions. For your main concern about the presentation of this paper, we have carefully rewritten the abstract and method, updated the figure2, and updated the demonstration of prompt for LLM, for better readability. Please see the updated paper. Please also see our clarification below to your questions about the detail of the method.

Reviewer comment (Writing):
Many important concepts used in the paper are not clearly explained. For example "grounding object" need clear definitions.

Author response:
```
Q: Many important concepts used in the paper are not clearly explained. For example "grounding object" need clear definitions.
```
A: Thank you for the suggestion. I went through the paper and add necessary background information for better readability.

Reviewer comment (Experiments):
I find it hard to understand how the framework described in Figure 2 could be reliable for large scale experiments.

Author response:
```
Q: “How the framework described in Figure 2 could be reliable for large scale experiments.”
```
A: We apply multiple filtering strategies, as introduced in Section 4.1, to remove low-quality grounding objects/augmented captions/generated images. Filtering strategies makes the examples we finally used for training are reliable as possible. We agree that we still cannot avoid noisy examples, even BiAug is utilizing state-of-the-art language and visual foundation models. The experimental results on general visual-language retrieval task shown in Table 3, also demonstrate that the dataset augmented by BiAug is reliable enough for training a VL model, as the performance keeps increasing with finetuning on the augmented dataset.

Reviewer comment (Reproducibility):
Where do the samples in Figure 1 come from? Are they from a real dataset? Are they generated?

Author response:
```
Q1: Where do the samples in Figure 1 come from? Are they from a real dataset? Are they generated?
```
A1: The “brown dog” example, which is also shown in Figure2, is from the CC3M that we used for our experiments, and the augmented two are generated by our framework. The other examples are collected from the internet for better conveying our motivation.

Reviewer comment (Presentation):
In Figure 2, in step 3, why "tennis ball" is dropped? It was part of the original text. Why "dog" is selected as the grounding object?

Author response:
```
Q2: In Figure 2, in step 3, why "tennis ball" is dropped? It was part of the original text. Why "dog" is selected as the grounding object?
```
A2: Some objects are kept, and others are removed based on our filtering strategies introduced in the last paragraph in Section 4.1. We remove too small and too large objects. We only keep objects that do not occupy over 70% of another object’s area, and a relatively higher grounding confidence to ensure the quality of examples as possible. In Figure 2, dog and grass are kept after filtering, and we take combination of “color” of “grass” as an example to illustrate the following steps.

Reviewer comment (Reproducibility):
In section 3.2 how do you select from the list of predefined categories? For example, in figure 2, how come the "shape" is not selected for "grass"? Of course it does not make sense to modify the shape for grass, but how do you account for that and select "color"?

Author response:
```
Q3: In section 3.2 how do you select from the list of predefined categories? For example, in figure 2, how come the "shape" is not selected for "grass"? Of course it does not make sense to modify the shape for grass, but how do you account for that and select "color"?
```
A3: “color” is just an example for illustration. We ask LLM to generate commonsense knowledge for all the four preset attributes, while an attribute-object pair will be skipped (which will be decided by the LLM as we require) if the object-attribute pair does not make sense. Please refer to Prompt 2 in Appendix C. Here we take a sentence in the prompt that we use to do this:

_If the generated [COMMONSENSE] is unclear, unknown, or not applicable, based on the [CAPTION], please skip the pair of [OBJECTS] and [ATTRIBUTES];_

Reviewer comment (Reproducibility):
In 3.3 how do you mask the detected object?

Author response:
```
Q4: In 3.3 how do you mask the detected object?
```
GroundingDino predicts the grounding box for the object, then we apply that box area as the mask for inpainting model. The stable-diffusion-inpainting takes an image, a mask, and a text prompt as input to edit the original image.

Reviewer comment (Reproducibility):
In the example given in figure 3, what happens if none of the predicted objects are inside the image?

Author response:
```
Q5: In the example given in figure 3, what happens if non of the predicted objects are inside the image?
```
A5: In that case the grounding object detector will detect zero objects. As a result this example will be skipped because we get an empty list at Step 2.

Reviewer comment (Experiments):
In figure 4, the "blue boat" and "red boat" samples, what was the missing reference object here according to your framework? What happened in the extraction phase? boat was already part of the original caption.

Author response:
```
Q6: In figure 4, the "blue boat" and "red boat" samples, what was the missing reference object here according to your framework? What happened in the extraction phase? boat was already part of the original caption.
```
A6: In this example, we augment the original sample in terms of “missing attributive description” instead of “missing object”. Our proposed BiAug can augment the VL datasets in two folds: (1) complete the missing objects or missing attributes. (2) hard negative pairs guide the VL model learn to discern the attributive difference for identical object. Not all examples are applicable to all (but at least one) these perspectives.

In the extraction phase, the object of ‘boat’ (but without the description in terms of color) was detected as explicitly mentioned objects. Subsequently, in the phase II, the boat object will be augmented in terms of the four preset attributes. For the case of color, it would be ‘red boat’ or ‘blue boat’.

Please also refer to the table 2 in the updated paper, around 40\% of examples only contains valid object that are part of the original caption.

Reviewer comment (Writing):
What is "other" in the preset attribute list?

Author response:
```
Q7: What is "other" in the preset attribute list?
```
A7: It is the word “other”, in our prompt we ask LLM to infer commonsense concept for every object-attribute pairs, e.g., grass-color, grass-shape, grass-other. We add this to leverages the LLM’s generalization capabilities to infer relevant attributes from provided captions, highlighting the most relevant commonsense knowledge besides color, shape and material.