Paper: T-MARS: Improving Visual Representations by Circumventing Text Feature Learning (ICLR-2024)

Reviewer comment (Experiments):
In Section 4.1, why exactly 50% of the pool is filtered?

Author response:
While designing the approach and experimenting on the LAION-400M dataset, we tried three choices for the amount of data to be filtered— 40%, 50%, and 75%, with 50% working out to be the best. Note that these experiments further filtered an already filtered LAION dataset. One can definitely vary this and use other filtering metrics, say based on a threshold CLIP score after text masking. For example, on DataComp (filtering on the common crawl),  we use a CLIP-similarity threshold of 0.281, which is similar to that used in prior work, such as LAION filtering. Fine-tuning the threshold may bring further gains, but running such parameter searches is beyond our compute scope. We have clarified this in the updated manuscript as well.

Reviewer comment (Writing):
The writing of this paper is somewhat obscure, resulting in that it is some difficult to follow this paper.

Author response:
### **Writing**
**We have re-written** parts of the paper that we felt might have been hard to follow. This especially included a re-write of Section 6. If you have any particular comments on any other parts of the paper, we would be happy to make amends. Thanks!

Reviewer comment (Presentation):
Is it possible to directly remove all the text in the images? This may avoid the distractions of the text.

Author response:
### **Directly remove all the text in the images** 
The first step of TMARS, i.e. text-detection can indeed be seen as trying to identify and remove all the texts in the images. However, note that TMARS then filters out the images with low CLIP score after masking. 
This is because, as outlined in our pilot study (Section 3), a significant number of images primarily contain text as the sole feature correlating with the caption.  It would be suboptimal (waste of compute) to retain these images and spend training compute on them, as after masking, they do not have any visual features (correlated with caption) that could possibly help the model to learn better visual representations.

Reviewer comment (Experiments):
It would be better to conduct experiments on more datasets, except for LAION.

Author response:
### **Experiments on more datasets** 
We point to Table 2, where we also show results on the DataComp, a latest benchmark for multimodal data filtering. For example, on the medium scale of datacomp, TMARS observes around $6.5\%$ gains on ImageNet zeroshot accuracy compared to CLIP filtering and $4.4\%$ compared to TextMatch[1]. The goal of our work is to improve training on noisy, web-crawled datasets, and in that regard, we have shown effectiveness on both, filtering the common crawl, and pre-filtered LAION datasets. The common crawl is pretty much all the data on the web, and captures any subset of data that may be created out of it. We hope this helps acknowledge the significance of our results.

Reviewer comment (Evaluation):
Even though the proposed method has been evaluated on multiple datasets and various tasks, the metric is only the accuracy, which may be narrow and bias may exist for other metrics.

Author response:
### **Accuracy as the only metric** 
Sorry for not being precise about this, but we follow the standard evaluation metrics as used in the DataComp benchmark (https://github.com/mlfoundations/datacomp/blob/main/tasklist.yml). For example, on retrieval tasks (MSCOCO and Flickr) the metric is mean_recall@1 and on the WinoGAViL dataset it is jaccard_score. Further, we have updated Table 2 in the paper with an additional column of average performance across all the datasets in DataComp, which involves F1 score as the metric for some of the class imbalanced datasets like wilds-iWILDCam and worst region accuracy as the metric for wilds-FMoW.

Reviewer comment (Experiments):
The image's text overlaps with the caption which may not be helpful for learning visual features, a subsection for the discussion with multiple ways to resolve the issue can help researchers get more insight into this topic instead of simply exploiting the masking technique.

Author response:
### **New Discussion: Appendix G (Other ways to resolve text overlap with caption)**
We have added a discussion section on other ways to circumvent text feature learning in Appendix G  (Page 26 and Page 27), as requested by you.  We give a brief summary here:
- One can remove all inputs with any text features matching the caption. However, this is suboptimal as it removes datapoints with both visual and text features [1].
- Alternatively, one can consider a much more expensive process of modifying the input captions of the dataset. One way for the same would be to remove identified text OCR from the input captions. Another way would be to generate new captions altogether, as recently explored in [2].
- Finally, one could also consider modifying the input images to in-paint the OCR region. Note here that in T-MARS, we chose to use the masked images just to calculate new CLIP scores for filtering. A more elaborate discussion is provided in the appendix.

Reviewer comment (Evaluation):
The proposed filtering method is reasonable, but can this method be used for all different tasks with only one metric, i.e., accuracy? Is that possible that the method filtered some salient signals but isn't shown in this paper due to the single metric?

Author response:
### **Accuracy as the only metric** 
Sorry for not being precise about this, but we follow the standard evaluation metrics as used in the DataComp benchmark (https://github.com/mlfoundations/datacomp/blob/main/tasklist.yml). For example, on retrieval tasks (MSCOCO and Flickr) the metric is mean_recall@1 and on the WinoGAViL dataset it is jaccard_score. Further, we have updated Table 2 in the paper with an additional column of average performance across all the datasets in DataComp, which involves F1 score as the metric for some of the class imbalanced datasets like wilds-iWILDCam and worst region accuracy as the metric for wilds-FMoW.

Reviewer comment (Experiments):
When masking out the text of an image, will the inpaint technique alter the original data? If it's not an important issue, is that possible to leverage the power of generative models for it?

Author response:
### **Does inpainting alter original data?**
If the text overlays a patch with significant visual features, inpainting might create some aberrations. However, in general, we observe that the text detection algorithm (FAST[3]) gives very tight bounding boxes around the text, leading to no major aberrations of visual features when inpainting the text area, serving our main purpose in this work well. In Appendix E, we have added some pairs of original and masked images to highlight the same. 

Additionally, the histogram of CLIP score distribution in Appendix F (which you suggested) provides evidence that the method works well at masking out text features, thereby, lowering CLIP scores of only the images with just text features. However, we agree with you that exploring more nuanced ways to remove the text from the image is an interesting direction for future work.

Reviewer comment (Evaluation):
It'll be good if the authors provide the distribution scores (cosine similarity) before and after filtering out the data that can help understand the distribution of the good/bad data.

Author response:
### **Distribution of scores before and after masking**
Thank you for the suggestion. Recall our 500-example pilot study where we hand-labeled examples into various categories, like those with only visual features or those with only text features. We calculated the CLIP similarity score of these image-caption pairs, before and after masking the text in the respective images. **A histogram of this distribution of scores (before and after masking) is now uploaded in Appendix F**. Our observations indicate that after masking, the CLIP score of images with just text features (OCR) drops significantly (indicating they will be filtered out by T-MARS). At the same time, CLIP scores of images with only visual features stay nearly the same. This demonstrates the efficacy of the filtering method for removing samples with only text features.

Reviewer comment (Experiments):
In this paper, the proposed method removes the text of an image, will it be different if using different percent of the masking?

Author response:
### **Impact of percent of Masking**
To decide the threshold of image-caption pairs to be removed post-masking, we use a CLIP-similarity threshold of 0.281, which is the same as that used in prior work, such as LAION filtering. Fine-tuning the threshold may indeed bring further gains, but running such parameter searches is beyond our compute scope.

Reviewer comment (Experiments):
After filtering out the data, is data augmentation used in the experiments? Will it provide a more significant improvement?

Author response:
### **Data augmentation after filtering** 
We do not perform any additional data augmentation after filtering out the bad data to ensure a fair comparison with the baselines. We follow the standard and widely used CLIP training implementation given here[https://github.com/mlfoundations/open_clip]. Indeed, data augmentation may provide additional improvements in visual representation, but our work aims to improve representations by intervening on the data, not the training process.