Paper: MOFI: Learning Image Representations from Noisy Entity Annotated Images (ICLR-2024)

Reviewer comment (Experiments):
The training of the MOFI model is limited to image-level correspondence to entities. However, in reality, a single image may contain multiple entities, necessitating the inclusion of region-level correspondence in the model's training. Unfortunately, MOFI does not take this aspect into consideration.

Author response:
**Regarding multiple entities** 

Because we extracted the entity from the alt-text, and the nature of short text length of the alt-text, we observed that it usually only describes the primary object / entity in the image. Taking the multi-entity annotation with region (bounding box) is a good direction and we believe it will be the future of MOFI model as well, we leave it as a future work.

Reviewer comment (Experiments):
I am more concerned about how MOFI performs on existing hot tasks, such as whether it is more advantageous to combine with LLM than CLIP, or whether MOFI can be used for image generation tasks.

Author response:
**Re Questions**

Thanks for the suggestion. We agree combining MOFI with a LLM or an image generation task is interesting and can be useful. However, we believe it is beyond scope of the paper, as it will require a significant amount of space to fully explore the impact when connecting with a LLM or a image generation task. It can be explored as one of the follow-up works.

Reviewer comment (Presentation):
The number of training data can be shown in Table 2 for more clear comparison.

Author response:
We appreciate the reviewer’s positive feedback and recognition of our contribution. We will add the number of training examples in table 2, and more baselines for table 3 and 4, as suggested.

Reviewer comment (Experiments):
Incomplete experimental comparison in Tables 3 and 4. Why is there only CLIP for comparison? Other methods for solving zero-shot/linear probe classification should be discussed and compared.

Author response:
We appreciate the reviewer’s positive feedback and recognition of our contribution. We will add the number of training examples in table 2, and more baselines for table 3 and 4, as suggested.

Reviewer comment (Novelty):
In terms of technical novelty, there seems to be minimal innovation in the proposed approach. The authors employ a standard contrastive learning approach alongside a standard classification objective, combined through a linear combination. In the context of entity extraction, the authors utilize a named entity recognizer, following an existing approach for learning entity embeddings from Wikipedia. Therefore, in terms of actual technical contribution, the paper seems rather limited, with the primary contribution appearing to be the extensive scale at which the model operates.

Author response:
First of all, we want to clarify that our main contributions are: 1) the way we curate the large scale classification dataset,  2) explore the different training objectives of the constructed dataset,  and 3) demonstrate the effectiveness of the resulting model. 
We advocate for an expansive view of novelty beyond methodology alone. Valuable contributions can stem from innovative experimental designs, the unveiling of new findings or results, or the introduction of novel training paradigms that have the potential to inspire and benefit the community.
    
Secondly, we agree that both contrastive and classification objectives have been well studied in the research community. However, we want to point out several differences of our work: 

1. Studying learned generic image representation using a noisy large scale classification dataset is new. When combined with the contrastive objective in a multitask setup, we demonstrate that the MOFI model achieves significantly better results on both zero-shot classification and image retrieval. Especially on the image-to-image retrieval benchmark GPR1200, the new model is 14% (absolute) better than OpenAI’s CLIP model.

2. The existing largest classification datasets people have used are JFT-3B and IG-3.6B, both of them are private and hard to reproduce. Our dataset is curated from public data, can be easily reproduced by others, and 66x larger in terms of number of classes. 

3. We need to adjust the normal classification objective (e.g. sampled negatives and add margin) to effectively learn from such a big number of classes, as detailed in section 3.1.
        
We will add more discussion about the contributions in the final version.

Reviewer comment (Experiments):
Regarding entity linking, the proposed approach appears weak. The authors do not conduct any form of multimodal entity linking. The process involves initial entity linking using entity embeddings on the text side, followed by a filtering step using a pre-trained CLIP model. A stronger approach could potentially involve integrating visual features into the knowledge embeddings, which might reduce the number of spurious entity linkings. Additionally, there is curiosity about whether the authors experimented with off-the-shelf entity linkers, like BLINK.

Author response:
We appreciate the reviewer’s suggestion on the entity linking, and will be happy to try BLINK as an alternative to our approach. Note that the main purpose of the entity annotation is to provide supervision to learn strong image representation. We find the results from the current noisy annotation have already shown strong performance on multiple benchmarks. Apart from the precision of entity annotation, it's crucial to factor in efficiency, enabling the processing of billions of texts in a feasible time frame. Our current approach can run 8.4B texts within 10 hours. While we think a better entity annotation approach is likely to further improve the performance, we leave it as a future work.

Reviewer comment (Experiments):
There is a question about why the training setting proposed in MOFI is considered optimal. Since CLIP, many works have explored contrastive model training. For example, the LiT paper demonstrated significant performance gains via locking the image branch and tuning the text model with a private dataset. There are also large foundation contrastive models like the open-source LAION-H-14 that could have been used for comparison. These models do not use MOFI's described objectives but seem to perform similarly, and they provide a more equitable comparison in terms of dataset size.

Author response:
It is hard to compare with other CLIP models as they are trained on different datasets. Some of them are private, e.g. LIT uses JFT-3B. And LAION is prohibited to use by our institution. OpenAI’s CLIP model has been widely adopted so we list it as a baseline model. In addition, we trained our in-house CLIP model using the image-text pairs from the same data source, and a similar number of examples as addressed in the previous paragraph. Given that, we believe our comparison is fair. 
        
We also note that the image representation of LiT model is learned from a classification objective, which is similar to our MOFI_sup-B/16 setting in Table 2, Row 7. MOFI_sup-B/16 has comparable performance on GPR1200, but worse performance on ImageNet retrieval metric.

Reviewer comment (Novelty):
It remains unclear if the concept of contrastive learning with a classification objective on entities is necessary. Other works like CoCa demonstrate that similar or better objectives could be achieved with contrastive learning plus captioning, which also captures entities, verbs, events, actions, and visual properties. CoCa outperforms the authors significantly on ImageNet zero-shot benchmarks. MOFI’s focus solely on entities omits other important relations that contrastive captioning could capture. This raises uncertainty about whether MOFI’s approach is superior to contrastive captioning-based approaches.

Author response:
CoCa is trained on private JFT-3B + LAION-1.8B data. It is not a fair comparison to compare our model v.s. original CoCa. We indeed implemented the contrastive captioning model and trained it on our data. However, in our preliminary experiments we didn’t observe the same performance gain compared to CLIP only as it is described in its original paper. We suspect it is because of the private training data (e.g. JFT-3B) they used.
        
Note, JFT 3B is also a classification dataset with 30k unique labels.

Reviewer comment (Experiments):
The multitask learning approach presented may not be the optimal way to utilize the data, as there is a lack of clear performance analysis against contrastive captioning-based models. Additionally, results are difficult to interpret due to differing dataset sizes.

Author response:
CoCa is trained on private JFT-3B + LAION-1.8B data. It is not a fair comparison to compare our model v.s. original CoCa. We indeed implemented the contrastive captioning model and trained it on our data. However, in our preliminary experiments we didn’t observe the same performance gain compared to CLIP only as it is described in its original paper. We suspect it is because of the private training data (e.g. JFT-3B) they used.
        
Note, JFT 3B is also a classification dataset with 30k unique labels.

Reviewer comment (Experiments):
There is a concern that filtering out entities using CLIP could lead to error amplification. If CLIP poorly captures an entity, using it to filter the dataset might mistakenly exclude that entity.

Author response:
Regarding the dataset contribution. The approach should be robust to the entity embedding model as it is gated by a CLIP model after the entity linking step. e.g. It should be easy to replace it by an open sourced entity embedding model as suggested. We can add an ablation study in the final version.  Once the entity embedding is ready, it only takes 10 hours using 256 machines to extract the entities from the text.

Reviewer comment (Evaluation):
The paper might have been better positioned as a very large-scale visual entity linking paper rather than a vision-language foundation model paper, since the key novelty appears to be the ability to disambiguate various entities. However, this raises further questions, such as how to ensure that common entities do not overwhelm long-tail entities. This issue is not explicitly handled in the approach, and the authors do not perform sufficient evaluation for the visual entity linking task.

Author response:
We also appreciate the reviewer’s suggestion on positioning the work as a very large scale visual entity linking paper. We are happy to extend the work to study more on the visual entity linking in the future work. However, the motivation of our work is to study the image representation learned from the large scale noisy entity annotated data. The results are strong, we believe it is worth sharing them as the current form and the intuition and results of the paper will benefit the community already