Paper: PaLI-3 Vision Language Models: Smaller, Faster, Stronger (ICLR-2024)

Reviewer comment (Experiments):
I strongly encourage authors to provide more comparisons between CLIP and SigLIP under this paper's setting. The current ablation only includes the comparison between SigLIP and vanilla classification.

Author response:
We thank the reviewer for the suggestion. We would like to note that we do not claim contrastively pretrained visual encoder is always superior to a classification-pretrained one for all purposes. Instead, we reported our observations on contrastively trained SigLIP that “despite SigLIP slightly underperforming on image classification, it performs much better on Visual Language tasks.” (See our contribution #1) with a focus on the relative advantage of contrastively trained models on VL tasks, which we believe holds true for similarly trained models including OpenCLIP. That said, we agree with the reviewer that the comparison to OpenCLIP may provide a complementary datapoint for the community. We are working on setting up and performing such a comparison and are trying to add an additional baseline based on OpenCLIP before the final version of the draft.

Reviewer comment (Experiments):
I understand the paper mainly focuses on a smaller and cheaper model, as stated in the title. However, I think it is important to study the scaling results to check the effectiveness on a larger scale. Can the SigLIP still be so effective when using a larger vision encoder and language models? It would also be interesting to further scale down the model and see what would happen.

Author response:
We agree with the reviewer that it is important to study the scaling laws to check the effectiveness on a larger scale. The result in Table 1 is about the scaling behavior, where we first tested SigLIP sizes B/16 and L/16, and the promising scaling behavior was exactly what drove us to push SigLIP to 2B (G/14) parameter scale used for PaLI-3.

Reviewer comment (Novelty):
Could you provide a detailed discussion about the difference between PaLI-3 and the previous versions? It would be better to have a table and show the differences directly.

Author response:
We thank the reviewer for this suggestion. Please see the two main points below. We will expand and add to the revised paper.
- PaLI-3 used SigLIP pretrained ViT, while previous versions used the classification pretrained ones.
- The training mixture of PaLI-3 is based on the learnings of PaLI-X, with a few key new components, including the further filtered and refined WebLI dataset, especially on the OCR data, and we curated and added web-scale document images in the training.

Reviewer comment (Experiments):
The main weakness of PaLI-3, from my perspective, is the way the authors used to draw their conclusion. Specifically, the authors claim that because SigLIP shows better performance than the classification-pretrained visual encoder used by PaLI and PaLI-X, they conclude that a contrastively pretrained visual encoder is superior to a classification-pretrained one. However, it's worth noting that most of the accessible contrastively pretrained visual encoders for the vision and vision-language community are members of the OpenCLIP family. Have you ever attempted to utilize OpenCLIP as a vision encoder?

Author response:
We thank the reviewer for the suggestion. We would like to note that we do not claim that “contrastively pretrained visual encoder is superior to a classification-pretrained one for all purposes”. Instead, we reported our observations on contrastively trained SigLIP that “despite SigLIP slightly underperforming on image classification, it performs much better on Visual Language tasks.” (See our contribution #1) with a focus on the relative advantage of contrastively trained models on VL tasks, which we believe holds true for similarly trained models including OpenCLIP.

For OpenCLIP, We have not tried it at the time of submission, since SigLIP is also open-source and significantly better across benchmarks. However, we agree this can be an interesting ablation, for which we have started to set up and are trying to add an additional baseline based on OpenCLIP before the final version of the draft.

Reviewer comment (Experiments):
The results in Section 4 are per-benchmark finetuned. What's the performance of PaLI-3 without task-specific fine-tuning (zero-shot)? Is it possible to generate target answers with few-shot demonstrations by prompting (in-context learing)?

Author response:
We expect the zero-shot performance to be relatively poor, since there is no task aimed for this in the pre-training mixture, and hence the model cannot know "the coco vocabulary" from which it should generate. PaLI-3 is also not a chatbot, but rather a (compact) foundation model capable to be transferred to various tasks.

Meanwhile, few-shot in-context ability is a good idea that we could and likely should add in the pre-training mixture, and would provide an in-between alternative between fine-tuning and zero-shot.