Paper: Improved Probabilistic Image-Text Representations (ICLR-2024)

Reviewer comment (Evaluation):
Presented as Table 1 and 2, when ViT-B/32 was chosen as the backbone, the performance of PCME++ is inferior to VSE\infnite. Additionally, the performances of P2RM and DAA published in the original paper actually won PCME with a large margin, while their performances presented here are lower than PCME. The authors should make necessary explanations on these issues.

Author response:
**[W1] PCME++ performance**

Thanks for the question. First, we would like to emphasize that PCME++ is not inferior to VSE infty in all ranking-based metrics, such as mAP@R, R-Precision, and RSUM. It is because VSE infty is based on triplet loss using the hardest negative mining that is overfitted to recall@1 score as supported by Chun et al. 2022. Moreover, when the backbone size becomes larger, VSE infty shows inferior performances than PCME++ with a large margin in all metrics.

We presume that it is because when the backbone size is sufficiently complex to be overfitted to the false negatives, deterministic methods easily fall to the overfitted solutions. We remark that the training dataset suffers from severe false negatives (88.2% of caption-to-image positives and 72.1% of image-to-caption positives are labeled as “negative” [Chun et al 2022]). On the other hand, the probabilistic methods are designed to handle many-to-many relationships, by less penalizing plausible matches (e.g., if an image-text pair is annotated as negative, but if the model predicts they are positive with high probability, the loss value becomes smoothed by its design – As discussed in Sec 2.3). It means that the effect of the FNs is saturated during training PCME++; hence it is profitable to be trained with large backbones.

Second, both P2RM and DAA are basically an additional regularization to the existing loss function (P2RM employs less panelized additional triplet loss, and DAA employs CIDER-based extra information), usually triplet loss with the hardest negative mining. Furthermore, when we trained the models, we found that their methods are very sensitive to the hyperparameter selection, as shown in below:

| .                  | mAP  | RSUM  |
|--------------------|------|-------|
| P2RM (a=1)         | 8.3  | 158.8 |
| P2RM (a=0.2)       | 39.0 | 530.2 |
| DAA (b=1)    | 39.2 | 530.9 |
| DAA (b=0)    | 40.0 | 536.5 |
| PCME++               | 40.1 | 537.0 |

Perhaps, if we more carefully tune the hyperparameters for the methods, we would get slightly better performances than the reported numbers. However, when we tried to apply DAA and P2RM, we frequently faced training instability by choosing larger hyperparameters than our choice. Note that we use the official implementation of DAA for the comparison, and P2RM is our own implementation because there is no official code.

Finally, the backbones used for our experiments differ from those of DAA, P2RM, and even PCME and VSE infty. Our implementation is a new and very strong setting based on CLIP backbones, showing the state-of-the-art retrieval performances against the existing visual semantic embeddings (VSEs) as shown in Tab C.1.

Reviewer comment (Experiments):
Most importantly, could the proposed PP and MSDA be applied to P2RM and DAA to enhance the many-to-many performance? If not, please make the necessary explanations; if yes, pls make comparisons.

Author response:
**[W1] Applying PP and MSDA to comparison methods**

We would like to emphasize that applying PP and MSDA to the triplet-based method is non-trivial. This is because the triplet loss is not designed for taking smooth labels. When we construct a triplet, we select an anchor sample (e.g., a visual embedding), its positive sample (e.g., its corresponding textual embedding), and its negative sample (e.g., one of the other textual embeddings). Here, for selecting the negative sample, triplet methods use the hardest negative mining, which uses the closest textual embedding as the negative. We argue that as there are abundant false negatives, it can harm the performances. The hardest negative mining is, therefore, very sensitive to the choice of batch size, as shown by VSE++ (https://github.com/fartashf/vsepp/issues/27).

If we use a smooth label for a given pair, it is impossible to construct a triplet of <anchor, positive and negative>. For example, assume we set a match annotation of <image_a, caption_b> to 0.6 from 0.0. How can we build triplets using this annotation? Moreover, if we introduce mixed samples and mixed labels, the problem becomes more complex. How can we handle <image_a_mixed_b, caption_a> and <image_b_mixed_a, caption_b> using a triplet relationship? Therefore, it is impossible to apply PP and MSDA for triplet-based methods.

One of our technical contributions is allowing smooth labels and mixed samples by designing a pairwise loss that is not affected by the other data samples. As shown in Fig 2., each loss computation of PCME++ is independent on the other pairs, while triplet loss or batch-wise contrastive loss is dependent on the relationships of other pairs.

Reviewer comment (Experiments):
The datasets used in this submission including CxC and ECCV both come from MSCOCO, and thus the proposed method needs further verification on larger datasets like vision-language pretraining methods (CC3M dataset).

Author response:
**[W2] Larger datasets**

We already provided the results of our model trained on the RedCaps dataset that contains 12M+ image-text pairs (Table 5). It is four times larger than the CC3M dataset. We compare the RedCaps pre-trained PCME++ and InfoNCE (i.e., CLIP) on zero-shot ImageNet classification. As shown in Table 5, we show that the probabilistic representation has a huge benefit when we fully utilize the knowledge from the learned uncertainty. Note that our main goal is not to propose a new vision-language pre-train model, but to examine the possibility of the uncertainty-aware methods on large-scale pre-training.

Reviewer comment (Experiments):
The authors are also recommended to make comparisons against several recent works focusing on uncertainty modeling [A] and noisy correspondence [B].

Author response:
**[W2] Other comparison methods**

Thanks for the references. First, we would like to mention that MAP is an out-of-scope work of ours. It is because we only compare with the visual semantic embedding (VSE) methods that have separated visual and textual encoders, directly trained on the target dataset (i.e., MS-COCO). On the other hand, MAP is based on a cross-attention Transformer, which is not suitable for retrieval tasks. It should compute all pairwise similarity in the dataset, namely 5000 * 25000 = 125M forward time for measuring COCO-5K retrieval performances (it usually takes more than tens of hours with a single GPU), while VSE methods only need to forward 5000 + 25000 = 30k forward paths. However, we agree that MAP is a good reference that shows the effectiveness of uncertainty-aware learning in vision-language models. We will add a related discussion.

We also thank to the reviewer for introducing DECL [B]. DECL-SGRAF shows 69.6 COCO 1K R@1, 49.4 COCO 5K R@1, and 518.2 RSUM in 20% noise, while PCME++ shows 71.6, 50.4, and 525.6, respectively. In 50% noise, DECL-SGRAF shows 68.3 COCO 1K R@1, 46.8 COCO 5K R@1 and 513.5 RSUM, while PCME++ shows 67.6, 45.5, 511.0, respectively. It is encouraged that although our method is not specifically designed for noisy correspondence, PCME++ shows better (in 20%) or comparable (in 50%) performances against the noisy correspondence-targeted methods without bells and whistles. We will add the numbers in Table 2 and related discussions.

Reviewer comment (Evaluation):
About the advantage of PCME++, the authors mentioned the advantage of PCME++ when scaling-up backbones, which might be lacking further discussions. (See Questions)

Author response:
Thanks for your comment. The discussion related to the question is in Sec 3.2. Main results. “We conjecture that it is because the non-probabilistic counterparts are easily overfitted toward FNs when the model has more complexity, particularly with the hardest negative mining strategy.” Here, we provide more details beyond this explanation.

As shown in Fig 1, the training dataset already suffers from severe false negatives (88.2% of caption-to-image positives and 72.1% of image-to-caption positives are labeled as “negative” [Chun et al 2022]). As the backbone becomes more complex and larger, we presume that the backbone complexity is sufficiently large to capture the noisy FNs in the dataset. Moreover, VSE infty uses a triplet loss with the hardest negative mining, which will make the effect of FNs more significant. On the other hand, the probabilistic methods are designed to handle many-to-many relationships by less penalizing plausible matches (e.g., if an image-text pair is annotated as negative, but if the model predicts they are positive with high probability, the loss value becomes smoothed by its design – As discussed in Sec 2.3). It means that the effect of the FNs is saturated during training PCME and PCME++. However, as we discussed in Sec 2.3., at the same time, it can cause a fast loss saturation for PCME. We address the problem by proposing two techniques: pseudo-positives and mixed data sample augmentation. Note that it is non-trivial to employ the techniques to triplet-based metric learning methods (VSE infty) and batch-wise contrastive learning methods (InfoNCE).

Reviewer comment (Evaluation):
In Table 1, experimental results on different backbones are provided. While methods like VSE and InfoNCE show performance degradation, PCME and PCME++ show consistent improvements. Does this phenomenon relate to specific advantages of probabilistic modeling? (and How)

Author response:
Thanks for your comment. The discussion related to the question is in Sec 3.2. Main results. “We conjecture that it is because the non-probabilistic counterparts are easily overfitted toward FNs when the model has more complexity, particularly with the hardest negative mining strategy.” Here, we provide more details beyond this explanation.

As shown in Fig 1, the training dataset already suffers from severe false negatives (88.2% of caption-to-image positives and 72.1% of image-to-caption positives are labeled as “negative” [Chun et al 2022]). As the backbone becomes more complex and larger, we presume that the backbone complexity is sufficiently large to capture the noisy FNs in the dataset. Moreover, VSE infty uses a triplet loss with the hardest negative mining, which will make the effect of FNs more significant. On the other hand, the probabilistic methods are designed to handle many-to-many relationships by less penalizing plausible matches (e.g., if an image-text pair is annotated as negative, but if the model predicts they are positive with high probability, the loss value becomes smoothed by its design – As discussed in Sec 2.3). It means that the effect of the FNs is saturated during training PCME and PCME++. However, as we discussed in Sec 2.3., at the same time, it can cause a fast loss saturation for PCME. We address the problem by proposing two techniques: pseudo-positives and mixed data sample augmentation. Note that it is non-trivial to employ the techniques to triplet-based metric learning methods (VSE infty) and batch-wise contrastive learning methods (InfoNCE).

Reviewer comment (Novelty):
This work is deeply coupled with PCME( Chun et al. (2021)), which might limiting the inspiration and extensibility for other work.

Author response:
**[W1] Inspiration and extensibility for other work**

We would like to emphasize that our work is an independent image-text matching work, not depending on specific methods. PCME++ is not strongly coupled with PCME. We will discuss it in more detail in the next bullet (**Similarity between PCME loss function and PCME++ loss function**). We presume that the reason why the reviewer feels that PCME++ limits the inspiration and extensibility for other work is that other approaches are usually based on deterministic methods. The technical details for learning a probabilistic method are somewhat difficult to be applied to deterministic methods. However, our work brings attention to the importance of the false negatives in the Image-Text matching training dataset (as pointed out by Reviewer RNqC). We explain that the FNs can be problematic for deterministic methods, especially for larger backbones, such as ViT-L/14. We do not believe that a probabilistic approach is the only method to tackle FNs. We believe that our work can give inspiration to other ITM methods to tackle the FN problem raised in our work.

**[W1, W3] Similarity between PCME loss function and PCME++ loss function**

We would like to emphasize that the similarity between PCME and PCME++ is due to the nature of the probabilistic embeddings (probemb). The main goal of probemb is to map an input to a random variable (r.v.) rather than a deterministic vector. We would like to emphasize that the core ideas of other probembs are also similar to each other, for example, Oh et al. 2019, Sun et al., 2020, Shi and Jain, 2019, Chang et al., 2020, SilnovaPark et al., 2022a et al., 2020 and Neculai et al., 2022. PCME (Chun et al. 2021) and our method, therefore share the property of probemb: we train an encoder that maps an input to a mean vector and a variance vector and train the encoder by maximizing the negative loglikelihood (NLL) using the extracted distributions.

PCME optimizes NLL by defining “matching probability”: 
$p(m | x_v, x_t) = \mathbb E_{Z_v, Z_t} \text{sigmoid} (-a \|\| Z_v - Z_t \|\| + b) \approx \frac{1}{J^2} \sum_j^J \sum_{j^\prime}^J \text{sigmoid} ( - a \|\| z_v - z_t \|\| + b)$

On the other hand, PCME++ optimizes the NLL using binary cross-entropy loss:
$p(m | x_v, x_t) = \text{sigmoid} (-a \mathbb E_{Z_v, Z_t} \|\|Z_v - Z_t\|\|^2 + b)$

Although they look somewhat similar, PCME takes the expectation over the sigmoid of the pairwise distance of r.v.s, while PCME++ takes the expectation over the pairwise distance of r.v.s before taking the sigmoid. The latter has two benefits over the former: (1) our form has a closed-form solution, as shown in Eq (1), while PCME cannot. (2) our form can be naturally adopted into the binary cross entropy loss function, which is known to be stable and perform well in large-scale training [1].

Therefore, we argue that PCME and PCME++ use different loss functions in terms of the NLL formulation and the probabilistic distance (PCME uses the Monte-Carlo approximation of “matching probability”, and PCME++ uses the closed-form sampled distance, CSD).

[1] ResNet strikes back: An improved training procedure in timm

Reviewer comment (Evaluation):
As shown in Figure 5, the visualization could show the uncertainty of learned embeddings of visual features and textual features. However, the proposed closed-form sampled distance (CSD) could only simply measure the uncertainty but not the area of the overlap between two modality. I hope the authors could make more analysis.

Author response:
**[W2] Area of overlap between two modalities vs. CSD**

Thanks for the valuable feedback. First, we would like to emphasize that we never mentioned the area of overlap between the two modalities in the paper. The purpose of Fig 5 is to show how the learned embedding space can successfully capture the inherent ambiguity in the dataset. Qualitatively, PCME++ embedding space captures the ambiguity of the many-to-many relationships despite the false negatives.

Second, the overlap between the two distributions itself is not directly related to the uncertainty. Conceptually, the overlap between two Gaussian distributions can be represented as the Bhattacharyya coefficient (or Bhattacharyya distance). Here, we recall the CSD’s main property (b) discussed in Sec 2.2: if the match $m_{vt}$ is certain, then $Z_v$ and $Z_t$ have small variances. As we discussed in Sec 2.2, a similarity function that measures whether two distributions are the same or not cannot satisfy the property because it cannot handle the case when the distributions have the same $\mu$, but larger $\sigma$. There is no motivation to reduce the size of $\sigma$ using the distance. As shown in Table 4, Bhattacharyya distance is not effective probabilistic distance for learning probemb as much as CSD. On the other hand, the learned embedding space by CSD-trained PCME++ is a reasonable probabilistic space. Table C.5 shows that even though we use Wasserstein distance as the similarity function of retrieval, the overall precision-based retrieval performances are almost preserved; it means that the probabilistic space learned by PCME++ is a sound metric space of Wasserstein distance. 

Third, instead of focusing on the overlap between two distributions of the learned embedding space, we would like to suggest how CSD can learn the embedding space shown in Fig 5. We recall the properties of our desired probabilistic embedding space: (1) if the correspondence between a given image and caption is certain (i.e., they are certainly positive or negative), then the variance of each instance should be small, (2) if the correspondence is uncertain (i.e., the match is sometimes positive and sometimes negative. It can be happened due to the false negatives in the dataset as shown in Fig 1 and Sec 2.1), then the variance of each instance should be large. As we mentioned before, CSD can give proper supervision for the desired embedding space. For example, let’s approximate the plane figures in Fig 5 to a visual embedding $v$ and the plane captions to a textual embedding $t$ (because they have very high semantic similarity). In this case, by choosing different image-caption pairs, the supervision between $v$ and $t$ can be either positive or negative because our training dataset only has one-to-one positive correspondences. In this case, our objective function enforces the CSD between the matches to be larger until the penalty for the positive supervision and the negative supervision are balanced. As shown in Fig 5, the visual embeddings and textual embeddings are aligned into almost the same place by making the $\mu$ distance closer but penalizing the uncertain supervision by enlarging $\sigma$.

Reviewer comment (Novelty):
Some techniques such as Mixup (Zhang et al., 2018) or CutMix (Yun et al., 2019) are proposed in the previous works, and the loss functions are shared with PCME( Chun et al. (2021)). Please clarify your contribution and improvements.

Author response:
**[W1, W3] Similarity between PCME loss function and PCME++ loss function**

We would like to emphasize that the similarity between PCME and PCME++ is due to the nature of the probabilistic embeddings (probemb). The main goal of probemb is to map an input to a random variable (r.v.) rather than a deterministic vector. We would like to emphasize that the core ideas of other probembs are also similar to each other, for example, Oh et al. 2019, Sun et al., 2020, Shi and Jain, 2019, Chang et al., 2020, SilnovaPark et al., 2022a et al., 2020 and Neculai et al., 2022. PCME (Chun et al. 2021) and our method, therefore share the property of probemb: we train an encoder that maps an input to a mean vector and a variance vector and train the encoder by maximizing the negative loglikelihood (NLL) using the extracted distributions.

PCME optimizes NLL by defining “matching probability”: 
$p(m | x_v, x_t) = \mathbb E_{Z_v, Z_t} \text{sigmoid} (-a \|\| Z_v - Z_t \|\| + b) \approx \frac{1}{J^2} \sum_j^J \sum_{j^\prime}^J \text{sigmoid} ( - a \|\| z_v - z_t \|\| + b)$

On the other hand, PCME++ optimizes the NLL using binary cross-entropy loss:
$p(m | x_v, x_t) = \text{sigmoid} (-a \mathbb E_{Z_v, Z_t} \|\|Z_v - Z_t\|\|^2 + b)$

Although they look somewhat similar, PCME takes the expectation over the sigmoid of the pairwise distance of r.v.s, while PCME++ takes the expectation over the pairwise distance of r.v.s before taking the sigmoid. The latter has two benefits over the former: (1) our form has a closed-form solution, as shown in Eq (1), while PCME cannot. (2) our form can be naturally adopted into the binary cross entropy loss function, which is known to be stable and perform well in large-scale training [1].

Therefore, we argue that PCME and PCME++ use different loss functions in terms of the NLL formulation and the probabilistic distance (PCME uses the Monte-Carlo approximation of “matching probability”, and PCME++ uses the closed-form sampled distance, CSD).

[1] ResNet strikes back: An improved training procedure in timm

**[W3] Augmentation methods are already proposed in previous works**

Although we do not propose a new MSDA, our contribution lies in applying MSDA to the relational datasets. For example, applying MSDA to classification is straightforward because the mixed sample does not affect the other samples in the mini-batch. However, in the relational training objectives, such as triplet loss or contrastive loss, a mixed sample affects the other samples in the batch as well. Especially, the triplet loss is impossible to handle MSDA, because the core concept of MSDA is the smooth label but the triplet loss cannot handle smooth label, because it has to construct a triplet of the selected sample, the positive sample, and the negative sample. It is non-trivial to define positive and negative samples when the label is smoothed (Fig 2a). Similarly, a batch-wise contrastive loss, such as InfoNCE, is also a little bit tricky to control the effect of smooth labels (Fig 2b) because the mixed samples are combined in the denominator term of the InfoNCE softmax. On the other hand, our pairwise contrastive loss can directly apply smooth labels because each loss computation is invariant to the other samples, but only the given pairs affect the loss computation (Fig 2d).

Overall, we do not argue that we newly propose MSDA (CutMix and Mixup), but we argue that we first apply MSDA to image-text matching methods that are based on relational datasets, not instance-level datasets (e.g., image-only datasets).

Reviewer comment (Presentation):
Figure 2 is not easy-understood. Please make interpretations about it and formulate the comparison losses.

Author response:
**[Q1] Clarity**

We will revise Figure 2 following the reviewer’s comment. The main motivation of Figure 2 is to compare how different objective functions handle mini-batch samples. As we mentioned in **[W3] Augmentation methods are already proposed in previous works**, the figure shows that it is somewhat non-trivial to apply MSDA to the existing methods because a mixed sample affects across the entire relationships between images and captions.

Reviewer comment (Experiments):
The motivation of this paper is "PCME suffers from expensive computations due to Monte Carlo approximation and fast loss saturation". However, the comparison or analysis about computation costs is not presented in the experiments.

Author response:
**[Q2] Computation cost comparison**

Thanks for your comment. The computational cost of PCME depends on the number of MC samples $J$, because it needs to compute $O(J^2)$ pairwise distances between all samples. When we use the same setting of the paper ($J=8$), we observe that PCME++ 25 epoch training takes 106,311 secs (1 day and 5 hours), while PCME 25 epoch training takes 141,694 secs (1 day and 15 hours) on a single V100 GPU. Overall, PCME needs 33% more training time compared to PCME++. Note that if we increase the sampling size $J$, the gap becomes larger. Another issue of the PCME sampling is that we need more memory size when computing the Monte Carlo approximation for a larger sampling size. Overall, PCME needs more forward time than PCME++ (33% more), and more memory size than PCME++ (on average, 18% more, but it is not a rigorous comparison because PCME has higher peak memory usage).