Paper: Multitask Contrastive Learning (ICLR-2024)

Reviewer comment (Novelty):
The authors provide a derivation of the MtCon loss function to arrive at Eqn (8), but it ends up simply learning a scalar weight for different contrastive learning tasks and regularizing the scalar weights. Seeing that it is so straightforward, I think the paper could benefit from 1. more discussion on if this loss function is better suited for *specifically* for contrastive learning (unless I'm missing something, most of the analysis and theory could apply to any multi-task setting, even if it isn't contrastive), and 2. more task-weighting baselines like the ones mentioned in prior work. The one multi-task weighting baseline XEnt-MT already seems pretty close to MtCon in terms of performance, so I imagine the other multi-task weighting methods work well too.

Author response:
The idea behind MTCon weighting is to downweight tasks with more task uncertainty. As the reviewer correctly points out, this idea has been previously explored in the multi-task literature, but not in the contrastive setting.  Existing techniques do not extend directly to the contrastive setting because unlike existing methods, contrastive loss doesn't imply a likelihood function over the observed data. Thus, we develop the pseudo-likelihood to demonstrate that the temperature corresponds to uncertainty weighting. We do find improvements to using contrastive learning in conjunction with the multi-task learning (similar to previous self-supervised and supervised contrastive learning work such as Khosla et al 2020). Across out-of-domain tasks, MT-Con outperforms uncertainty weighted multi-task cross-entropy by an average of over 1.5% across out-of-domain tasks (Table 1).

Reviewer comment (Experiments):
Even at high noise levels, the weight of the noisy tasks doesn't fall that low compared to other tasks (in Figure 3). I think a useful ablation might be to manually set the weight of the noise task to a very low or very high number and show how performance changes.

Author response:
This is a great question.. As the uncertainty increases, the learned weight should converge to zero, but does not because of estimation error. We expect such estimation error to go to zero as the sample size goes to infinity. We like the idea of studying the behavior in a regime where we have no estimation error. When we manually set the task weight to 0 for the corrupted tasks, the generalization accuracies for the Zappos50k and MEDIC datasets are 40.5% and 80.8%, respectively. We will include these results in the Appendix.

Reviewer comment (Experiments):
How is the MtCon weighting scheme specifically suited for contrastive learning? If it isn't specific to contrastive learning, should more multi-task weighting baselines be compared to?

Author response:
The idea behind MTCon weighting is to downweight tasks with more task uncertainty. As the reviewer correctly points out, this idea has been previously explored in the multi-task literature, but not in the contrastive setting.  Existing techniques do not extend directly to the contrastive setting because unlike existing methods, contrastive loss doesn't imply a likelihood function over the observed data. Thus, we develop the pseudo-likelihood to demonstrate that the temperature corresponds to uncertainty weighting. We do find improvements to using contrastive learning in conjunction with the multi-task learning (similar to previous self-supervised and supervised contrastive learning work such as Khosla et al 2020). Across out-of-domain tasks, MT-Con outperforms uncertainty weighted multi-task cross-entropy by an average of over 1.5% across out-of-domain tasks (Table 1).

Reviewer comment (Novelty):
Is this method just a simple multi-task weighting scheme applied to contrastive learning tasks?

Author response:
The idea behind MTCon weighting is to downweight tasks with more task uncertainty. As the reviewer correctly points out, this idea has been previously explored in the multi-task literature, but not in the contrastive setting.  Existing techniques do not extend directly to the contrastive setting because unlike existing methods, contrastive loss doesn't imply a likelihood function over the observed data. Thus, we develop the pseudo-likelihood to demonstrate that the temperature corresponds to uncertainty weighting. We do find improvements to using contrastive learning in conjunction with the multi-task learning (similar to previous self-supervised and supervised contrastive learning work such as Khosla et al 2020). Across out-of-domain tasks, MT-Con outperforms uncertainty weighted multi-task cross-entropy by an average of over 1.5% across out-of-domain tasks (Table 1).

Reviewer comment (Novelty):
How does the MtCon weighting scheme connect to prior work on multi-task weighting?

Author response:
The idea behind MTCon weighting is to downweight tasks with more task uncertainty. As the reviewer correctly points out, this idea has been previously explored in the multi-task literature, but not in the contrastive setting.  Existing techniques do not extend directly to the contrastive setting because unlike existing methods, contrastive loss doesn't imply a likelihood function over the observed data. Thus, we develop the pseudo-likelihood to demonstrate that the temperature corresponds to uncertainty weighting. We do find improvements to using contrastive learning in conjunction with the multi-task learning (similar to previous self-supervised and supervised contrastive learning work such as Khosla et al 2020). Across out-of-domain tasks, MT-Con outperforms uncertainty weighted multi-task cross-entropy by an average of over 1.5% across out-of-domain tasks (Table 1).

Reviewer comment (Theory):
The theoretical analysis makes some simplifying assumptions (e.g. abundance of source tasks) that may not perfectly hold in practice.

Author response:
The reviewer is correct in that our theoretical analysis assumes that the source tasks contain sufficient information to cover target tasks. We stress, however, that this assumption is in line with the majority of theoretical work in the multitask and supervised contrastive learning fields, including [1] and [2]. The experimental results are consistent with what the theory suggests.

Reviewer comment (Experiments):
All datasets used are for computer vision. Testing MTCon on a wider variety of modalities (text, audio, etc) could better demonstrate generalization.

Author response:
It would indeed be an interesting future direction to apply our method to other modalities for which multiple similarity measures are common, but this work focuses on showing that the methods work for images on three multi-task vision datasets from very different domains. These datasets are a superset of those used in earlier contrastive multi-similarity representation work, including [3] and [4].

Reviewer comment (Experiments):
The experimental evaluation is quite thorough, but lacks ablation studies to isolate the impact of different components of MTCon (e.g. projection heads vs weighting scheme). Ablation studies would provide more insight.

Author response:
We provide an ablation study in Figure 3 that demonstrates that weighted MT-Con generalizes better than the unweighted version to out-of-domain tasks. We cannot ablate the multiple projection heads since MT-Con is dependent on it to capture disjoint similarities.  Can the reviewer clarify if there are other specific ablations they believe would strengthen the paper?

Reviewer comment (Experiments):
The comparison to prior work is limited to a few baselines. Comparing against a broader range of multi-task representation learning methods could better situate MTCon.

Author response:
We provide comparisons to self-supervised, single-task supervised, and the most recent multi-task multi-similarity supervised contrastive learning methods across all three evaluation datasets. We chose these baselines because they are commonly used in the literature [5,6,7].  We would be happy to include other baselines that the reviewer would like to see.

Reviewer comment (Experiments):
The hyperparameter analysis is quite brief. A more extensive sweep over training hyperparameters and architectural choices could be illuminating.

Author response:
We provide hyperparameter analyses for temperature and number of epochs for both the Zappos50k and MEDIC datasets in Figures 6,7 in the Appendix. We can change the main text to summarize these.

Reviewer comment (Experiments):
Have you considered any other proxies for estimating task uncertainty besides the constructed pseudo-likelihood? How does using other uncertainty estimates impact performance?

Author response:
We thank the reviewer for the thoughtful question. We have not considered other metrics of uncertainty since we found that the pseudo-likelihood worked well for us in theory and in practice. We believe that this would be an interesting direction for future work.