Paper: T-Measure: A Measure for Model Transferabilty (ICLR-2024)

Reviewer comment (Writing):
Can authors summarize and refine the contributions part? The "contribution" of the introduction seems to only summarize each section, not really "contribution".

Author response:
We appreciate your constructive feedback. Here is our response to the reviewers questions:  
1. Our contributions are as follows: We introduce the concept of label-agnostic model selection, presenting a method to estimate the transfer measure and employing it for label-agnostic model selection. The introduced method for estimating transferability is unsupervised and relies on representation learning, making it generalizable and applicable across various domains. Notably, our work is the first to estimate zero-shot transferability, and we adapt measures previously used for transferability estimation in a few-shot setting.  
We compare the adapted measure with our proposed measure (T-measure) and demonstrate the superiority of our introduced transferability estimation through experiments conducted on 11 datasets across 4 tasks.

Reviewer comment (Novelty):
It is not clear if this paper targets at domain adaptation or model fine-tuning. The paper mentions that it focuses on the case where source and target shares the same task, which is domain adaptation actually. If the paper further focuses on the case where Dtrg is not accessible during model selection, it is unsupervised domain adaptation I think. And if the paper deals with selecting source dataset, it is multi-source unsupervised domain adaptation. The literature papers it mentions like Bao et al., 2019; Nguyen et al., 2020; Tran et al., 2019 are all for fine-tuning. They have access to target labels, and they tried different ways to use the target labels. The authors should have a thorough literature review to put this paper into an appropriate position in the literature.

Author response:
We appreciate your constructive feedback. Here is our response to the reviewers questions:  
2. The broader problem of model selection can be distilled into either domain adaptation or model fine-tuning based on variables. Consequently, we have incorporated these perspectives into our related works. As correctly pointed out by the reviewer, in cases where the sole variant is the dataset (source and target datasets), our approach aligns with domain adaptation.  
3. We appreciate the suggested references and related work topics. In response, we have included domain adaptation in the revised version of the paper to provide a comprehensive perspective in addressing your concerns.

Reviewer comment (Theory):
This paper makes a key assumption that the source datasets is decomposable, which might not always be the case. Therefore, the generality of the approach to other datasets is not guaranteed, limiting its impact. Also, they assume that the source dataset is always available during model selection, which is generally not true if we consider the recent trend of foundation models where the source datasets are not available but only source models are released. Lastly, although a minor assumption, it might not always hold that the source and target tasks are same. For example, MLM (masked-langauge-modeling) has been shown to be universally applicable to many downstream tasks for few shot transfer.

Author response:
Our proposed method for learning a representation space for the dataset relies on information about the dependency among datapoints in the dataset. While the dataset doesn't necessarily have to be decomposable, our model remains applicable as long as we can identify dependencies among data points.
Acknowledging that source data might not always be available during the model selection phase, we aim to enhance the broad applicability of the T-measure. We plan to learn a model representation, estimating the transfer measure using the model instead of relying on the availability of the source dataset or other confounding transfer factors. In response to the observation that tasks in the source and target may not always be similar, our current focus in this paper is on models and zero-shot transfer, where the transfer function is an identity function. This function requires the task space to be similar for the source and target. We intend to extend our work to cover other functions in future research.

Reviewer comment (Experiments):
The paper, at its core, essentially evaluates which of the source dataset is the closest to a particular target dataset. In this sense, the authors must also include comparisons with several works in domain adaptation (DA) and robust optimization literature. For example, several measures like A-distance and H-divergence [1] also need to be included. In general, several works in UDA literature need to be cited and discussed.

Author response:
We appreciate the references you provided, and we have duly included them in the related work section.

Reviewer comment (Experiments):
It is really not clearly explained why triplet loss is chosen to learn a suitable representation space. Does the representation learnt using source labels work equally well? Can we also use contrastive loss with multiple negatives? This aspect should be studied in much greater detail.

Author response:
The choice of triplet loss is grounded in its ability to serve as a proxy for modeling the dependency of datapoints in the dataset without supervision. In triplet loss, anchor and positive points are considered dependent, while anchor and negatives are independent. Unlike learning representations using labels, which limits the representation to specific tasks, our approach learns a dataset representation that can be applied to multiple tasks (e.g., using daily dialog for emotion recognition and response selection). In our experiments, we employed multiple negative datapoints for contrastive loss (19 points for each anchor and positive).

Reviewer comment (Presentation):
I am curious to know why the authors chose to provide examples related to vision datasets and vision tasks (in Fig 2), while all the evaluation is done using NLP datastes. Even more so, Fig 1-4 are all really not explaining anything related to the problem or method, and could be improved to illustrate your idea better.

Author response:
It's important to note that the introduced T-measure is not confined to NLP; it is applicable to vision datasets as well, therefore, the examples in Figure 2 are for the vision domain. Although we plan to include vision datasets in our future work, the limited number of publicly accessible vision datasets has been a constraint. Figure 1 provides an illustration of the general model selection problem, and we have updated Figures 3 and 4 to better convey the ideas behind our method.

Reviewer comment (Writing):
Sec 3.2, Step1: Did you mean argmin in the equation?

Author response:
We addressed the minor issues in the new revision of the draft.

Reviewer comment (Writing):
For completeness, PVI should be explained using an equation or such, since not all readers might be familiar with the term (I am not!).

Author response:
We addressed the minor issues in the new revision of the draft.

Reviewer comment (Writing):
Equations are not numbered in the paper, which is lowering the readability of the paper. For example, equations in Section 2.1 and 2.2 are outputting \phi^* model, on the first read it's not clear what is the difference between the two equations. Similarly, in Section 3.2, PVI computation of data samples is not provided, hence the motivation for Step 2 is not clear from the draft. In the results section, Table 4 and 6 are referenced for results, but these tables are present in the Appendix, not the main section of the paper. Not sure if it violates the page length limit in the conference.

Author response:
We greatly appreciate your constructive comments. In the revised version, we will organize the equations and assign numbers accordingly.

Regarding the difference between equations 2.1 and 2.2: Equation 2.1 represents the general equation for model selection. However, in equation 2.2, the model is selected among transferred models under a specific transfer method (\alpha). The assumption is that the set of models {\phi} is transferred to a set of {\phi’} models under the transfer method \alpha. The objective of model selection becomes choosing the best transferred model {\phi’}. To enhance clarity, we have added the definition of PVI.

Reviewer comment (Evaluation):
In Table 3, it is not clear what average Kendall-tau distance signifies. There is no point averaging across datasets. Can authors specify if the model is outperforming other methods on individual datasets. Also, references for competing methods are not provided in the table.

Author response:
In reference to the Kendall-Tau ranking measure in Table 3, it illustrates how the ranking of models based on the selected transfer measure aligns with the actual performance of the models on the target. A higher Kendall-Tau value indicates a more accurate ranking of models, reflecting a ranking more similar to the models' actual performance on the target. We have updated the table caption to provide clarity on this aspect.

Furthermore, we have revised the table to emphasize the best transfer measure for each target. Notably, for six targets, the T-measure yields a higher ranking compared to other metrics (naive: 4, V-usability: 5, PARC: 4). In the initial paragraph of Section 4.3, where baselines are described, we have included the relevant references. It is crucial to note that, as the first zero-shot data-centric transfer measure, the baselines were modified to align with the T-Measure.

Reviewer comment (Evaluation):
The proposed metric requires access to the source data. However, the source data may not always be available during fine-tuning.

Author response:
We appreciate your feedback. We agree that the source data might not always be available during the model selection phase. To overcome this issue and make the T-measure broadly applicable, we plan to learn a model representation, i.e., instead of estimating the transfer measure using the source dataset, estimate it using the model, which does not require the availability of the source dataset and other confounders of transfer.

Reviewer comment (Experiments):
The number of pre-trained models is quite limited. According to Table 3, the metric is only evaluated on 3 or 6 models. It remains unclear whether the metric can be extended to a large number of pre-trained models.

Author response:
The reason behind the limited number of source models is that we fixed all of the parameters and factors(model architecture, transfer method, ...) in model selection except the source dataset so that the results can be fairly compared together. In this paper, our focus is to 
 solely estimate the effect of the source dataset on the transfer. And the number of models in such a setting becomes limited to the number of available datasets for that task under the same conditions. For example, we found only 3 annotated datasets for utterance-level conversation emotion recognition with 6 basic emotions.