Divergence-Based Domain Transferability for Zero-Shot Classification (2023)

Abstract: Transferring learned patterns from pretrained neural language models has been shown to significantly improve effectiveness across a variety of language-based tasks, meanwhile further tuning on intermediate tasks has been demonstrated to provide additional performance benefits, provided the intermediate task is sufficiently related to the target task. However, how to identify related tasks is an open problem, and brute-force searching effective task combinations is prohibitively expensive. Hence, the question arises, are we able to improve the effectiveness and efficiency of tasks with no training examples through selective fine-tuning? In this paper, we explore statistical measures that approximate the divergence between domain representations as a means to estimate whether tuning using one task pair will exhibit performance benefits over tuning another. This estimation can then be used to reduce the number of task pairs that need to be tested by eliminating pairs that are unlikely to provide benefits. Through experimentation over 58 tasks and over 6,600 task pair combinations, we demonstrate that statistical measures can distinguish effective task pairs, and the resulting estimates can reduce end-to-end runtime by up to 40%.

## introduction as the accuracy of neural models continues to increase, so does the computational cost of training and storing them. one approach of mitigating such cost is through using pretrained models to enhance performance on a downstream task, a paradigm commonly referred to as transfer learning. however, when and why transfer learning works is not concretely understood. traditionally, selecting the best settings, i.e. tasks and hyperparameters, for transfer often involves an extensive trial-and-error process over many combinations and can quickly make the prospect of applying transfer learning undesirable. as such, it would be valuable to estimate whether a task pair combination will be effective pre-training, i.e. estimate the transferability of a source task to a target task. the most optimal transferability metric would be resource-efficient, such that it is capable of accurately predicting the final performance of the model whilst minimising the amount of processing required to compute it. to this end, several works @xcite @xcite have focused on estimating transferability prior to finetuning, using statistical measures of divergence between the underlying feature spaces of model pairs. domain divergence measures are used to produce a notion of distance between pairs of domains by comparing their representations and have seen significant usage in works which investigate the correlation between their estimations and performance change @xcite . subsequent transfer learning works have also demonstrated that competitive model performance can be achieved on some target tasks even if no training samples for that task are available, an approach known as zero-data/shot learning @xcite . in this work, we investigate the effectiveness of domain divergence measures in estimating the performance of zeroshot classification models, wherein models further tuned on one source task are used to directly predict on the test set of a target task without any target training samples. specifically, we leverage the information captured by these measures as features to an auxiliary learner, whose outputs are used to rank the most effective source model for transfer to a given target task. through the analysis of 58 sentiment classification domains, we: (1) perform a correlation analysis between each independent measure and each source-target, macro-averaged f 1 -score performance output; (2) and, for each target task, we train a series of auxiliary regression models to predict their projected performance; representations: to compute the above metrics, we use two different representations from prior work by @xcite , specifically 1) discrete probabilities of the most common terms across domains, using a fixed-size vocabulary v , where |v | = 10, 000; and 2) a summation over probability-weighted term embeddings in each document, averaged to produce a single vector: generally, since we are using these representations in a zero-shot setting, we compute divergences between the source-task training set (d s ) and the target-task test set (d t ). entropy and moments-based measures are not used to estimate divergence between domains but used only to compute within-domain characteristics, i.e. on individual term distributions. ## conclusions and future work in this paper, we have shown that domain divergence measures and other statistical quantities are predictive of zero-shot transferability between tasks, and that this can be used to markedly reduce time when developing effective zero-shot models. indeed, by predicting which source-target task pairs were likely transferable pre-tuning, we were able to reduce the end-to-end time taken to find the best source-target task pairs (trained on 1,000 sourcetask samples) by 40%. on the other hand, while we have demonstrated the value of using these metrics in performance estimation, there are a number of further directions worth investigating, namely: