What explains the success of cross-modal fine-tuning with ORCA ? (2024)

Abstract: ORCA (Shen et al., 2023) is a recent technique for cross-modal fine-tuning, i.e., applying pre-trained transformer models to modalities beyond their training data. The technique consists primarily of training an embedder and fine-tuning the embedder and model. Despite its high performance on a variety of downstream tasks, we do not understand precisely how each of these components contribute to ORCA’s success. Therefore, we run a series of ablations and find that embedder training does not help 2D tasks at all, contrary to what the original paper posits. In 1D tasks, some amount of embedder training is necessary but more is not better. In 4 out of 6 datasets we experiment with, it is model fine-tuning that makes the biggest difference. Through our ablations and baselines, we contribute a better understanding of the individual components of ORCA.

## introduction modern ai is based on a pipeline of pre-training general-purpose models on vast amounts of data and then adapting them to specific tasks. examples across natural language processing (nlp) and computer vision (cv) typically focus on withinmodality adaptation across, e.g., tasks or domains, but there is also a recent line of work that looks at leveraging pre-trained models across modalities, e.g., frozen pretrained transformers (fpt) @xcite , orca @xcite , om-nipred @xcite , unified pde solver (ups) @xcite , inter alia. orca is a recent example of a method for crossmodal fine-tuning @xcite . it consists of a three-phase pipeline, shown in figure 1 . first, a pre-trained transformer model is chosen, and a custom embedder and predictor are created to support new tasks with any input and output dimensions. second, a within-modality proxy dataset is chosen. the embedder is trained to minimize the distance between the target dataset and this proxy dataset, in * equal contribution. order to map the target dataset into the embedding space of the model. finally, all three components are fine-tuned on data from the target task. according to @xcite , embedder training is the reason for orca's success. we expand on their ablations to better understand the contributions of orca's individual components, focusing on ablating the second and third stages of the pipeline. our specific research questions are: ## experimental setup unless otherwise specified, we follow the orca paper in using roberta-base @xcite and swin-base @xcite as the pre-trained transformers, a convolutional architecture for the embedder, and a linear transformation for the predictor (see appendix c for details). we also use optimal transport dataset distance (otdd; alvarez-melis and fusi, 2020) as the loss function during embedder training. all our experiments use their publicly available code. 1 for training, we use the same hyperparameters as they do, except for the batch size when training on satellite (64) and ecg (32) data. we evaluate on six target datasets that appear in the original paper, chosen to represent all pairs of dimensions and types, and we experiment with various proxy datasets. dataset details are shown in appendix b. target datasets. we select three 2d datasets (ni-napro, cifar-100, and darcy flow) and three 1d datasets (satellite, deepsea, and ecg) from the nas-bench-360 benchmark @xcite . 2d and 1d refer to the input being either a matrix (2 dimensions) or a sequence (1 dimension). proxy datasets. the original paper uses cifar-10 (krizhevsky, 2009) as the proxy dataset for all 2d tasks, and conll 2003 @xcite for all 1d tasks. we experiment with additional proxy datasets to analyze their in-1 https://github.com/sjunhongshen/orca/ fluence on overall performance. for the 2d tasks, we compare to two other image datasets that maintain the same number of classes: mnist @xcite , a different image dataset, and fakedata foot_0 , a dataset of randomly classified white noise images @xcite . for the 1d tasks, we compare to a custom-created fake dataset classifying randomly generated language feature vectors into the same number of classes as conll. in this section, we experiment with the choice of proxy dataset for the tasks. as a baseline, we compare to just fine-tuning the embedder, model and predictor, without training the embedder first. as figure 2 shows, all fine-tuning curves for the 2d datasets (first row) overlap, indicating that the choice of proxy dataset is not important. even fake data as a proxy dataset results in the same performance. similarly, for the 1d tasks (second row), there is no real difference between using conll and fake embeddings. together, this shows that the choice of proxy dataset for embedder training does not matter for orca to work. comparing to a naive fine-tuning baseline allows us to evaluate the claim that "orca consistently outperforms naive fine-tuning" @xcite . we find that embedder training does play a role in the 1d tasks, but does not matter for 2d tasks, even in the early stages of fine-tuning. ## (more) embedder training is not the secret to orca's success the previous results motivate us to more closely examine the role of embedder training in orca. in this stage, the otdd metric is used to quantify the distance between the proxy and target embeddings. the authors minimize otdd, claiming that "as the dataset distance decreases, the fine-tuning accuracy increases" @xcite . however, when we examine the relationship between otdd and downstream task performance, we find that embedder training is unnecessary in two out of six tasks (figures 3a and 3c ). for the remaining four tasks, training the embedder more can even lead to worse task performance. as this section and the previous one show that embedder training does not affect final performance on the 2d tasks, we focus on the 1d tasks for our remaining experiments. ## which components of orca are really necessary? to better understand how the fine-tuning phase affects the multiple components of orca, we experiment with freezing different parts of the pipeline: the embedder, the pre-trained model, or both. we compare our results with the original setup. row 1 of figure 4 shows the results of freezing both the embedder and the pre-trained model, and only fine-tuning the predictor. across all datasets, the frozen versions perform much worse than the original setup, regardless of embedder training. this indicates that these datasets are not simple enough to be solved by training a simple predictor. in row 2, we freeze only the pre-trained model, but fine-tune the embedder and the predictor. these frozen versions also perform much worse than the original setup, indicating that fine-tuning the pretrained model is a critical component of orca, regardless of dataset and embedder training. finally, in row 3, we only freeze the embedder, allowing the fine-tuning stage to affect both the model and the predictor. as we already saw in figure 2 , training the embedder is important across all three datasets. however, once this training is done, even if it is frozen, adapting the pre-trained model is sufficient for good task performance. this shows that while training the embedder is important for orca's success on these datasets, it need not be fine-tuned beyond that. until the 30b data scale, however, deepsea performance remains within the variance of simply fine-tuning a randomly-initialized model, whereas ecg does benefit from even a small amount of pre-training. this shows that even for non-trivial tasks, the amount of pre-training has a noticeable effect only at certain scales. ## conclusion we perform a series of ablations to investigate how the different components of orca, a recentlyproposed method for cross-modal fine-tuning, affect its performance. contrary to the original results, we find that embedder training does not help 2d tasks at all, compared to just fine-tuning without training the embeddder. in 1d tasks, some amount of embedder training is necessary, but unlike the claim in the original paper, more embedder training can even hurt performance on the target task. when we freeze various components of the orca pipeline, we find that fine-tuning the model is crucial for good task performance. finally, we find that for one of the 1d tasks, using a pre-trained model is actually not necessary, indicating the importance of no pre-training baselines in evaluations of cross-modal transfer. language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45.