Improving Vision-Language Cross-Lingual Transfer with Scheduled Unfreezing (2024)

Abstract: Large-scale pretraining of vision-language (VL) models brought dramatic improvements across numerous tasks, from visual question-answering to cross-modal retrieval but these gains are mostly limited to English. Massively multilingual VL encoder models (mVLMs) hold promise for other languages: after fine-tuning on only English task data, they can perform the task in other languages in what is termed zero-shot cross-lingual transfer (ZS-XLT). Still, ZS-XLT sees a large performance gap to English, especially for low-resource languages. In this work, we reduce this gap with a fine-tuning strategy known as Scheduled Unfreezing (SUF): instead of updating all parameters from the start, we begin with the top layer(s) of the vision-language encoder and gradually unfreeze (i.e., update) its layers top to bottom. SUF forces reliance on encoder’s representations from higher layers: the fact that in multilingual models these representations encode higher-level semantics rather than low-level language-specific idiosyncrasies, we hypothesize, should render SUF beneficial for ZS-XLT. Experiments with two mVLMs (UC2 & CCLM) on three downstream tasks (xGQA, XVNLI, xFlickrCo) show that SUF brings consistent gains in ZS-XLT, especially for visual Q&A (xGQA) by up to 10 points.

## introduction recent vision-language (vl) models @xcite @xcite @xcite , trained on massive amounts of image-text data, led to dramatic improvements on virtually all vl tasks (e.g., image-text retrieval or visual q&a). this progress, however, benefits primarily english. large vision-language models (lvlms) @xcite @xcite )which align an image encoder to a large language model (llm)-excel in generalizing zero-shot to new tasks (without task-specific fine-tuning). most lvlms use english llms and are not highly multilingual; they fail to follow instructions in other languages or produce english output @xcite @xcite . multilingual lvlms are much less available foot_0 and generally underperform their english counterparts @xcite . the alternative is task-specific fine-tuning of smaller, but massively multilingually pretrained vl encoder models (mvlms) @xcite @xcite . here, however, task-specific training data exists predominantly in english which forces us to rely on zero-shot crosslingual transfer (zs-xlt) @xcite : due to the massively multilingual pretraining, the encoders fine-tuned on english task data can be used for inference in other languages. still, zs-xlt results in substantial performance drops in other languages compared to english, especially for less represented target languages in m(v)lm's pretraining. while fewshot training for specific target languages can reduce this performance gap @xcite , annotating sufficient data (for training and model validation) is expensive and does not scale to hundreds of languages. in this work, we improve zs-xlt with mvlms using a training method known as scheduled unfreezing (suf) @xcite . suf, which we apply in task-specific fine-tuning of an mvlm on english data, gradually increases the set of encoder's (i.e., transformer's) parameters that are being fine-tuned (i.e., updated), starting from the last layer(s) and gradually adding lower layers of the transformer stack as the training progresses. multilingual language-only encoders have been shown to encode language-agnostic high-level semantic knowledge in higher layers and language-specific idiosyn-crasies in lower layers @xcite . if the same holds for mvlms, then suf-by enforcing stronger reliance on representations from higher layers of an mvlm-should facilitate zs-xlt for vl tasks. put differently, with suf fine-tuning on english-only data, idiosyncratic english-specific knowledge from lower layers of the encoder is less available, forcing the model to rely on more language-agnostic knowledge from higher layers of the encoder. we evaluate the effects of suf fine-tuning on zs-xlt for two multilingual vision-language encoders: uc2 @xcite and cclm @xcite ; and on three distinct downstream tasks: visual qa (xgqa @xcite ), imagetext retrieval (xflickrco @xcite ), and visual entailment (xvnli @xcite ). we find that suf consistently improves performance compared to standard fine-tuning: by up to 3 points in retrieval and entailment and by a massive 10 points for visual qa. our further fine-grained analysis of model behavior on xgqa reveals that: (1) in standard finetuning the performance for most target languages stagnates or degrades over the course of (english) training, while the english performance steadily improves. (2) in suf fine-tuning, in contrast, trajectory of target language performance longer mirrors that of english performance, suggesting that the model relies on more language-agnostic representations; this results in massive improvements especially for some languages distant from english, such as korean and bengali. using parallel data, we show that suf fine-tuning indeed leads to crosslingually more aligned representations of the sequence start token ([cls]), which is input to the classifier. finally, we compare suf against two other strategies that similarly reduce reliance on lower layers of the encoder: (1) layer-wise learning rate decay and (2) fixed training of only the top layers. while both these also yield some performance gains, they underperform suf. suf-based fine-tuning not only improves zs-xlt of mvlms but is also computationally more efficient than standard fine-tuning: we thus hope that our work motivates broader investigation of suf strategies in the context of multilingual vl models. for visual question answering in particular, there has been work dedicated to reducing the crosslingual performance gap. @xcite assessed that a high ambiguity in the label space makes learning more difficult, attempting to remedy for this with several strategies, including addition of a similarity-based loss to standard classification cross-entropy loss, codeswitching at the instance level and a sparse finetuning approach. @xcite reduce the zs-xlt performance gap by replacing the standard single-layer classifier with a deeper two-layer architecture. observing stark performance differences across different question types, they also introduced a special question-type token. finally, @xcite find that fine-tuning a multilingual lvlm that relies on mt0 @xcite as the llm backbone nearly closes the zs-xlt gap. training and fine-tuning billion-parameter lvlms is, however, much more computationally expensive; crucially, the same is true for inference, which hinders model application for most users. moreover, @xcite show that the cross-lingual performance gap is highly dependent on the backbone llm, observing larger zs-xlt gaps with bloomz @xcite . in this work, we focus on encoder mvlms, due to their smaller computational footprint and thus broader applicability. to the best of our knowledge, our suf is the first strategy shown to substantially reduce the zs-xlt gap for vl encoders. ## scheduled unfreezing the exact setup on which we focus in this work is zero-shot cross-lingual transfer (zs-xlt) for downstream vision-language tasks (e.g., visual qa) with massively multilingual vision-language encoder models (mvlms) as vehicles of the transfer. in this setup, we fine-tune the mvlm on task-specific data in english only and evaluate its performance on task-specific data in other languages. based on the observation (from multilingual language-only encoders) that multilingual encoders encode more language-agnostic higher-order semantics in their upper transformer layers and language-specific information in their lower layers @xcite , we propose fine-tuning based on top-to-bottom scheduled unfreezing (suf) as a method to facilitate cross-lingual transfer with mvlms. the motivation for suf in this context is as follows: by (initially) freezing lower transformer layers, the classification head is forced to solve the task by tuning language-agnostic knowledge from higher transformer layers of the mvlm first. contrary, in full fine-tuning, the classifier can additionally leverage language-specific knowledge from lower layerswhen fine-tuned on english tasks data only. this means that the classifier is more likely to overfit to english-specific features, harming the effectiveness of cross-lingual transfer to other languages. to test this hypothesis, we use a fixed-schedule unfreezing in this work, illustrated in figure 1 . the general idea is not to train the full model from the start, but freeze (i.e., not update) all but the top k layers at the beginning and then gradually unfreeze k layers top-to-bottom in every epoch. architecture-specific implementation. compared to unimodal language-only encoders @xcite , mvlms additionally contain components for encoding the visual modality (i.e., images). moreover, mvlms come with different architectures, differing primarily w.r.t. where cross-modal information aggregation occurs. as such, we introduce architecturespecific unfreezing schedules for the two mvlms with which we experiment in this work: uc2 @xcite and cclm @xcite . uc2 is an encoder transformer model, architecturally identical to the language-only xlm-r encoder @xcite . uc2 encodes an image offline, relying on an object detection model @xcite 2 ; the features for image regions given by this model are linearly projected and then concatenated with the text embeddings as input to the model. the image region vectors are treated by the transformer like any other text token. as a result, we can use general suf without any adjust-ments: uc2, using a base-size xlm-r architecture, has 12 transformer layers. in the first epoch, the task-specific classification head, the embedding layer foot_3 , and the top @xmath0. after every training epoch, we unfreeze 3 additional layers, top to bottom. cclm, also a transformer-based encoder, comprises n layers for processing only the text input, followed by m more cross-modal layers, which additionally have a cross-attention component. through this cross-attention, the model attends to the image features extracted by a separate vision transformer (vit) @xcite . for cclm base , which we use in our experiments, there are @xmath1, followed by @xmath2. we keep the vit fully unfrozen during training. the motivation for this is twofold: (i) the resolution of images in fine-tuning is larger (384x384) than in its pretraining (224x224), requiring vit to adapt; and (ii) we employ suf to reduce the impact of language-specific (i.e., english) overfitting in fine-tuning and image encoding with vit is inherently language-agnostic. we thus keep the vit, task-specific classification head, and embedding layer unfrozen throughout training. in the first epoch, we additionally start with the top @xmath3. ## evaluation we provide details of our experimental setup and then consider results over three downstream tasks with the two architectures (uc2 & cclm). ## further analysis we further analyze suf fine-tuning through the lens of cross-language similarity of [cls] tokens for parallel data. we then compare suf with conceptually similar alternatives: (i) layer-wise learning rate decay and (ii) updating only the top layers transformer layers throughout the whole training. finally, we report the results of few-shot crosslingual transfer (fs-xlt). ## conclusion cross-lingual zero-shot allows us to train massively multilingual vision-language models on english task-specific data and then use them for other languages without additional target language training data. still, there is a large performance gap to english. in this work, we leverage scheduled unfreezing -a finetuning strategy where we initially keep all but the upper model layers frozen and gradually unfreeze the model top-down during training -as a method for reducing the transfer gap. experiments with two different models on three downstream vision-language tasks show that scheduled unfreezing can help improve non-english performance; results in visual question answering are especially promising with massive gains in accuracy. subsequent analysis suggests that scheduled unfreezing can help the zero-shot transfer by forcing the model to learn more language-agnostic features and overfit less on english-specific idiosyncrasies in the training data.