D e FT - X : Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer (2025)

Abstract: Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model’s parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X, a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.

## introduction pretrained language models (lms) are the de-facto choice for nlp, achieving state-of-the-art results across diverse benchmarks. however, effectively adapting these models to specific tasks remains a challenge owing to their large model sizes and the substantial training costs incurred during full finetuning. furthermore, full fine-tuning approaches are prone to issues such as catastrophic forgetting and negative interference when adapted to multiple tasks. to mitigate these challenges, parameterefficient fine-tuning (peft) techniques are a popular choice @xcite . these approaches include sparse fine-tuning (sft) that refers to identifying a sparse subnetwork of the full model to train, and adapter-based methods that insert additional trainable modules while keeping the original model parameters fixed @xcite . multilingual nlp introduces an additional layer of complexity, especially in the context of lowresource languages. a key objective of recent work in multilingual nlp has been to facilitate cross-lingual transfer by leveraging high-resource language data to improve performance on lowresource languages. zero-shot cross-lingual transfer refers to the more constrained setting of having access only to task-specific labeled data in a highresource (source) language and no labeled data in a low-resource (target) language. peft-based approaches such as mad-x @xcite and lt-sft @xcite are designed to support zero-shot cross-lingual transfer. in mad-x, a task adapter is learned using labeled data in a source language and a language adapter is learned using unlabeled data in a target language. to achieve zero-shot cross-lingual transfer, the task adapter in the source language is combined with the language adapter in the target language. lt-sft (lottery ticket sparse fine-tuning) removes the need for new trainable modules as in adapters by first identifying sparse subnetworks (a.k.a. "lottery tickets") within the model using magnitude pruning. fine-tuning these subnetworks while keeping the rest of the model frozen yields sparse vectors; these vectors can be estimated independently to learn task-specific and language-specific knowl-edge. these vectors are composed via simple addition to obtain the final model for zero-shot crosslingual transfer. in this work, we propose deft-x, a novel sparse composable approach for zero-shot crosslingual transfer. adopting the lt-sft template, deft-x aims to identify subnetworks for further finetuning using a low-rank approximation and improves the quality of sparse fine-tuned vectors by denoising higher-order components. specifically, we use singular value decomposition (svd) @xcite to decompose each weight matrix into lower and higherorder components (corresponding to high and low singular values, respectively). the higher-order components are denoised and added to the svd lower-order components, resulting in a signalamplified weight matrix (refer to §3.1). next, we apply magnitude pruning to identify a more effective sparse subnetwork by removing noise, followed by fine-tuning to obtain the final sparse finetuned vectors (detailed in §3.1). these vectors are finally composed via simple addition for effective cross-lingual transfer (as shown in §3.2). an illustration of deft-x is shown in figure 1 . the idea of identifying a subnetwork can be mapped to the notion of intrinsic dimensionality in fine-tuning. @xcite shows that common pretrained models have a surprisingly low intrinsic dimension i.e., fine-tuning can be just as effective as using a much smaller subspace of parameters compared to the full parameter space. lt-sft already leverages this idea by isolating sparse sub-networks. building on this insight, deft-x seeks to uncover a better subspace by explicitly denoising weight updates before pruning and sparse fine-tuning. we evaluate the zero-shot cross-lingual performance of deft-x on sentiment analysis (sa) and natural language inference (nli) tasks for extremely low-resource indonesian/indigenous languages in the nusax/americasnli benchmarks, respectively. our results demonstrate that deft-x performs at par or outperforms the state-of-the-art baselines. our findings highlight the importance of each step in the deft-x pipeline -denoising the model before selecting a sparse subnetwork, enforcing sparsity after denoising and fine-tuning the subnetwork, particularly in low-resource settings. we limit our experiments to encoder-only models. encoder-only models remain highly competitive for nlu in resource-constrained scenarios and are often more parameter-efficient than decoderonly models. for example, recent work @xcite shows that encoders can achieve significantly better nlu performance than decoder-only models on several scandinavian languages despite having orders of magnitude fewer parameters. thus, for our target setting of zero-shot transfer to under-resourced languages, we focus on encoder-only models. ## experimental setup we evaluate zero-shot cross-lingual transfer learning on two publicly available low-resource benchmarks licensed under cc-by-sa: sentiment analysis (nusax) and natural language inference (americasnli). ## results and discussion we report f1 scores for nusax using both methods of rank selection: 90% variance and uniform rank r, as shown in similarly, we report the accuracy for americas-nli using both methods of rank selection: 90% variance and uniform rank @xmath2, as shown in in summary, we observe that denoising higherorder components before selecting the sparse subnetwork (via magnitude pruning) improves the quality of sparse vectors for composition. both uniform rank selection and 90% variance-based rank selection perform comparably well. however, very low-resource languages may benefit from a higher rank to capture more meaningful knowledge. ## related work parameter-efficient fine-tuning. parameterefficient fine-tuning (peft) adapts large pretrained models to downstream tasks with minimal trainable parameters. diffpruning @xcite task arithmetic and multi-task transfer learning. @xcite introduces task vectors obtained by subtracting the weights of a pretrained model from those of a fine-tuned model. these task vectors can be manipulated through arithmetic operations to steer model behavior. approaches like @xcite and dare @xcite propose identifying better task vectors by dynamically dropping and learning parameters. various approaches were proposed to mitigate the destructive interference during task arithmetic in multi-task setting like resolving sign conflicts while merging @xcite and learning mutually sparse vectors @xcite . deft-x can also be used in multi-task setting to obtain better task vectors by denoising the redundant task information and mitigating destructive interference. cross-lingual transfer learning. crosslingual transfer learning improves task performance in low-resource languages by leveraging knowledge from high-resource languages. approaches like mad-x @xcite use modular language and task specific adapters for composable transfer. mad-g @xcite introduces a contextual parameter generator trained on typological features from uriel to build efficient adapters for low-resource languages. lt-sft @xcite proposes sparse task and language vectors that can be arithmetically composed for zero-shot transfer. subsequent works combine few-shot fine-tuning with lt-sft @xcite and explore scaling in task arithmetic @xcite . @xcite proposes a bilingual distillation approach to extract language-specific models from massively multilingual transformers for cross-lingual transfer. layer swapping for transferring linguistic knowledge to reasoning tasks has also been explored @xcite . our approach deft-x, builds on lt-sft by improving sparse vector quality via svd-based denoising. low-rank approximation using svd. singular value decomposition (svd) is widely used for low-rank approximation, aiding both efficiency and interpretability in neural networks. laser @xcite shows that pruning small singular components from transformer weight matrices can improve reasoning by denoising internal representations without retraining. lors-merging @xcite merges multilingual speech models using coarse-grained singular value pruning to retain essential structures and fine-grained magnitude pruning to remove redundancy. our approach deft-x, applies similar low-rank denoising to enhance the quality of sparse fine-tuned vectors. ## conclusion and future work we introduced deft-x, a composable, denoised, sparse fine-tuning approach for efficient zero-shot cross-lingual transfer. by leveraging svd to denoise model weights, deft-x identifies better subnetworks for sparse fine-tuning. we also explored different strategies for selecting the matrix rank during denoising. compared to the state-of-theart lt-sft approach, deft-x demonstrates improvements in sa and nli tasks for low-resource languages. in future work, we plan to explore the broader applicability of deft-x beyond crosslingual transfer, including its potential in multimodal learning and domain adaptation.