KS -Lottery: Finding Certified Lottery Tickets for Multilingual Transfer in Large Language Models (2025)

Abstract: The lottery ticket hypothesis posits the existence of “winning tickets” within a randomly initialized neural network. Do winning tickets exist for LLMs in fine-tuning scenarios? How can we find such winning tickets? In this paper, we propose KS-Lottery, a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. Our key idea is to use Kolmogorov-Smirnov Test to analyze the distribution shift of parameters before and after fine-tuning. We further theoretically prove that KS-Lottery can find the certified winning tickets in the embedding layer, fine-tuning on the found parameters is guaranteed to perform as well as full fine-tuning. Comparing KS-Lottery with other tuning algorithms on translation tasks, the experimental results show that KS-Lottery finds a much smaller set of parameters for fine-tuning while achieving the comparable performance as full fine-tuning LLM. Surprisingly, we find that fine-tuning 18 tokens’ embedding of LLaMA suffices to reach the fine-tuning translation performance .

## introduction can we find an ultra-small subset of a well-trained large language model (llm; @xcite @xcite such that fine-tuning these few parameters suffices to achieve the same performance as full tuning? the lottery tickets hypothesis @xcite states that a small subnetwork (less than 10-20% of the whole model size), referred to as "winning tickets", in a large, randomly initialized neural network can achieve comparable performance to the original network with the same amount of training. yet, the existence of winning tickets is not investigated for fine-tuning scenarios. prior work @xcite presents evidence that there are a small number of additional parameters corresponding to an intrinsic dimension (li 1 https://github.com/cone-mt/ks-lottery.git et al., 2018) on which fine-tuning leads to good performance. however, it remains an unsolved challenge to uncover such a small subset of fine-tuning efficient parameters within the original model. in this paper, we show that there exist key parameters -winning tickets, for transferring llm to multiple new languages. as shown in figure 1 (ks-lottery), we found that as tuning as few as 18 (18/32000 = 0.0006) token embeddings of a well-trained llm could achieve test performance comparable to full tuning on machine translation tasks. based on the observation, we state the lottery ticket hypothesis for multilingual fine-tuning. generally, the fine-tuning process can be represented by a transition in m parameters from θ = [θ 0 , θ 1 , • • • , θ m ] to θ = [ θ 0 , θ 1 , • • • , θ m ], where θ and θ denote the sets of parameters that characterize the llm f (•, θ) before and after the fine-tuning process, respectively. typically, the entire set of parameters θ is adjusted during finetuning to enhance the model's ability to represent and learn the new tasks more effectively. the fine-tuning lottery ticket hypothesis. na pre-trained neural network contains a small subset of parameters ( in this work, we examine the fine-tuning lottery ticket hypothesis on llms in a multilingual transfer scenario, and explore the following inquiries: identifying winning tickets. we propose ks-lottery, a method to identify a winning ticket by merely fine-tuning the embedding layer of an llm. and then fine-tune these identified tickets, keeping the remaining parameters frozen. the whole ks-lottery consists of three steps: ## observation in this section, we first treat each token embedding values as a distribution, then use kolmogorov-smirnov test to test the embedding distribution before and after tuning. finally, we discuss some phenomena observed before and after fine-tuning. ## certified winning tickets via ks-lottery in this section, we first introduce the kolmogorov-smirnov test @xcite and propose ks-lottery ( § 4.1). then we apply ks-lottery to find certifiable winning tickets in a mul- ## analysis ks-lottery . there are three different ways to use the winning tickets. partial tuning, partial transfer and frequency tuning, which only train the high-frequency tokens in the corpus. other baselines. original model directly using the llama-7b @xcite )/mistral-7b @xcite weight on test data, without any tuning. random tuning only the tokens randomly selected in the embedding layer are fine-tuned, while the remaining parameters are kept frozen. full tuning is an approach to transfer learning where the weights of a pretrained entire model are trained on new data. embed tuning merely fine-tuning the embedding layer of a model keeping the remaining parameters frozen. lora (hu et al., 2022) utilizes low-rank matrices for approximating parameter updates. qlora @xcite reduces memory usage enough to efficiently finetune. prefix-tuning @xcite introduce a lightweight prefix module into the input layer and each transformer layer, enabling efficient training over these modules. ## conclusion this work presents a novel method, ks-lottery, which applies the lottery ticket hypothesis to llms fine-tuning. by employing the kolmogorov-smirnov test, ks-lottery first analyzes the shift in the parameter distribution before and after finetuning, and then identifies a small but effective subset of llm parameters. theoretical evidence confirms that ks-lottery can pinpoint certified winning tickets within the embedding layer, thereby ensuring performance equivalent to full tuning. notably, ks-lottery surpasses other parameterefficient tuning algorithms by identifying fewer parameters for fine-tuning while maintaining similar performance levels. figure 10 : the translation performance of updated models with replacement method on flores-101 with different token numbers. we observe the minimal threshold is sufficient to maintain approximately 80% of the performance achieved through embedding tuning (marked in orange). meanwhile, for each plot, there exists a trend of initial increase followed by a decline, with the presence of an optimum point.