S ^4 -Tuning: A Simple Cross-lingual Sub-network Tuning Method (2022)

Abstract: The emergence of multilingual pre-trained language models makes it possible to adapt to target languages with only few labeled examples. However, vanilla fine-tuning tends to achieve degenerated and unstable results, owing to the Language Interference among different languages, and Parameter Overload under the few-sample transfer learning scenarios. To address two problems elegantly, we propose S ^4 -Tuning, a Simple Cross-lingual Sub-network Tuning method. S ^4 -Tuning first detects the most essential sub-network for each target language, and only updates it during fine-tuning.In this way, the language sub-networks lower the scale of trainable parameters, and hence better suit the low-resource scenarios.Meanwhile, the commonality and characteristics across languages are modeled by the overlapping and non-overlapping parts to ease the interference among languages.Simple but effective, S ^4 -Tuning gains consistent improvements over vanilla fine-tuning on three multi-lingual tasks involving 37 different languages in total (XNLI, PAWS-X, and Tatoeba).

## introduction recently, a variety of multilingual pre-trained language models (plms) have been proposed, including mbert @xcite and xlm-r @xcite . based on these plms, it is possible to adapt the model to specific target languages, with only a handful of labeled examples in the downstream tasks, which is called few-shot crosslingual transfer learning @xcite @xcite . however, traditional fine-tuning tends to obtain degenerated and unstable results, due to the following two challenges. (1) parameter overload: given only few labeled data for a target language, it is challenging to update all model parameters, * joint work between alibaba and peking university. † corresponding authors. and such a mismatch between the scale of data and trainable parameters can cause overfitting @xcite . (2) language interference: sharing commonality though, different languages also possess their own characteristics. hence, the adaption towards a specific target language can interfere with that of other languages @xcite , which also damages the transfer performance. therefore, it is natural to ask the question, how to address the parameter overload and language interference problem elegantly? in this paper, we propose a simple cross-lingual sub-network tuning method, s 4 -tuning, which tries to deal with these two problems jointly. as shown in figure 1 , s 4 -tuning detects the most fundamental language sub-networks (with a simple and intuitive criterion in sec. 3.2), and only updates the specific sub-network corresponding to the input language during training. for one thing, we update the language sub-network on a matching scale, which better suits the low-resource scenarios and addresses the parameter overload problem. for another, the commonality across languages is modeled by the overlap among different language sub-networks, while the characteristics are also allowed by the non-overlapping parts. with such a better trade-off, the language interference problem is alleviated. simple to implement, s 4 -tuning also reveals evident effectiveness in the downstream tasks in our experiments. compared with vanilla fine-tuning, s 4 -tuning consistently offer improvements across different multi-lingual downstream tasks. for example, it improves by 0.9 and 5.6 average points on xnli and tatoeba tasks, respectively. ## related work towards better few-shot cross-lingual transfer, @xcite freeze the embedding and encoder layers of the plm during fine-tuning, which is not effective and flexible enough. @xcite adopt the traditional meta-learning method maml @xcite , but it is not practical enough, since it requires extra abundant labeled data for meta-training. differently, we try a more elegant and effective way to handle the parameter overload and language interference problem through language sub-networks. some works also find a sub-network for each language pair in machine translation @xcite , or each task in multi-task learning @xcite . however, their forward and backward are both based on sub-networks, which is more like pruning. instead, we update parameters within the sub-network during the backward process, but still forward on the whole network to fully utilize the knowledge stored in the entire model. our work most closely resembles the work of @xcite . however, s 4 -tuning deals with multiple sub-networks simultaneously rather than a single sub-network in more challenging few-shot multi-lingual scenarios, and adopts different criteria for language sub-network detection. we empirically show the superiority of s 4 -tuning in figure 3 in section 4.5. ## s 4 -tuning: simple cross-lingual sub-network tuning we formally present the problem formulation (sec. 3.1). then we introduce our proposed method, s 4 -tuning, which firstly detects the most important sub-network for each target language (sec. 3.2), and then only updates the corresponding subnetwork during the backward process (sec. 3.3). ## conclusion towards better few-shot cross-lingual transfer learning, we propose s 4 -tuning. s 4 -tuning detects the most essential sub-network for each target language, and only updates these parameters during the backward process, while still utilizing the full model for the forward process. in this way, we reduce the scale of trainable parameters that better suits low-resource scenarios to address overfitting, and better deal with the interference across languages. our experiments show that s 4 -tuning consistently outperforms other fine-tuning methods in different downstream tasks.