Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing expert modules trained separately on language or task data. Our method composes \textit{language} and \textit{task} PEFT adapters via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT adapters trained on languages related to the target. Empirical results on summarization demonstrate that our method is a strategy that obtains consistent gains using minimal training of PEFT parameters.

## introduction large language models (llm) have achieved impressive performance on various real world applications in many different human languages @xcite @xcite @xcite . summarization @xcite is a particularly interesting and useful task because it allows users to quickly aggregate and access relevant information from large amounts of textual data. developing a competitive text summarization system for a language typically involves fine-tuning a pretrained model on labeled summarization data in the given language. standard supervised fine-tuning of llms can be very expensive due to the large model size. parameter-efficient tuning (peft) is an effective alternative that achieves competitive performance while incurring much less computational and memory cost @xcite @xcite . despite the effectiveness of peft @xcite , it also has several limitations if we want to develop competitive multilingual summarization systems. first, current peft methods generally require access to labeled task data in a given language. while there are several existing datasets in english to train competitive summarization systems @xcite @xcite , many languages in the world with millions of speakers do not have such resources @xcite @xcite . second, standard peft methods optimize a separate set of parameters for each language, resulting in thousands of fine-tuned checkpoints, which need to be stored and deployed individually @xcite . finally, as the standard peft methods are fine-tuned in isolation, they cannot leverage information from related tasks. in this paper, we want to improve zero-shot multilingual summarization with peft to better support languages that might lack labeled summarization data. to this end, we propose a simple yet effective method that composes language and task information stored in different trained peft parameters through element-wise operation. we leverage unlabeled data to train language parameters with peft, and perform element-wise arithmetic operations with pretrained task and language parameters to construct new parameters for a language without labeled summarization data. while several prior works have studied methods that compose peft methods for zero-shot cross-lingual transfer @xcite , these methods generally incur an additional inference cost. our method provides a simpler and more flexible framework to leverage many related languages at a fixed inference cost. confidential -google deepmind θ pt;@xmath1;xlsum + (1-λ)(θ pt;lm -θ en;lm ) palm 2 inputs: o fenómeno taylor swift está prestes a chegar a portugal e, se calhar, targets: ninguém está preparado para o frenesim que irá gerar. breslin will play the lead role of baby houseman. dirty dancing is the songand-dance love story about 17-year-old baby, who falls for her working-class dance instructor, johnny. … in 2011, lionsgate announced plans to remake the film for the big screen. summary: the 1987 movie dirty dancing is to be remade for tv, by us network abc. inputs: tastes like strawberries on a summer evenin' targets: and it sounds just like a song pt lm en xlsum nos estados unidos, o filme arrecadou us @xmath018 milhões no seu primeiro dia de apresentação. inference time pt xlsum en lm palm 2 te xlsum bn xlsum si xlsum θ si;@xmath2;xlsum + θ bn;xlsum palm 2 en xlsum ja xlsum zh xlsum θ zh;@xmath3;xlsum + θ ko;xlsum + θ ja;xlsum our method is inspired by the lottery ticket hypothesis @xcite , which posits that distinct models fine-tuned on the same dataset follow linear trajectories while maintaining a consistent loss @xcite . this hypothesis implies that element-wise operations on different fine-tuned models can also remove biases of the pretrained model @xcite , allowing the accumulation of information from auxiliary tasks @xcite , or improve adaptation to unforeseen textual domains @xcite . our work is the first to extend this observation to improve cross-lingual transfer by combining pretrained language and task parameters. our contributions are the following: ## language and task arithmetic prior work has applied element-wise operations to the weights of fine-tuned models @xcite @xcite @xcite , or peft modules @xcite . these studies demonstrate that interpolating the weights of fine-tuned models (or specific layers) effectively creates multi-task and multidomain models. we hypothesize that element-wise operations can also be used to combine knowledge acquired in different languages. our work is the first to propose the arithmetic composition of language and task peft modules for cross-lingual natural language generation. figure 1 illustrates an overview of our approach. 1. our goal is to enable large language models (llms) to support summarization in an unseen target language (t ) for which we lack labeled data. we assume access to labeled task data in other languages, as well as unlabeled monolingual data in both the source language (s) and the target language (t ). in particular, we can use either labeled or unlabeled data to train small peft modules that capture the attributes of a given task or language. task adapter: we fine-tune an llm using lora adapters on labeled data from xlsum @xcite in the source language s. we refer to the fine-tuned model as task adapter. language adapter: we fine-tune lora parameters with llms on monolingual data in the source or target language (s or t ). we refer to the finetuned model as language adapter. we use the prefix-lm pretraining objective from t5 @xcite with mc4 data to train language adapters. we propose to compose the language and task vectors to better support summarization into the target language t . next, we introduce our method under two different data settings. ## related work llms have shown impressive performance in various natural language processing tasks @xcite @xcite , often requiring no extra training to adapt to downstream tasks. numerous parameter-efficient methods have been proposed, each addressing the challenge of enhancing efficiency . these methods can be categorized as input composition, function composition, and parameter composition @xcite . input composition methods, such as prompt tuning, incorporate soft prompts into the input layers to guide the model's behavior @xcite . function composition strategies, like adapters @xcite , introduce non-linear functions within pretrained layers to adapt the intermediate representations of the model. parameter composition is exemplified by methods like lora @xcite , which introduces a limited number of learnable low-rank matrices into each pretrained layer. recent work which is based on the linear mode connectivity @xcite suggests averaging the weights of pretrained models fine-tuned on the same dataset with different hyperparameters to improve downstream performance @xcite @xcite . it has also been shown that averaging the weights of models fine-tuned on different tasks improves outof-domain generalization without leaking information about potentially private labeled datasets @xcite . composing weights of models finetuned on tasks related to the target task is also beneficial @xcite 2023 show that a model can acquire multi-task learning abilities using model merging, while @xcite propose model merging by reducing gradient mismatch. there is also work on averaging domain-specific adapter layers @xcite or domain-expert lms @xcite with large gains for unseen domains. however, there is no work on peft cross-lingual transfer using language and task arithmetic. in a similar line of thought and to mitigate interference of different tasks during training, @xcite train task peft modules and learn attention parameters to select the most useful of them, while karimi mahabadi et al. (2021) learn adapters with hypernetworks. asai et al. (2022) efficiently integrate knowledge from multiple tasks with a mix of trainable soft prompts. ponti et al. (2023) propose polytropon, which learns both adapters and a binary task-module routing matrix, determining which module should be active for each task; caccia et al. (2023) extend it to a more granular level by mixing subsets of adapter dimensions. another research direction considers training peft parameters and combining them for crosslingual transfer. mad-x @xcite stacks task bottleneck adapters with language adapters and using them for cross-lingual transfer. @xcite identify the parameters that are most useful for a task and a language, and compose them; this work is based on the lottery ticket hypothesis @xcite . @xcite propose factorizing a prompt into a language and task and training each part while keeping the other frozen. newly learned knowledge is combined with the existing model using peft modules to permit cross-lingual transfer in multiple recent works @xcite @xcite @xcite . to the best of our knowledge, our work is the first to propose improving cross-lingual transfer of a llm via a combination of weights of peft parameters. ## conclusion we present a new method to compose knowledge from parameter-efficient modules using arithmetic operations in order to improve zero-shot crosslingual transfer. our experiments in summarization on a wide set of languages using palm 2 as the pretrained model show that our language and task arithmetic achieves consistent improvements over the baselines and introduces a modular approach that can be leveraged for improved generalization of a llm in languages that lack labeled data.