Chat Vector: A Simple Approach to Equip LLM s with Instruction Following and Model Alignment in New Languages (2024)

Abstract: Recently, the development of open-source large language models (LLMs) has advanced rapidly. Nevertheless, due to data constraints, the capabilities of most open-source LLMs are primarily focused on English. To address this issue, we introduce the concept of \textit{chat vector} to equip pre-trained language models with instruction following and human value alignment via simple model arithmetic. The chat vector is derived by subtracting the weights of a pre-trained base model (e.g. LLaMA2) from those of its corresponding chat model (e.g. LLaMA2-chat). By simply adding the chat vector to a continual pre-trained model’s weights, we can endow the model with chat capabilities in new languages without the need for further training.Our empirical studies demonstrate the superior efficacy of the chat vector from three different aspects: instruction following, toxicity mitigation, and multi-turn dialogue. Moreover, to showcase the adaptability of our approach, we extend our experiments to encompass various languages, base models, and chat vectors. The results underscore the chat vector’s simplicity, effectiveness, and wide applicability, making it a compelling solution for efficiently enabling conversational capabilities in pre-trained language models. Our code is available at https://github.com/aqweteddy/ChatVector.

## introduction large language models (llms) have garnered significant attention due to their strong performance across a wide range of natural language tasks, showcasing remarkable proficiency in following instructions. despite the rapid development of llms, the language capabilities of most of these models are constrained to english due to limitations in data chat model (llama2-chat) plm (llama2) cp model target lm target lm chat vector parameter subtraction cp sft & rlhf parameter addition figure 1 : an illustration to demonstrate the difference between the traditional approach and our method. the blue arrows on the top right side depict the conventional method of constructing a non-english lm. first, an open-source plm (e.g. llama2) undergoes continual pre-training (cp) on the target language, followed by sft and rlhf alignment procedures. in contrast, the gray arrow on the left illustrates how we obtain the chat vector through simple parameter subtraction. this chat vector can be added to the cp model to produce the chat model in the target language, as depicted by the dual-color arrow. for individuals working with non-english languages, creating a llm from scratch can be computationally intensive. as a result, many turn to adopt open-source, english-based pre-trained llms, such as bloom @xcite , llama2 @xcite , and mistral-7b @xcite , as foundational models. inspired by @xcite , building a non-english llm involves continual pre-training on the target language to enhance the model's fluency, and is followed by sft using specific instructional data to sharpen task-specific performance and ensure instruction-following capabilities in the target language @xcite @xcite . however, to align the model with human preferences, reinforcement learning from human feedback (rlhf) @xcite presents a more complex challenge. it involves the development of alignment criteria, the acquisition of human feedback, and final learning adjustments based on this feedback. llama2 @xcite is currently one of the publicly available models utilizing rlhf, with other models such as webgpt @xcite , instructgpt @xcite , and gpt-4 (openai, 2023) being proprietary. implementing rlhf is intricate, stemming not only from the need for human annotations but also due to technical challenges. these include overfitting in reward models and instabilities during the reinforcement learning training phase @xcite . additionally, the tedious procedure of training multiple lms including the model being aligned, the reward model, and the inference model at the same time substantially amplifies memory and computational demands, particularly for larger models. @xcite proposed direct preference optimization (dpo) to align models with human preferences instead of complex and unstable rlhf. nevertheless, one still needs to collect human-labeled preference data in the target language. in this work, we aim to enhance the alignment of non-english llms with human preferences. inspired by the concept of task vectors @xcite , we hypothesize that given a consistent base model, pre-existing knowledge and acquired behaviors can be synergized through a straightforward vector addition in the parameter space. to achieve this, we propose an approach to restructure the conventional training paradigm for non-english llms from @xmath0. the chat vector is derived by subtracting llama-2's pre-trained weights from those of its chat-enhanced counterpart, llama-2-chat. by introducing this chat vector to a llama-2-based model that's continually pre-trained on non-english content, the evolved model responds in the target language, both in providing answers and declining inappropriate requests, and it aligns more deeply with human preferences. the main process of our method is illustrated in figure 1 . we assess the efficacy of the chat vector across multiple target languages, focusing primarily on traditional chinese, by considering three aspects: the ability to follow instructions, toxicity, and multi-turn dialogue. the models are evaluated on three benchmarks: safetyprompts @xcite , realtoxicityprompts @xcite , and the vicuna benchmark @xcite , with gpt-4 handling the translation of the latter two into the target language. the results demonstrate that the strategy of incorporating the chat vector after continual pre-training yielded superior outcomes compared to direct pretraining on llama-2-chat. furthermore, applying fine-tuning prior to the integration of the chat vector optimizes performance irrespective of the fine-tuning dataset's scale or the language of the pre-trained model. additionally, we demonstrated that the chat vector does not cause catastrophic forgetting @xcite ) by examining it from both linguistic and knowledge-based perspectives. beyond merely augmenting an llm's conversational skills, it offers crucial insights into the meaning of learning weights in the parameter space and the integration of added vectors with pre-existing knowledge. most importantly, performing arithmetic operations on the chat vector is substantially more efficient than reimplementing rlhf in the target language. our primary contributions are the following: ## experimental result in this section, we demonstrate our experimental result from three perspectives: instruction following ability, safety, and multi-turn conversations. ## conclusion in this work, we present a novel approach to endow llms with chat capabilities in a new language. by continual pre-training and integrating the chat vector into an english-based pre-trained language model, the model gains the ability to follow instructions, align with human values, generate safe responses, and engage in multi-turn dialogues. unlike current methods involving cp, sft, and human preference training, our approach relies solely on cp and straightforward arithmetic operations. this significantly reduces the cost associated with aligning models with human preferences, making it a more efficient and scalable solution for imbuing llms with chat capabilities across multiple languages.