More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLM s (2024)

Abstract: The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.

## introduction large language models (llms) are sequentially trained on general pre-training corpus, pairs of instruction-response and preference-alignment datasets, thus covering tasks involving writing @xcite @xcite , math @xcite general capabilities integration 🤔 i have already accumulated 6 demerit points on my driving license this year, and i just accidentally ran a red light. i want to know if all my points will be deducted? ## alora in figure 3 , we introduce the framework of alora, which integrates a multi-head attention module upon lora. vanilla lora integrates trainable adapters alongside pre-trained linear weights. however, the input to lora predominantly seizes the representation of the current token, resulting in the adapter's over-fitted alignment with the patterns of domain knowledge. in order to facilitate the model in switching between domain-specific knowledge and general capabilities at appropriate time steps, we replace the vanilla lora with an adapter that features a context-aware multi-head attention module. let q l-1 , k l-1 , v l-1 denote the query, key, value vector of transformer layer l -1 respectively. following @xcite , a hidden states h @xmath0, including 2 machine reading comprehension 3 http://cail.cipsc.org.cn/ 4 https://wenshu.court.gov.cn/ scaled dot-product attention, normalization, etc., where t is the length of the sequence, and d is the dimension. w is the pre-trained linear layers, nh and dh are the number of attention heads and the dimension of each foot_1 . for the attention, query representation given the key and value vectors k l-1 , v l-1 ∈ r t×nh×dh from the attention block of the last layer, the attention output hv @xmath1, the language modeling objective is where p ϕ (y|x) models the probability of y given x with parameter ϕ. to prevent the model from over-fitting to the distribution of the domain data, an additional regularization loss of kl divergence is added, the main experiments and ablation study for gci setting are discussed in the following subsections. we mainly follow luo et al. (2023) for baselines, including direct supervised fine-tuning with lora, wise-ft, methods with normalization. for multitask learning, we fine-tune the llms on the mixture of a general instruction-following dataset and the domain-specific ("article qa" or "legal ie") dataset. in addition, we also include ke et al. (2023) and @xcite , which are denoted by "das" and "mixda*" foot_2 respectively. the details of the baselines and implementation can be found in appendix a and b. we discuss the limitation of domain knowledge injection and adapter fusion for decoder-only model in appendix d. we also investigate the influence of foundation model and scale to alora in appendix e. as alora is adaptive with other baselines, we combine alora with "mixda*" to illustrate the adaptability in appendix f. the case study and prompts details are shown in appendix i and j. for all tables, we highlight the best results with bold, and underline the second best results. ## conclusion it is difficult to integrate the general capabilities and domain-specific knowledge within domaininstances, even when cf is addressed. therefore we propose a new setting beyond cf, requiring further harmonization and utilization of both sets of skills in a cohesive manner, called general capabilities integration (gci). taking legal domain as an example, we manually design three groups of practical tasks and carefully construct the corresponding datasets. additionally, a novel adapter structure based on attention and lora, named alora, is proposed to facilitate gci. extensive experiments demonstrate the distinction of gci and the effectiveness of alora.