MUSCLE : A Model Update Strategy for Compatible LLM Evolution (2024)

Abstract: Large Language Models (LLMs) are regularly updated to enhance performance, typically through changes in data or architecture. Within the update process, developers often prioritize improving overall performance metrics, paying less attention to maintaining compatibility with earlier model versions. Instance-level degradation (instance regression) of performance from one model version to the next can interfere with a user’s mental model of the capabilities of a particular language model. Users having to adapt their mental model with every update can lead to dissatisfaction, especially when the new model has degraded compared to a prior version for a known use case (model update regression).We find that when pretrained LLM base models are updated, fine-tuned user-facing downstream task adapters experience negative flips – previously correct instances are now predicted incorrectly. We observe model update regression between different model versions on a diverse set of tasks and models, even when the downstream task training procedures remain identical. We argue for the importance of maintaining model update compatibility during updates, and present evaluation metrics designed specifically for generative tasks, while also being applicable to discriminative tasks. We propose a training strategy to minimize the extent of instance regression in model updates, involving training of a compatibility adapter that can enhance task fine-tuned language models. We show negative flips reduce by up to 40% e.g. when updating Llama 1 to Llama 2 with our proposed method.

## introduction large language models (llms) are often pretrained on large-scale corpora to obtain a base model with general world knowledge. these base the present is not expensive. it is a present. the present will be bought. ## problem formulation while existing methods provide valuable approaches to mitigating model update regression, they predominantly focus on discriminative classification tasks and often require additional memory at inference time. we propose a flexible approach to updating models without sacrificing performance or compatibility across downstream tasks. ## knowledge transfer now that we have metrics to indicate model update regression, we propose a knowledge distillation approach to minimize this regression for the taskspecific models m v1 and m v2 . typically, knowledge distillation minimizes the kl divergence between the soft targets σ(z t ) and σ(z s ), where z s and z t are logits predicted by student and teacher models, respectively. ## conclusion in this work, we study the task-specific compatibility problem when updating llms. we show that llm updates with different scenarios, e.g., changes in model architecture, optimization, or training dataset, exhibit significant negative flips -instances previously classified or generated correctly, and incorrectly after the model update. we extend the negative flip metric for discriminative and, for the first time, generative tasks, and report results for various models and tasks. we propose a novel method (muscle) to train task-specific compatibility adapters when updating an old llm to a new llm to reduce negative flips while maintaining performance gain. our proposed method does not require a modification to the base model training and is only based on adapter training. further, as opposed to previous works, the proposed solution does not require both versions of the model in memory to enable compatibility, which is often infeasible due to the large size of llms. we observe a mitigation of negative flips of 40% for multiple-choice type evaluations, and 27% for continuous summarization evaluation. we also show insights into model properties that facilitate transfer, finding that our alignment masking strategy provides best results with the additional benefit of mitigating inconsistent update behavior.