F a LA : Fast Linear Adaptation for Replacing Backbone Models on Edge Devices (2023)

Abstract: In this work, we study the language model backbone replacement problem for personalized downstream tasks in a non-stationary on-device scenario. In real world, company may periodically update the knowledge and architectures of backbones to keep the competitive in the market, meanwhile, to accommodate the users’ own preference, models are personalized to fit users’ own distribution locally. Traditional full model tuning or transfer learning for such replacements often incur considerable local device training costs and necessitate extensive backpropagation within deep transformer layers. Addressing this issue, we propose a novel, lightweight tuning method for personalized NLP classification tasks post-backbone replacement. Our approach leverages a personalized matrix calculated from documents corresponding to users’ old and new backbones. This matrix facilitates top-layer parameter tuning, drastically reducing backpropagation computation. To further mitigate training costs associated with matrix linear optimization, we employ correlation clustering to curate a few examples from personalized cluster sets for individuals. Our method achieves over 1000 times computation reduction in Flops for backpropagation and brings the user-specific initialization for personal matrix yielding significant performance boost compared with popular transfer learning methods.

## introduction current natural language processing (nlp) models, including the ones developed for edge devices @xcite , heavily rely on pretrained backbone models, such as bert @xcite and roberta @xcite . recent studies show that it is important to personalize models on edge devices because of the privacy concerns and the distribution discrepancies between local data and public training data @xcite . ## methodology an edge device model provider regularly devises and trains new models on public or companyowned data. when deploying new models to edge devices, the goal is to minimize local tuning costs to achieve superior performance than the previous ones, while preserving privacy. in the following, the models trained on the public data is referred to as global models, while the models deployed to local devices are called local models. ## ablation study to further explore and validate our methods, we conduct ablation study focusing on diverse factors throughout our personalized matrix computation process. ## conclusion in this paper, we introduce fala , the fast linear adaptation method for backbone model replacement. we consider a novel and realistic dynamic training scenario that the service providers want to minimize the computational cost of updating the backbone of the on-device models deployed to their client devices while preserving privacy of the local data. specifically, we reformulate the complex model replacement problem as a simple ridge regression problem. our approach notably outperforms established baseline techniques in terms of both accuracy and computational efficiency. this work identifies new research problems regarding backbone replacement. for future work, we may move beyond classification tasks to broader applications with diverse backbones.