Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion (2023)

Abstract: When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression errors. This inconsistent behavior during model upgrade often outweighs the benefits of accuracy gain and hinders the adoption of new models. To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without significant compromise in performance. Despite the progress, these approaches attained an incremental reduction in regression which is still far from achieving backward-compatible model upgrade. In this work, we propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models. Empirical results on two distinct model upgrade scenarios show that our method reduces the number of regression errors by 62% on average, outperforming the strongest baseline by an average of 25%.

## introduction in order to achieve a smooth continuous improvement of nlp applications, it is critical to guarantee consistent operation of the system after an upgrade. new errors introduced during the model upgrade interfere with the existing user experience and are considered to be a regression in the quality. due to the difficulty of modularizing or explaining the behavior of deep neural networks, traditional software regression tests are inapplicable to neural based systems. the cost of arduous error analysis and model patching often exceeds the benefits of model upgrades. developing methods that ensure backward compatibility during model upgrades without compromise in performance becomes a valuable research direction @xcite @xcite . @xcite to elec-tra @xcite for classification. red circles and green squares denote examples of different classes. ## the backward-compatible model upgrade problem the goal of backward-compatible model upgrade is to minimize regression errors without compromising the accuracy performance during model upgrade @xcite . in this work, we aim to improve the backward compatibility of model predictions in the nlp classification tasks. following @xcite , we study the scenario where the underlying pretrained language model (lm) is being upgraded. let x be a natural language input with a class label @xmath0, 2, ..., c}. @xmath1, y i } n @xmath2. a classifier f estimates the class probabilities given the input ⃗ f (x) = (p(@xmath3, ..., p(@xmath4. when upgrading from an old model f old to a new model f new , normally with distinct architectures and trained on the same data, an improved model f * is produced based on f old and f new . our goal is for f * to minimize regression errors as an additional objective, while still achieving comparable performance to f o new , the new model trained in the vanilla setting. note that f * could be multiple times larger than f o new , with model ensemble of f o new as one example @xcite . ## discussion gated fusion requires to host both old and new models at inference time, which could raise a concern regarding the increased computational burden. however, in practice, old model's logits of previous inference instances can be cached in storage and later leveraged in our gated fusion. that is, we only need to host the new model with the gate at inference time and leverage old predictions from cache. and for the out-of-cache inputs, backwardcompatibility would be less of an issue since users have not observed such examples to make conclusions on the underlying regression. for real-world applications, there could be multiple model updates and thus multiple legacy versions. we note that in this scenario, user experience would be primarily grounded on predictions of the latest legacy version, which are also saved in cache. our gated fusion can hence leverage them and make new model's predictions compatible to those from the latest legacy version. in addition, we emphasize that the main challenge in the regression reduction research problem is to find the best trade-off between model effectiveness and backward compatibility. in this work, we show that the weighted ensemble of old-new models with a learned gate, which we call gated fusion, achieves a better negative flip rate than previously explored methods for regression reduction, while straight-forward ensemble approaches cannot naturally weigh on this trade-off. we don't claim to invent the gated ensemble of old and new models but rather that our main contribution is to show that by repurposing the classic gating mechanism, the gated ensemble can become the most competitive approach to the challenging model-upgrade regression reduction problem, with no overall per-formance degradation on two realistic model update scenarios across three different datasets. recently, more and more nlp products have been deployed in the industry as this field matures. we would like to stress that as better nlp models are being developed, the backward-compatible model upgrade problem naturally emerges as the new research topic strongly motivated by the realworld challenges. while backward-compatibility is currently a niche research topic, we believe that there are many thrilling future directions worth to be investigated. reducing regression during model upgrade is also related to continual learning @xcite @xcite @xcite , incremental learning @xcite and concept drifting @xcite @xcite @xcite . in these problems, models are required to learn from and deal with continuously changing data (in terms of examples, classes or tasks), and also need to prevent the forgetting of previously learnt knowledge. this could be one potential cause of regression observed at inference. however, in backward-compatible model upgrade, a new model, usually with distinct network architecture, is trained from scratch to perform the same task and is expected to behave similarly wherever the previous model predicts correctly. the gating mechanism is widely adopted by recurrent neural networks to effectively control information flows across networks @xcite @xcite @xcite and contextualize embeddings @xcite . it is then repurposed to act as a switch for the mixture of different prediction modes, notably to combine input word copying based on the pointer network @xcite with the word generation from output vocabulary @xcite @xcite . our proposed approach is inspired by these works and leverages the gating mechanism to effectively combine old and new models to improve backward compatibility during model upgrade. ## conclusion ensuring backward compatibility during model upgrade has become a critical topic in real-world nlp applications. in this work, we proposed a new approach, gated fusion, that achieves significantly better backward compatibility without compromising accuracy performance on two challenging upgrade scenarios for nlp classification. experiments demonstrated that our approach outperforms competing methods and achieves negative flip rate reductions by up to 73.2%. our future research includes improving backward compatibility beyond classification to span detection, model upgrades with very large language models, and upgrades on training data or label schema. we hope that this work can inspire further research and make progress towards smoother transitions of prediction powers as nlp systems evolve.