CAMERO : Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing (2022)

Abstract: Model ensemble is a popular approach to produce a low-variance and well-generalized model. However, it induces large memory and inference costs, which is often not affordable for real-world deployment. Existing work has resorted to sharing weights among models. However, when increasing the proportion of the shared weights, the resulting models tend to be similar, and the benefits of using model ensemble diminish. To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensemble learning approach based on perturbed models, named CAMERO. Specifically, we share the weights of bottom layers across all models and apply different perturbations to the hidden representations for different models, which can effectively promote the model diversity. Meanwhile, we apply a prediction consistency regularizer across the perturbed models to control the variance due to the model diversity. Our experiments using large language models demonstrate that CAMERO significantly improves the generalization performance of the ensemble model. Specifically, CAMERO outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0.7 with a significantly smaller model size (114.2M vs. 880.6M).

## introduction deep neural networks (dnns) have achieved remarkable success in various fields and have become very powerful in learning complicated models @xcite @xcite . however, their remarkable representation powers come at the expense of large model variance, which may hurt the model generalization performance. a popular approach for reducing such variance is model ensemble, where the weights or predictions of a set of models are aggregated to produce the predictions @xcite . for example, @xcite show that a simple 2-model ensemble leads to notable improvement over a single model in computer vision tasks. despite such notable benefits, model ensemble has not been widely applied to large language models. the major barriers are its enormous storage and expensive inference cost, which linearly scales with the size and the number of models. therefore, it is often not affordable to ensemble large language models for deployment using memory-constrained and low-latency edge devices. to alleviate the memory burden, recent works have resorted to a weight-sharing strategy, where all models share the same set of bottom-layer weights, on top of which branches out a set of parallel, un-shared top-layer weights @xcite @xcite . since the shared weights are optimized to accommodate multiple diverse un-shared branches, they can learn shared representations with better generalization @xcite @xcite . for large models, however, such a weightsharing strategy no longer enjoys the same benefits. due to memory constraints, a significant proportion of bottom-layer weights need to be shared. accordingly, top-layer branches have only limited capacity, and therefore the resulting models tend to be similar @xcite @xcite @xcite (figure 1 (left)). due to the lack of the model diversity, their ensemble cannot gain much improvement in generalization. as shown in figure 1 (right), both the generalization performance and model variance of the ensemble model are similar to those of a single model when the branch size is small. to retain a light memory cost while maintaining the ensemble benefits, we propose a new consistency-regularized ensemble learning approach based on perturbed models -camero. specifically, we share the bottom-layer weights across all models and apply different perturbations to the hidden representations for different models. such a perturbation strategy effectively promotes the model diversity. accordingly, the weights at each layer are optimized to produce consistent outputs given diverse input representations from differently perturbed models. in other words, the shared weights are essentially an on-the-fly ensemble of all perturbed models. in the end, we ensemble all branches on top of the shared weights to produce the final model, which has both a low variance and good generalization performance. since we apply perturbations in large models with significant depth, different models' hidden representations may end up being extremely diverse, especially in upper layers. as a result, optimizing the shared weights to accommodate such perturbations can be very challenging. to prevent the models from being over-diverse, we apply a consistency regularizer to reduce variance across different models. specifically, such a consistency regularizer can be viewed as collaborative distillation across models @xcite @xcite @xcite . by regularizing the discrepancy between each model's output logits and the ensemble of these logits, it encourages all models to be consistent in their predictions. we thus adopt consistency regularization to control the perturbed models' diversity from being too large, and thus ease the optimization of the shared weights. we conduct thorough experiments to demonstrate camero's effectiveness and efficiency in ensembling large number of models with more than hundreds of millions of learning parameters. specifically, our experiments in fine-tuning the bert-base model on the glue benchmark achieve 0.7 points of gain in terms of taskaverage score with a significantly smaller parameters over the vanilla ensemble approach (114.2m vs. 880.6m) and achieve 1.2 points of gain with the same amount of learning parameters over the single model. camero also achieves significant improvements in neural machine translation on both low-resource and high-resource language pairs. furthermore, we verify that camero can learn shared layers with better generalization and ensemble model with smaller variance. we also investigate the effects of using different types and strengths of perturbation and consistency regularization techniques. in particular, we observe that models created with virtual adversarial perturbation @xcite and neuron dropout @xcite lead to ensemble models with the best generalization performance. lastly, we demonstrate camero's effectiveness on a larger-scale model, roberta-large @xcite , where it achieves 0.8 and 0.9 points of gain over the vanilla ensemble approach and single model performance, respectively. our codes are released at https: //github.com/cliang1453/camero . ## background notations. we use f (•; θ) to denote a mapping f associated with the parameter θ from the input sample to an output space, where the output is a multi-dimensional probability simplex for classification tasks and a scalar for regression tasks. we denote the model's final logits as g(•; θ), where f (•; θ) = σ(g(•; θ)) and σ(•) is the softmax function. we denote n pairs of data samples of the target task as {(x i , y i )} n @xmath0. the training loss of f (•; θ) is computed as ℓ(f (x i ; θ), y i ) for any given training instance (x i , y i ) where ℓ(•; •) denotes the loss function. we use d kl (p ||q) = k p k log(p k /q k ) to denote the kl-divergence of two discrete distributions p and q with the associated parameters of p k 's and q k 's, respectively. collaborative distillation. collaborative distillation approaches train two or more models in parallel while regularizing the consistency of their final prediction distributions @xcite @xcite @xcite . specifically, we use {f (•; θ j )} m @xmath1, ..., θ m , respectively, and denote θ = {θ 1 , ..., θ m }. a typical collaborative distillation approach solves the following optimization problem: for notational simplicity, we will omit the subscript i throughout the rest of the paper. here, e(x; θ) defines a mapping function associated with θ, which maps the input sample x to a multi-dimensional probability simplex or a scalar depending on the tasks. a commonly adopted ensemble-distillation approach makes e(•; θ) = σ( m @xmath2; θ j )) , where {w j } m @xmath3. d(•, •) denotes the distance metric of two discrete distributions p and q or two scalars p and q. d(p, q) can take the form of symmetric kl-divergence, 1 2 (d kl (p ||q)+d kl (q||p )). d(p, q) or euclidean distance p -q 2 2 . weight-sharing. weight-sharing technique has been adopted in several representation learning scenarios, e.g., multi-task learning @xcite @xcite , multidomain learning @xcite @xcite and multi-lingual tasks @xcite . weight-sharing strategy can reduce the number of free parameters in the model, which helps prevent overfitting in the training and lead to better generalization abilities. ## method we introduce camero, a weight-sharing ensemble learning approach based on consistencyregularized perturbed models. ## experiment we verify the effectiveness of camero on widely used benchmarks for natural language understanding and neural machine translation. ## analysis we first verify that camero leads to a wellgeneralized and low-variance ensemble model. we then demonstrate how the perturbation and consistency regularization strength influences the model diversity and performance. finally, we demonstrate camero's effectiveness on various types of perturbation and regularization techniques. ## conclusion we propose camero, a consistency-regularized ensemble learning approach based on perturbed models. such a strategy significantly improves the parameter efficiency of model ensemble in large language models, making it an accessible and powerful technique for learning ensemble models with better generalization performances.