Multi- CLS BERT : An Efficient Alternative to Traditional Ensembling (2023)

Abstract: Ensembling BERT models often significantly improves accuracy, but at the cost of significantly more computation and memory footprint. In this work, we propose Multi-CLS BERT, a novel ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model. Multi-CLS BERT uses multiple CLS tokens with a parameterization and objective that encourages their diversity. Thus instead of fine-tuning each BERT model in an ensemble (and running them all at test time), we need only fine-tune our single Multi-CLS BERT model (and run the one model at test time, ensembling just the multiple final CLS embeddings). To test its effectiveness, we build Multi-CLS BERT on top of a state-of-the-art pretraining method for BERT (Aroca-Ouellette and Rudzicz, 2020). In experiments on GLUE and SuperGLUE we show that our Multi-CLS BERT reliably improves both overall accuracy and confidence estimation. When only 100 training samples are available in GLUE, the Multi-CLS BERT_Base model can even outperform the corresponding BERT_Large model. We analyze the behavior of our Multi-CLS BERT, showing that it has many of the same characteristics and behavior as a typical BERT 5-way ensemble, but with nearly 4-times less computation and memory.

## introduction bert (bidirectional encoder representations from transformers) @xcite is one of the most widely-used language model (lm) architectures for natural language understanding (nlu) tasks. we often fine-tune the pretrained bert or its variants such as roberta @xcite so that the lms learn to aggregate all the contextualized word embeddings into a single cls embedding for a downstream text classification task. during fine-tuning, different initializations and different training data orders significantly affect bert's generalization performance, especially with a small training dataset @xcite @xcite . one simple and popular solution to the issue is to finetune bert model multiple times using different random seeds and ensemble their predictions to improve its accuracy and confidence estimation. although very effective, the memory and computational cost of ensembling a large lm is often prohibitive @xcite . naturally, we would like to ask, "is it possible to ensemble bert models at no extra cost?" to answer the question, we propose multi-cls bert, which enjoys the benefits of ensembling without sacrificing efficiency. specifically, we input the multiple cls tokens to bert and encourage the different cls embeddings to aggregate the information from different aspects of the input text. as shown in figure 1 , the proposed multi-cls bert shares all the hidden states of the input text and only ensembles different ways of aggregating the hidden states. since the input text is usually much longer than the number of inputted cls embeddings, multi-cls bert is almost as efficient as the original bert. allen @xcite discovered that the key of an effective ensembling model is the diversity of individual models and the models trained using different random seeds have more diverse predictions compared to simply using dropout @xcite or averaging the weights of the models during training @xcite . to ensure the diversity of cls embeddings without fine-tuning multi-cls bert using multiple seeds, we propose several novel diversification techniques. for example, we insert different linear layers into the transformer encoder for different cls tokens. furthermore, we propose a novel reparametrization trick to prevent the linear layers from learning the same weights during fine-tuning. we test the effectiveness of these techniques by modifying the multi-task pretraining method proposed by @xcite , which combines four self-supervised losses. in our experiments, we demonstrate that the resulting multi-cls bert can significantly improve the accuracy on glue @xcite and su-perglue @xcite , especially when the training sizes are small. similar to the bert ensemble model, we further show that multiple cls embeddings significantly reduce the expected calibration error, which measures the quality of prediction confidence, on the glue benchmark. ## method in sections 2.1 and 2.2, we first review its state-ofthe-art pretraining method from @xcite . in section 2.3, we modify one of its losses, quick thoughts (qt), to pretrain our multiple embedding representation. in section 2.4, we encourage the cls embeddings to capture the fine-grained semantic meaning of the input sequence by adding hard negatives during the pretraining. to diversify the cls embeddings, we modify the transformer encoder in section 2.5 and propose a new reparametrization method during the fine-tuning in section 2.6. ## experiments the parameters of neural networks are more restricted as more training samples are available @xcite and the improvement of deep ensemble models comes from the diversity of individual models @xcite , so the benefits of ensembling are usually more obvious when the training set size is smaller. therefore, in addition to using the full training dataset, we also test the settings where the models are trained by 1k samples @xcite or 100 samples from each task in glue @xcite or super-glue @xcite . another benefit of the 1k-and 100-sampling settings is that the average scores would be significantly influenced by most datasets rather than by only a subset of relatively small datasets @xcite . ## conclusion in this work, we propose representing the input text using k cls embeddings rather than using the single cls embedding in bert. compared to bert, multi-cls bert significantly increases the glue and superglue scores and reduces the expected calibration error in glue, while its only added cost is to reduce the maximal text length by k and add a little extra time for computing the inserted linear transformations. therefore, we recommend the wide use of multiple cls embeddings for the almost free performance gain. to solve the collapsing problem of cls embeddings, we modify the pretraining loss, bert architecture, and fine-tuning loss. the ablation study shows that all of these modifications contribute to the performance improvement of multi-cls bert. in our analyses for investigating the source of the improvement, we find that a) ensembling the original bert leads to greater improvement than ensembling the multi-cls bert and b) the disagreement of different cls embeddings highly correlates with the disagreement of the bert models from different fine-tuning seeds. both findings support our perspective that multi-cls bert is an efficient ensembling method.