To BERT or Not to BERT : Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging (2020)

Abstract: Leveraging large amounts of unlabeled data using Transformer-like architectures, like BERT, has gained popularity in recent times owing to their effectiveness in learning general representations that can then be further fine-tuned for downstream tasks to much success. However, training these models can be costly both from an economic and environmental standpoint. In this work, we investigate how to effectively use unlabeled data: by exploring the task-specific semi-supervised approach, Cross-View Training (CVT) and comparing it with task-agnostic BERT in multiple settings that include domain and task relevant English data. CVT uses a much lighter model architecture and we show that it achieves similar performance to BERT on a set of sequence tagging tasks, with lesser financial and environmental impact.

## introduction exploiting unlabeled data to improve performance has become the foundation for many natural language processing tasks. the question we investigate in this paper is how to effectively use unlabeled data: in a task-agnostic or a task-specific way. an example of the former is training models on language model (lm) like objectives on a large unlabeled corpus to learn general representations, as in elmo (embeddings from language models) @xcite and bert (bidirectional encoder representations from transformers) @xcite . these are then reused in supervised training on a downstream task. these pre-trained models, particularly the ones based on the transformer architecture @xcite 1 have achieved state-of-the-art results in a variety of nlp tasks, but come at a great cost financially and environmentally @xcite . in contrast, cross-view training (cvt) @xcite is a semi-supervised approach that uses unlabeled data in a task-specific manner, rather than trying to learn general representations that can be used for many downstream tasks. inspired by selflearning @xcite and multi-view learning @xcite , the key idea is that the primary prediction module, which has an unrestricted view of the data, trains on the task using labeled examples, and makes task-specific predictions on unlabeled data. the auxiliary modules, with restricted views of the unlabeled data, attempt to replicate the primary module predictions. this helps to learn better representations for the task. we present an experimental study that investigates different task-agnostic and task-specific approaches to use unsupervised data and evaluates them in terms of performance as well as financial and environmental impact. on the one hand, we use bert in three different settings: 1) standard bert setup in which bert pretrained on a generic corpus is fine-tuned on a supervised task; 2) pre-training bert on domain and/or task relevant unlabeled data and fine-tuning on a supervised task (pretrained bert); and 3) continued pretraining of bert on domain and/or task relevant unlabeled data followed by fine-tuning on a supervised task (adaptively pretrained bert) @xcite . on the other hand, we use cvt based on a much lighter architecture (cnn-bilstm) which uses domain and/or task relevant unlabeled data in a task-specific manner. we experiment on several tasks framed as a sequence labeling problem: opinion target expression detection, named entity recognition and slot-labeling. we find that the cvt-based approach using less unlabeled data achieves similar performance with bert-based models, while being superior in terms of financial and environmental cost as well. ## models and experimental setup we describe the various models we compare in this work and the experimental setup for each of them. experiments are geared towards comparing the performance accuracy of the models, while also measuring impact on the environment and the resources required for training these models. details on model architecture and training are provided in appendix a. cross-view training (cvt) cvt is a semisupervised approach proposed by @xcite that leverages unlabeled data in a taskspecific manner. the underlying model is a twolayer cnn-bilstm sentence encoder followed by a linear layer and a softmax per prediction module. there are two kinds of prediction modulesprimary and auxiliary. cvt alternates between learning from labeled and unlabeled data during training. the key idea is that the primary prediction module, which has an unrestricted view of the data, trains on the task using labeled examples, and makes task-specific predictions on unlabeled data. the auxiliary modules, with different restricted views of the unlabeled data, attempt to mimic the predictions of the primary module. standard cross-entropy loss is minimized when learning from labeled examples, while for unlabeled examples, kl-divergence @xcite between the predicted primary and auxiliary probability distributions is minimized (see @xcite for more details). we demonstrate the training strategy in figure 2 . thus, the model is trained to produce consistent results despite seeing partial views of the input -thereby improving underlying representations. we use glove 840b.300d embeddings (pennington et al., 2014) instead of glove 6b.300d embeddings used by the authors for a larger vocabulary coverage. for each of the labeled datasets (section 2), we use the corresponding domain/task-relevant unlabeled data to train a sequence tagging model for 400k steps, with early stopping enabled using validation set convergence. bert base bert @xcite has achieved state-of-the-art results on many nlp tasks. the key innovation lies in the use of bi-directional transformers as well as the masked language model (mlm) and next sentence prediction (nsp) objectives used during training. learning happens in two steps: 1) training the model on a very large generic dataset (using the two objectives above); 2) fine-tuning the learned representations on a downstream task in a supervised fashion. for our experiments we use bert base , which has 12 layers, 768 hidden dimensions per token and 12 attention heads and is pre-trained on the cased english wikipedia and books corpus data (wiki+books). in order to fine-tune on the downstream sequence tagging task, the model we use consists of the bert base encoder, followed by a dropout layer and a classification layer that classifies each token into b-label, i-label, o, where @xmath0, label i+1 , ..., label n }. ## results we present here metrics-based and resource-based comparison of cvt and bert models on all tasks. state-of-the-art (sota) baselines are included for reference. results on conll-2003 dataset model unlabeled data mean f1 cvt cnn-dm (∼17m) 92.26±0.11 bert base wiki+books (∼192m) 91.22±0.21 pre-bert base cnn-dm (∼146m) 85.54±0.19 apbert base cnn-dm (∼138m) 88.02±0.18 cloze wiki+books (∼192m) 93.5 results on conll-2012 dataset model unlabeled data mean f1 cvt cnn-dm (∼18m) 89.26±0.1 bert base wiki+books (∼192m) 89.0±0.23 pre-bert base cnn-dm (∼146m) 84.20±0.19 apbert base cnn-dm (∼138m) 85.88±0.17 bert-mrc+dsc wiki+books (∼192m) 92.07 performance metrics we report mean f1 (with standard deviation) on the labeled test splits for each task over 5 randomized runs, and compare the models using statistical significance tests over these runs. further, we report the approximate number of unlabeled sentences seen by each model. in tables 3 and 4 , we present f1 results on ner and slot-labeling task, respectively. for all 3 datasets, we find cvt to outperform all bert models (statistically significant for conll-2003 and mit movies dataset, at p-values 0.0086 and 0.0085, respectively). for these tasks, bert base outperforms apbert base models. furthermore, cvt outperforms sota for slot-labeling task. these results show that the cvt model, using unlabeled data in a task-specific manner, is more robust across different tasks and types of unlabeled data. for ote detection, the unlabeled data is closely related to both domain and task, while for ner and slot-labeling, the unlabeled data is related to genre (newswire) and domain (movies), but not necessarily to the specific tasks. in line with the findings of @xcite , adaptive pre-training shows best results when using unlabeled data that is domain and task relevant (superior results for the ote task). it is also worth noting that cvt requires significantly smaller amount of unlabeled data than the bert models (tables 2 , 3 and 4 ). resource cost ## conclusion & future work we compare the task-specific semi-supervised method, cvt, with a task-agnostic semi-supervised approach, bert (with and without adaptive pretraining), on a variety of problems that can be modeled as sequence tagging tasks. we find that the cvt-based approach is more robust than bertbased models across tasks and types of unsupervised data available to them. furthermore, the financial and environmental costs incurred are also significantly lower using cvt as compared to bert. as part of future work, we will explore cvt on other sequence-labeling tasks such as chunking, elementary discourse unit segmentation and argumentative discourse unit segmentation, thus moving beyond entity-level spans. moreover, other supervised tasks such as classification could also be studied in this context. furthermore, we intend to implement cvt as a training strategy over transformers (bert) and compare it with adaptively-pretrained bert.