Does QA -based intermediate training help fine-tuning language models for text classification? (2021)

Abstract: Fine-tuning pre-trained language models for downstream tasks has become a norm for NLP. Recently it is found that intermediate training can improve performance for fine-tuning language models for target tasks, high-level inference tasks such as Question Answering (QA) tend to work best as intermediate tasks. However it is not clear if intermediate training generally benefits various language models. In this paper, using the SQuAD-2.0 QA task for intermediate training for target text classification tasks, we experimented on eight tasks for single-sequence classification and eight tasks for sequence-pair classification using two base and two compact language models. Our experiments show that QA-based intermediate training generates varying transfer performance across different language models, except for similar QA tasks.

## introduction the framework of fine-tuning pre-trained language models (lms), especially transformer-based lms, for downstream tasks has shown state-of-the-art performance on many natural language processing (nlp) tasks @xcite . it is believed that the pre-training stage leads lms to develop general-purpose abilities and knowledge that can then be transferred to downstream tasks @xcite . to further improve the performance of pretrained lms on target tasks, two novel training approaches have been recently researched, namely further pre-training and intermediate training. a further pre-training stage for lms @xcite is a stage between pre-training and fine-tuning, which further pre-trains lms on an extra dataset using unsupervised objectives. it has been found that further pre-training lm on the target domain (domain-adaptive pre-training) leads to improved performance on target tasks @xcite . another effective transfer learning approach named intermediate training that chooses to train a lm model on an intermediate task via supervised manner and then fine-tune it on target tasks. this also leads to promising results across various nlp tasks including text classification, qa and sequence labeling @xcite @xcite ). text classification is the problem of classifying text into categories or classes which has been widely studied. in terms of input, there are mainly two forms of text classification problems: singlesequence classification tasks (e.g., sentiment classification and topic classification) and pairwise tasks (e.g., nli and ir-related qa). in recent years, a common approach to tackle text classification problems is to fine-tune a pre-trained lm on target text classification tasks. recently, advanced transfer learning-based approaches have been proposed to further improve the performance. for example, a recent work @xcite has studied how to finetune bert for text classification. they found that further pre-training lm using data within-task or in-domian can improve the performance of bert for text classification tasks. more recently, cross-task transfer learning technique for text classification has been investigated @xcite , and it is found that tasks that require high-level inference and reasoning abilities, such as natural language inference and question answering (qa) @xcite , are often the best intermediate tasks for text classification tasks. in a recent study @xcite , it is found that natural language inference and qa tasks are generally helpful as intermediate tasks. @xcite showed that squad-2.0 is the most favourable intermediate task for text classification. there are only a few text classifications tasks and only one language model (bert) in their experiments, making it hard to conclude that squad-2.0 as the intermediate task can generally improve the performance of all types of text classification tasks. in this paper, we investigate the effectiveness of intermediate training for four different lms -electra, roberta, mobilebert, and squeezebert)-using the most popular qa resource squad-2.0 as the intermediate task for eight target text classification tasks. we found that intermediate training shows varying transfer performance across different language models. particularly contrary to previous thoughts, intermediate training with high-level inference qa tasks does not generally show positive transfer for low-level inference text classification tasks. ## related work as a large quantity of labeled data is not always available for training deep learning models, transfer learning becomes quite important for many of nlp problems. with transfer learning, widely available unlabeled text corpora containing rich semantic and syntactic information can be leveraged for learning language models, such as bert @xcite , @xcite , and t5 @xcite . then, these language models are fine-tuned on downstream tasks, which is the dominant transfer learning method adopted in nlp at the moment. the second way of using transfer learning in nlp is to further pre-train pre-trained language models in domain data before fine-tuning on downstream tasks @xcite . the third approach, which is the method we investigate in our work, is to transfer models fine-tuned on an intermediate task for a target task @xcite . a recent work @xcite investigated when and why intermediate-task training is beneficial for a given target task. they experimented with 11 intermediate tasks and 10 target tasks, and find that intermediate tasks re-quiring high-level inference and reasoning abilities tend to work best, such as natural language inference tasks and qa tasks. another recent work @xcite ) has explored transferability across three types of tasks, namely text classification/regression, question answering and sequence labeling. they found that transfer learning is more beneficial for low-data source tasks and also found that data size, task and domain similarity, and task complexity all can affect transferability. ## methods to find out whether using squad-2.0 as the intermediate training task is generally helpful for text classification tasks for different language models, we experiment with 8 single-sequence text classification tasks and 8 sequence-pair text classification tasks, across four language models. in squad-2.0, each question is given a context from which to infer the answer. a qa system is expected to extract a span of text from that given context. more specifically, given a context c which consists of n tokens ([t 1 , t 2 , ...t n ]) and a question q, a qa model is expected to predict the position of the start and end tokens of the answer in the context c. to correctly extract the answer span, on one hand an squad-2.0 model needs to learn word-level dependencies between two sequences (semantic similarity); on the other hand it learns how to infer an answer from the context given a question. training a transformer-based lm for squad-2.0 intuitively enforces model's ability on inference and measuring semantic similarity, which is shown in previous studies @xcite to benefit text classification target tasks at the lower, sequence-level, either classification of single sequences or classification of the inference or similarity for sequence pairs. when using transformer-based models for pairwise text classification, often a special token (e.g., @xcite ) is added between two sequences, similar to the qa input. we are interested in whether such a similarity between qa tasks and sequencepair text classification tasks can make a difference. in terms of training procedure, we follow previous works @xcite . specifically, we first fine-tune a pre-trained lm on squad-2.0 (intermediate training stage) and then fine-tune it on each text classification tasks. after we train a squad-2.0 model, the next step is to transfer it for text classification tasks. when transferring a squad-2.0 model, we only need to change a span classification head to a sequence classification head. the transferred transformer with a new sequence classification head will then be fine-tuned on text classification tasks. the weights of both the transferred squad-2.0 model and the classification head will be updated during the finetuning stage. therefore, the training process consists of three training stages, namely pre-training stage (pre-training a lm), intermediate training stage (fine-tuning on squad-2.0), and fine-tuning stage (fine-tuning on each text classification tasks). ## conclusion we studied using the squad-2.0 qa intermediate task for target text classification across different language models. our experiments on eight classification target tasks and four language models show that squad2-tuned models do not generally have better performance, whether single-sequence or sequence-pair, or data-rich or data-poor settings. this result highlights that high-level inference intermediate tasks may not generally produce positive transfer as previously thought. on the other hand, squad-tuned models always have positive transfer results for qa tasks, which suggests further research is needed to investigate if task similarity rather than task complexity plays a significant role for intermediate training.