M a C h A mp at S em E val-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets. (2023)

Abstract: To improve the ability of language models to handle Natural Language Processing(NLP) tasks and intermediate step of pre-training has recently beenintroduced. In this setup, one takes a pre-trained language model, trains it ona (set of) NLP dataset(s), and then finetunes it for a target task. It isknown that the selection of relevant transfer tasks is important, but recentlysome work has shown substantial performance gains by doing intermediatetraining on a very large set of datasets. Most previous work uses generativelanguage models or only focuses on one or a couple of tasks and uses acarefully curated setup. We compare intermediate training with one or manytasks in a setup where the choice of datasets is more arbitrary; we use allSemEval 2023 text-based tasks. We reach performance improvements for most taskswhen using intermediate training. Gains are higher when doing intermediatetraining on single tasks than all tasks if the right transfer taskis identified. Dataset smoothing and heterogeneous batching did not lead torobust gains in our setup.

## introduction the introduction of word embeddings and later contextualized transformer-based models, i.e. large language models (llm), have led to performance improvements on many natural language processing tasks. technically these approaches are multi-task learning approaches (with parameter sharing over time), where we first train a language model on raw data, save the weights, and then retrain the weights for our target task. it has been shown that intermediate steps of training can be beneficial for downstream performance. this intermediate step of training can be done through language modeling to adapt to new domains or languages (gururangan et al., 2020; muller et al., 1 code available at: https://bitbucket.org/ robvanderg/semeval2023/ 2021) or on nlp datasets directly @xcite , exploiting the synergies across tasks. intermediate training on nlp datasets has benefits both for performance and efficiency (retraining on tasks is cheaper compared to training a full language model). one important desideratum in finding the right language model for a downstream task is the training data and its distance to the target data. for an intermediate training step on nlp datasets this is even more complex, as there are more design decisions to make; e.g.: which nlp tasks are relevant for the target task? which datasets are closest to the target data? how many (or how much) of them do we use? which llm to use as a starting point? previous work has investigated mostly single dimensions in this choice of intermediate training datasets (in a carefully curated setup), less is known about what to do if one has a more varied set of nlp tasks. in this paper, we use all text-based tasks of sem-eval 2023 (tasks 2-12) as a seemingly arbitrary set of nlp tasks for evaluating the effect of intermediate training. more concretely, we seek to answer: we test all of these under a setting of highly varied datasets, in different languages, with different tasks, different task types (single labels, sequences, etc.), different training sizes, and predictions over different input lengths (word, sentence, document). ## intermediate training an intermediate step of language model training has shown to be beneficial to adapt a language model to a new domain, like social media or biomedical data @xcite as well as to new languages @xcite . here, the intuition is to repurpose the knowledge of previously trained models, and specialize them towards the target domain and/or language by doing further finetuning on a language modeling objective. @xcite showed that an intermediate training step on an nlp tasks can also be beneficial for performance of llm's on the glue datasets, which are all on the sentence level. follow-up work attempted to identify how datasets can be selected for the intermediate training step. @xcite find that language modeling is hard to beat as intermediate task, and that multitask pretraining outperforms single-task pre-training. they also included sequence-to-sequence tasks, and conclude that these are too distant to be beneficial for classification tasks. correlations on performance of dataset pairs are low showing that it is hard to predict which datasets are beneficial. this is confirmed by the findings of @xcite , who conclude that task complexity is not a good predictor for being a good transfer dataset, whereas @xcite find the opposite (complex tasks are good to transfer from), although they do conclude that future work is necessary. @xcite compare intermediate training versus joint training on sentence-level tasks, concluding that for small datasets, joint training is more beneficial, and for larger datasets intermediate training should be used. @xcite evaluate a variety of (supervised) approaches to automatically identify which source datasets are beneficial, and conclude that pre-computable sentence representations are efficient for this task, and confirm that within task-type transfer outperforms cross-task-type transfer which is in line with the findings of @xcite who compare transfer across and within task types. instead of selecting tasks, recent work has attempted to train on a wide variety of tasks. this is commonly done in the space of generative language models, where training on a variety of tasks is easier because many nlp tasks can be converted to generation tasks, and can then directly be used to (re-)train an autoregressive language model @xcite @xcite . in this setup it is easier to exploit a large variety of task types and a much higher amount of datasets (~50-1,800 datasets) is used compared to previous work. however, autoregressive language models still lag behind in performance compared to autoencoder language models for many tasks. work that focused on intermediate training for encoder models and a large variety of task types is mainly done in the biomedical domain @xcite . compared to previous work on autoencoder models, we have a larger variety of tasks as well as languages. our selection of datasets is somewhat arbitrary (all semeval 2023 tasks), leading to a more challenging setup for multi-task learning compared to previous work who usually used a carefully curated set of datasets (with often 1 language and/or task type). the most similar to our setup is van der goot (2022), as they also use an arbitrary set of tasks (all semeval 2022 textbased tasks) and also compare joint training with intermediate training on the full collection of the data. however, their results show no clear trend on when one approach outperforms the other. we build on this work by considering also single-task intermediate training, and systematically analyzing the differences in performance based on properties of the datasets. furthermore, we evaluate the effect of diverse batching and dataset smoothing. ## setup we first find the best strategy within each dataset (section 4); we refer to these models as we use machamp (van der goot, 2022) v0.4 with default hyperparameters for all experiments, with the bert-base-multilingual-cased language model for efficiency reasons. for our final test submissions, we compare 7 language models on the best strategy for each task. we used a pre-selection of multilingual language models based on previous empirical results achieved by machamp. foot_0 . the list of language models and their results can be found in appendix c. ## data and baselines for each task, we describe an overview of the task, the baseline approaches we evaluated, and the results. an annotated example for each task can be found in the internal implementation of these metrics in machamp for model selection when available. we use the scikit-learn implementation for f1 scores and conlleval.pl for spans for the scores reported in this paper. for tasks without publicly available dev data (3,7,8,9,10, and 12), we use 80% for train and 20% for dev. task 6 is described in appendix b, as we did not manage to officially participate. ## results the scores on the development data for all our setups (figure 3 ) show that the single-task intermediate training is most successful (highest for 12/21). it should be noted, however, that we only plot the best single transfer task, which does need to be found first. the multi-task setup only performs best for one task (8-1), this is because it is the only model that has to share parameters in the model that is used for prediction. for most tasks, the gains obtained with intermediate training are substantial, but there are five cases where the single model performs best; two of these are the soft labels of task 11. figure 3 : absolute performance (official metrics) of four settings; the single dataset models, the multi-task models that are trained on all tasks, and the retrained models that are trained on a single (we show the best one only), or on all tasks. for task 11 we use 1/cross_entropy, so higher is better for all tasks. though we rank high for 3 tasks (4-2, 8-1, and 8-2), for most tasks, we rank somewhere in the middle. this was expected, as we do almost no tuning per task, no ensembling, use no domain/languagespecific embeddings and do not use any non-semeval data nor data augmentation. ## analysis we plot the difference of each retraining setup to the single task baseline in figure 4 ; positive numbers mean the baseline was outperformed. results show that multi-task learning and intermediate training leads to performance loss more often than performance gains in our setup, which is probably a result of having large training sets and good performing baselines. in general, some datasets are particularly good at being on the receiving end of transfer, whereas there is no clear dataset to transfer from (rows are more consistently colored than columns). tasks that particularly benefit from intermediate training are tasks 6-3, 10, 11-1, and 11-2. for task 6-3, we have a very low baseline, and results seem unstable, task 10 gains from almost all single task transfers, but mostly from itself (note that we just train twice on the same data here). the performance of the models trained on all datasets is not more robust compared to the singletask source models, although scaling up further might change this effect @xcite . diverse batching as well as dataset smoothing only lead to performance improvements in specific cases in our setup. we do not find improvements similar to @xcite for diverse batching, which is probably either because they use the same decoder for all tasks (in a generative model), or because of the scale of datasets. to evaluate which properties are important for dataset selection in multi-task learning we perform a correlation study. we use the single dataset transfer results (differences to baseline on dev, i.e. first 21 columns of figure 4 ) as target variable, and evaluate the correlation against a variety of properties of the tasks. results ( target task -0.3 -0.2 -0.1 -76.0 -0.8 -2.2 -0.5 -0.2 -0.7 -1.9 -0.6 -1.1 -0.0 -1.0 -0.7 -0.9 -0.5 -0.4 0.1 -0.1 -0.2 -2.7 -10.7 -2.3 -3.1 -3.6 -16.4 -2.7 -4.3 1.0 -3.7 -6.5 -56.6 -4.1 -6.3 -0.8 -8.3 -4.1 -29.1 -3.0 2.1 -8.9 -6.3 -8.9 -0.6 -2.2 -5.1 -4.2 0.2 -4.2 -7.4 -15.5 -8.9 -3.5 -14.1 -24.8 -24.3 -21.0 -9.0 -20.0 -8.8 -51.8 -11.2 -20.2 -9.1 -14.4 -10.0 -9.4 -11.1 -20.4 -8.1 -14.8 -7.8 -14.2 -13.5 -11.2 -9.9 -10.1 -11.8 -0.8 -5.9 -5.0 -5.2 -2. to gauge the effect of simply upscaling the number of datasets, we also trained models on the combination of the data in section 4 and the semeval 2022 datasets used by @xcite . this leads to approximately twice as large training data, but no robust gains in performance; for some tasks the combined data increases performance for others it does not (results in appendix d). although the scale of data does not match recent studies in the space of generative language models @xcite , our results do suggest that scaling up might not be the most prominent direction for obtaining robust multi-task autoencoder models. ## conclusion we evaluate the effect of multi-task learning and intermediate training on all text-based semeval 2023 tasks and showed that for a non-curated set of benchmarks it is hard to obtain consistent improvements. we found that specific target tasks are likely to gain from intermediate training, but finding the right source dataset/task is non-trivial. training on a wide variety of tasks has shown to not be more robust compared to finding the best source task to transfer from. in contrast to previous work, having task-heterogeneous batches did not lead to consistent performance gains in our setup.