M a C h A mp at S em E val-2022 Tasks 2, 3, 4, 6, 10, 11, and 12: Multi-task Multi-lingual Learning for a Pre-selected Set of Semantic Datasets (2022)

Abstract: Previous work on multi-task learning in Natural Language Processing (NLP) oftenincorporated carefully selected tasks as well as carefully tuning ofarchitectures to share information across tasks. Recently, it has shown thatfor autoregressive language models, a multi-task second pre-training step on awide variety of NLP tasks leads to a set of parameters that more easily adaptfor other NLP tasks. In this paper, we examine whether a similar setup can beused in autoencoder language models using a restricted set of semanticallyoriented NLP tasks, namely all SemEval 2022 tasks that are annotated at theword, sentence or paragraph level. We first evaluate a multi-task model trainedon all SemEval 2022 tasks that contain annotation on the word, sentence orparagraph level (7 tasks, 11 sub-tasks), and then evaluate whetherre-finetuning the resulting model for each task specificially leads to furtherimprovements. Our results show that our mono-task baseline, our multi-taskmodel and our re-finetuned multi-task model each outperform the other modelsfor a subset of the tasks. Overall, huge gains can be observed by doingmulti-task learning: for three tasks we observe an error reduction of more than40%.

## introduction recently, language models have become the defacto standard in natural language processing (nlp), where we first train a set of parameters on raw data, which are then finetuned on the task at hand. this in itself is a multi-task setup (languagemodeling + target task). however, traditionally, multi-task learning was mainly done between multiple nlp tasks with gold annotation. in this setup, many questions arise: not only how to share the information between different tasks, but also when to share and even which tasks to use, as it is non-trivial to decide which auxiliary tasks are beneficial for a certain target task @xcite . early work on multi-task learning in nlp often used up to a handful of tasks, carefully curated dataset/task combinations, and carefully tuned how to share the information between these tasks (e.g. @xcite ; @xcite ). a line of recent work has shown that an intermediate step can be used to finetune the language model on a set of nlp tasks, which leads to a model that is more apt for learning other nlp tasks. this is also called supplementary training on intermediate labeled-data tasks (stilt), and was introduced by @xcite . @xcite train on three classification tasks from the glue benchmark @xcite , and then retrain for all glue tasks, showing a 1.4 point of improvement over all glue tasks. @xcite shows that this positive transfer also holds cross-lingually when using multilingual language models and only doing intermediate training on english tasks. @xcite . similar as with earlier models, it remains an open question for stilt models which tasks transfer well to which tasks @xcite @xcite . it should be noted that most work on stils for autoencoder language models is done on text (i.e. sentence) classification only. later work used autoregressive language models, which learn to generate texts (as opposed to the autoencoding models, which learn to predict one token at a time, used by the previously mentioned stilt papers). these language models are commonly used for different types of tasks, namely generation tasks (e.g. question answering, machine translation, summarization), whereas autoencoding models are commonly used for classification and word-level tasks (text classification, pos-tagging, parsing etc.). recent work has shown that many nlp tasks can be converted to generation tasks, and semeval task included sub-tasks languages citation 2: multilingual idiomaticity detection idiomaticity detection (1-shot) en, pt, gl tayyar madabushi et al. (2022, 2021) 3: pretens 1: binary acceptability en, it, fr zamparelli et al. (2022) 2: regression acceptability en, it, fr 4: patronizing and condescending language detection 1: binary pcl detection en pérez-almendros et al. (2022); perez almendros et al. (2020) 2: multi-label pcl classification en 6: isarcasmeval 1: sarcasm detection en, ar abu farha et al. (2022) 2: irony-labeling en 3: paraphrase sarcasm detection en, ar 10: structured sentiment analysis expressions, entities and relations ca, en, es, eu, no barnes et al. (2022) 11: multiconer -multilingual complex named entity recognition named entity recognition bn, de, en, es, fa, hi, ko, mi, nl, ru, tr, zh malmasi et al. (2022) 12: symlink entities and relations en dac lai et al. (2022) in this paper, we set out to examine whether we can obtain performance improvements with multi-task learning and re-finetuning after multitask learning (i.e. stilt) for a pre-defined set of semantically focused nlp tasks. more precisely, we will use the pre-defined set of semeval 2022 tasks, and train a multi-task model for all text-based semeval tasks that include annotation on the word, sentence or paragraph level. 2 we compare a strong single task baseline to a default multi-task learning model, where the encoder is shared, each task has its own decoder, and training is done on all tasks simultaneously (shuffled batches). finally, we use the parameters from the multi-task model to train a task-specific model for each task again. we seek to answer the following research question: ## datasets ## model we implemented all of our models in machamp v0.3beta @xcite . machamp is a toolkit that focuses on multi-task learning for nlp task-types based on allennlp @xcite and the transformers library @xcite . it supports a wide variety of tasks, and a variety of options for multi-task learning (for within as well as cross-dataset multi-task learning). a typical machamp model consists of a shared encoder (i.e. language model), with multiple decoders on top (one for each task), which all share the same encoder. we use all default hyperparameters of machamp for our experiments, except for the dataset smoothing (section 3.8). our general setup is shown in figure 1 . as baseline, we take the data of a single semeval task, and finetune a machamp model with all default hyperparameters (single). the first multi-task setup, is a machamp model trained on a combination of all semeval tasks we consider (multi), where each task has its own decoder. finally we take the hyperparameters from the multi model, and refinetune them for a single task at a time (multi-fine). for the relation extraction tasks (task 10 and 12), we first converted the data to a word-level sequence labeling task, and we contributed a regression tasktype in machamp, to be able to tackle task 3-2. for all sub-tasks with multiple languages/datasets, we evaluate also whether learning these in separate decoders is useful (so we split the datasets, and learn them as separate tasks). below, we describe the choices we made for each of the tasks (the machamp task-types can be found in ## results for all the tasks where the shared task organizers released an evaluation script, we used the official script for the results reported in this section (for the model-picking we used internal equivalent metrics, see section 3 for the details per task); for task11 we used conlleval.pl, and for task12 we used an average of accuracy over our converted data. ## conclusion we have compared three setups in this work: sin-gle: single task finetuning of language models, multi: multi-task finetuning of language models, multi_fine: using the output of multi and finetuning on single target tasks again. our setup is both multi-lingual and uses pre-defined set of tasks with a large variety in types of tasks. our results confirm the findings of recent and concurrent work @xcite , showing that for some task combinations, we can benefit from an intermediate task-trained model (multi_fine). however, we also show that all three evaluated setups perform well for certain tasks. we hypothesize that this is an effect of using a pre-defined set of tasks. in our setup the differences between the setups are in some cases extremely large (error reductions larger than 40% compared to the single task baseline have been obtained for three tasks), whereas for some other tasks our single task baseline performed best. this leads to a positive answer to our research question, and the conlusion that intermediate finetuning can be beneficial. however, care should be taken, as our results also show that multi_fine does not outperform multi nor single in all situations, which raises the question: how can we predict whether the intermediate model is better or we need to finetune one more time on the target task?