Massive Choice, Ample Tasks ( M a C h A mp): A Toolkit for Multi-task Learning in NLP (2021)

Abstract: Transfer learning, particularly approaches that combine multi-task learning with pre-trained contextualized embeddings and fine-tuning, have advanced the field of Natural Language Processing tremendously in recent years. In this paper we present MaChAmp, a toolkit for easy fine-tuning of contextualized embeddings in multi-task settings. The benefits of MaChAmp are its flexible configuration options, and the support of a variety of natural language processing tasks in a uniform toolkit, from text classification and sequence labeling to dependency parsing, masked language modeling, and text generation.

## introduction multi-task learning (mtl) @xcite has developed into a standard repertoire in natural language processing (nlp). it enables neural networks to learn tasks in parallel @xcite while leveraging the benefits of sharing parameters. the shift-or "tsunami" (manning, 2015)-of deep learning in nlp has facilitated the wide-spread use of mtl since the seminal work by @xcite , which has led to a multi-task learning "wave" @xcite in nlp. it has since been applied to a wide range of nlp tasks, developing into a viable alternative to classical pipeline approaches. this includes early adoption in recurrent neural network models, e.g. @xcite @xcite @xcite , to the use of large pre-trained language models with multi-task objectives @xcite . mtl comes in many flavors, based on the type of sharing, the weighting of losses, and the design and relations of tasks and layers. in general though, outperforming single-task settings remains a challenge @xcite . for an overview of mtl in nlp we refer to @xcite . as a separate line of research, the idea of language model pre-training and contextual embeddings @xcite @xcite is to pre-train rich representation on large quantities of monolingual or multilingual text data. taking these representations as a starting point has led to enormous improvements across a wide variety of nlp problems. related to mtl, recent research effort focuses on fine-tuning contextualized embeddings on a variety of tasks with supervised objectives @xcite @xcite . we introduce machamp, a flexible toolkit for multi-task learning and fine-tuning of nlp problems. the main advantages of machamp are: as a result, the flexibility of machamp supports up-to-date, general-purpose nlp (see section 2.2). the backbone of machamp is al-lennlp @xcite , a pytorch-based @xcite python library containing modules for a variety of deep learning methods and nlp tasks. it is designed to be modular, highlevel and flexible. it should be noted that contemporary to machamp, jiant @xcite was developed, and allennlp included multi-task learning as well since release 2.0. machamp distinguishes from the other toolkits by supporting simple configurations, and a variety of multi-task settings. ## model in this section we will discuss the model, its supported tasks, and possible configuration settings. ## usage to use machamp, one needs a configuration file, input data and a command to start the training or prediction. in this section we will describe each of these requirements. ## hyperparameter tuning in this section we describe the procedure how we determined robust default parameters for machamp; note that the goal is not to beat the state-of-the-art, but to reach competitive performance for multiple tasks simultaneously. 7 for the tuning of hyperparameters, we used the glue classification datasets @xcite @xcite @xcite @xcite @xcite and the english web treebank (ewt 2.6) @xcite with multilingual bert foot_6 (mbert) as embeddings. 9 for each of these setups, we averaged the scores over all datasets/tasks and perform a grid search. the best hyperparameters across all datasets are reported in ## conclusion we introduced machamp, a powerful toolkit for multi-task learning supporting a wide range of nlp tasks. we also provide initial experiments demonstrating the usefulness of some of its options. we learned that multi-task learning is mostly beneficial for setups in which multiple datasets are annotated for the same set of tasks, and that dataset embeddings can still be useful when employing contextualized embeddings. however, the current experiments are just scratching the surface of machamp's capabilities, as a wide variety of tasks and multi-task settings is supported.