Language Modelling as a Multi-Task Problem (2021)

Abstract: In this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability. Based on hypotheses derived from linguistic theory, we investigate whether language models adhere to learning principles of multi-task learning during training. To showcase the idea, we analyse the generalisation behaviour of language models as they learn the linguistic concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a multi-task setting naturally emerges within the objective of the more general task of language modelling. We argue that this insight is valuable for multi-task learning, linguistics and interpretability research and can lead to exciting new findings in all three domains.

## introduction humans are optimising their behaviour towards a multitude of objectives to reach their goals in dayto-day life. by learning many things at the same time and exploiting their commonalities, they acquire more general knowledge about the world, which in turn helps them to learn new things quicker @xcite @xcite . this idea of finding more general solutions through the diversification of tasks has found its way also to the machine learning community, in the field of multitask learning (mtl) @xcite . in mtl, multiple tasks are optimised jointly, enabling the transfer of relevant information across tasks. mtl research yields fruitful results in both application (e.g. @xcite @xcite @xcite and theory (e.g. @xcite @xcite . however, deciding on a setup requires making many arbitrary choices. the researcher or engineer has to decide which tasks to train together (e.g. @xcite ; at which hierarchy-level to allow tasks to interact (e.g. @xcite ; which degree of parameter sharing to employ @xcite ; which distribution of training data to employ (e.g. @xcite , and so on. having to make so many arbitrary choices is inconvenient for modellers, but also stands in the way of understanding the learning principles of neural models in multi-task settings. the highly constructed learning scenarios make it difficult to see whether outcomes should be attributed to one of the many a-priori decisions or to inherent properties of the learning process. in this paper, we propose to study mtl not in a constructed, artificial scenario, but in a more natural setting. to do so, we consider the objective of language modelling and exploit the fact that it can be seen as a conglomerate of many different tasks. to give an example: rules of word ordering have to be learned simultaneously to rules of feature agreement and the monotonicity properties of different linguistic environments. these different tasks all need to be learned to achieve the greater goal of producing acceptable sentences, and they have to be optimised in parallel when the language model (lm) is trained. language modelling is in that sense a natural multi-task learning problem with a naturally given task hierarchy provided by linguistic theory (see also figure 1 ). studying language modelling as a multi-task problem has several distinct advantages. from an mtl perspective, it gives us a complete hierarchy of relevant tasks that can freely interact throughout the learning process, unconstrained by prior assumptions. we can make theoretically informed decisions about these tasks, drawing on linguistic theory. we can also deduce from linguistics how these tasks relate to each other (or, in other words, how similar they are), which in mtl is considered to be one of the crucial factors for the learning outcomes (e.g. @xcite . mtl has not yet been studied from this dynamic and unconstrained perspective. then, somewhat more delicately, the extent to which models can exploit similarities hypothesised by linguistic theory can play a role in confirming or refuting specific linguistic hypotheses. lastly, when it comes to interpretability research, applying concepts from mtl can be valuable to better understand the learning dynamics of models. by understanding how models are finding solutions, we can infer what these solutions are. outline in the remainder of this paper, we will first provide some basic background about mtl ( § 2.1), the subset of linguistic tasks we focus on (negative polarity items, where we consider their different licensing contexts as tasks, § 2.2) and discuss some related work in interpretability ( § 2.3). then, in § 3 and § 4, respectively, we present our approach and empirical results that showcase our idea. in § 5, we discuss our results and framework in the light of the three fields mentioned before. we conclude in § 6. ## background in this paper, we aim to bring together three strands of research: mtl, linguistics and interpretability research. as a proof of concept, we focus on one specific complex subset of linguistic tasks: licensing of negative polarity items (npis). below, we give a short overview of the most important characteristics of the three fields of interest. ## approach we consider two different types of experiments. first, to understand to which extent models can understand and use the similarity between different licensing contexts (our tasks) during learning, we exploit the effect that frequency of the different contexts has on learning. second, we manipulate the lms' training corpus to constrain their ability to leverage information from other licensing contexts during learning. in accordance with the mtl-literature, we expect the lms to learn tasks more data-efficient and to a higher final accuracy if they can leverage information across contexts. before we describe our experiments in more detail, we present our model architecture and training, the evaluation procedure of the licensing contexts, and the filter procedure we use to manipulate the training corpus. ## experiments and results as a first step, we assess whether the lms can adequately represent all nine categories of the evaluation task. to do so, we train five models on the regular training corpus, and compute their final accuracy on our nine tasks. all models show adequate performance on most contexts (see ## discussion in this paper, we studied language modelling as a multi-task problem. we show that neural language models can find and exploit similarity between the different language construction rules that we deduced from linguistic theory and that their transfer behaviour mirrors the generalisation behaviour in traditionally constructed mtl settings. in this section, we now reflect on how our setup and results contribute to the three different areas that we mentioned in the introduction: mtl, linguistics and interpretability research. ## conclusion in the current study we explored the possibility to use multi-task learning as a framework to study learning behaviour within a task. to this end we considered lms as multi-task learners and investigated how they learn the task-cluster of npilicensing. we find that lms pick up on similarities that we assume from linguistic theory and exploit them to learn similar language constructions with less data and to a higher accuracy. especially less frequent tasks benefit from this effect. these results resemble positive transfer in 'traditional' mtl. we lined out the possible benefits that our study may have for mtl research, interpretability and linguistics. from here there are many directions for future work: targeting less comprehensively researched areas in linguistics to add empirical data to otherwise usually theoretical linguistic discussions, investigating the change of internal representations in place of the behavioural measure used here to more precisely describe the learning process, or applying the approach to other high-level tasks in other modalities obeying other knowledge domains are just few of theses possibilities.