  In this paper, we propose to study language modelling as a multi-task
problem, bringing together three strands of research: multi-task learning,
linguistics, and interpretability. Based on hypotheses derived from linguistic
theory, we investigate whether language models adhere to learning principles of
multi-task learning during training. To showcase the idea, we analyse the
generalisation behaviour of language models as they learn the linguistic
concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a
multi-task setting naturally emerges within the objective of the more general
task of language modelling.We argue that this insight is valuable for
multi-task learning, linguistics and interpretability research and can lead to
exciting new findings in all three domains.
