Autoregressive language models pretrained on large corpora have been successful at solving  downstream tasks, even with zero-shot usage. However, there is little theoretical justification for their success. This paper considers the following questions: (1) Why should learning the distribution of natural language help with downstream classificationtasks ? (2) Why do features learned using language modelinghelp solve downstream tasks with linear classifiers ? For (1), we hypothesize, and verify empirically, that classification tasks of interest can be reformulated as next word prediction tasks, thus making language modeling a meaningful pretraining task. For (2) , we analyze properties of the cross-entropy objective to show that \epsilon-optimal language models in cross-entropy (log-perplexity) learn features that are \mathcal{O}(\sqrt{\epsilon}) -good on natural linear classification tasks , thus demonstrating mathematically that doing well on language modeling can be beneficial for downstream tasks. We perform experiments to verify assumptions and validate theoretical results. Our theoretical insights motivate a simple alternative to the cross-entropy objective that performs well on some linear classification tasks.
Autoregressive language models , pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot usage. However, there is little theoretical understanding of this success. This paper initiates a mathematical study of this phenomenon for the downstream task of text classification by considering the following questions: (1) What is the intuitive connection between the pretraining task of next word prediction and text classification ? (2) How can we mathematically formalize this connection and quantify the benefit of language modeling ? For (1), we hypothesize, and verify empirically, that classification tasks of interest can be reformulated as sentence completion tasks, thus making language modeling a meaningful pretraining task. With a mathematical formalization of this hypothesis, we make progress towards (2) and show that language models that are \epsilon-optimal in cross-entropy (log-perplexity) learn features that can linearly solve such classification tasks with \mathcal{O}(\sqrt{\epsilon}) error , thus demonstrating that doing well on language modeling can be beneficial for downstream tasks. We experimentally verify various assumptions and theoretical findings, and also use insights from the analysis to design a new objective function that performs well on some classification tasks.
R
pretrained on large corpora
, pretrained using large text corpora to do well on next word prediction,
A
many
R
justification for their
understanding of this
R
considers the
initiates a mathematical study of this phenomenon for the downstream task of text classification by considering the
R
Why should learning the distribution of natural language help with downstream classificationtasks
What is the intuitive connection between the pretraining task of next word prediction and text classification
R
Why do features learned using language modelinghelp solve downstream tasks with linear classifiers
How can we mathematically formalize this connection and quantify the benefit of language modeling
R
next word prediction
sentence completion
R
For
With a mathematical formalization of this hypothesis, we make progress towards
R
, we analyze properties of the cross-entropy objective to show that \epsilon-optimal language models
and show that language models that are \epsilon-optimal
R
are
can linearly solve such classification tasks with
R
-good on natural linear classification tasks
error
D
mathematically
R
perform experiments to verify assumptions and validate theoretical results. Our theoretical insights motivate a simple alternative to the cross-entropy objective
experimentally verify various assumptions and theoretical findings, and also use insights from the analysis to design a new objective function
D
linear