  Humans can learn a new language task more efficiently than machines,
conceivably by leveraging their prior experience and knowledge in learning
other tasks. In this paper, we explore whether such cross-task generalization
ability can be acquired, and further applied to build better few-shot learners
across diverse NLP tasks. We introduce CrossFit, a task setup for studying
cross-task few-shot learning ability, which standardizes seen/unseen task
splits, data access during different learning stages, and the evaluation
protocols. In addition, we present NLP Few-shot Gym, a repository of 160
few-shot NLP tasks, covering diverse task categories and applications, and
converted to a unified text-to-text format. Our empirical analysis reveals that
the few-shot learning ability on unseen tasks can be improved via an upstream
learning stage using a set of seen tasks. Additionally, the advantage lasts
into medium-resource scenarios when thousands of training examples are
available. We also observe that selection of upstream learning tasks can
significantly influence few-shot performance on unseen tasks, asking further
analysis on task similarity and transferability.
