C ross F it: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021)

Abstract: Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

## introduction pre-trained language models fine-tuned with abundant task-specific data have become the predominant recipe for state-of-the-art results in nlp. however, these approaches are heavily dependent on large-scale labeled datasets that are expensive to create, and the resulting models still generalize poorly to out-of-distribution inputs created with small, harmless perturbations @xcite . in retrospect, researchers have advocated for building more human-like, general linguistic intelligence that can "reuse previously acquired knowledge about a language and adapt to a new task quickly" @xcite ). 1 our code is at https://github.com/ink-usc/crossfit . others (regression, etc.) nlp few-shot gym a repository of 160 diverse few-shot tasks in nlp stage 1. upstream learning (multitask learning, meta-learning, etc.) text-to-text transformer text-to-text transformer stage 2. downstream fine-tuning training tasks (e.g., non-cls. tasks) test tasks (e.g., classification) classification question asnwering conditional generation the cr ossfit challenge can we build a few-shot learner to generalize beyond task boundaires? figure 1 : we present the crossfit challenge to study cross-task generalization in a diverse task distribution. to support this problem setting, we introduce the nlp few-shot gym, a repository of 160 diverse few-shot, text-to-text tasks in nlp. existing work has approached this problem via better few-shot fine-tuning, by re-formulating target tasks into cloze questions that resembles the pretraining objective @xcite , generating prompts and using demonstrations @xcite . such progress primarily focus on improving instance-level generalization, i.e., how to better generalize from few labeled instances to make predictions about new instances, within the scope of one individual task. from a broader perspective, human-like learning ability also benefits from task-level generalization, or cross-task generalization, i.e., how to learn a new task efficiently given experiences of learning previous tasks. such ability has been widely studied in computer vision and robotics community @xcite , but is relatively underexplored in nlp. @xcite and @xcite study transferability between one intermediate task and a given target task, while it's possible to further improve performance with multiple intermediate tasks. @xcite and @xcite focus on cross-task generalization within the scope of classification tasks, whereas hu-mans can generalize across different task formats (classification, multiple choice, generation, etc.), goals (question answering, fact checking, etc.) and domains @xcite . towards developing general linguistic intelligence, we present crossfit, a few-shot learning challenge to acquire, evaluate and analyze crosstask generalization in a realistic setting, with standardized training pipeline, data access and evaluation protocol. the crossfit challenge requires a model to first learn from a set of seen tasks in an upstream learning stage, and then perform fewshot learning on a set of unseen tasks, as illustrated in fig. 1 . in accompany, we introduce the nlp few-shot gym, a repository of 160 few-shot nlp tasks gathered from open-access resources, covering a wide range of capabilities and goals, and formulated into a unified text-to-text format. to analyze the capability and limitation of existing approaches to the crossfit challenge, we design eight specific seen/unseen task partitions. with the crossfit challenge and the nlp fewshot gym, we aim to investigate the following research questions: to address the above questions, we empirically analyze the performance of multi-task learning and three meta-learning algorithms (maml @xcite , first-order maml and reptile @xcite ). we observe that these approaches can indeed lead to better few-shot performance on unseen tasks. interestingly, simple multi-task learning outperforms existing meta-learning methods in many cases, encouraging future research on identifying the reasons and developing improved meta-learning methods. for q2, we observe that performance of individual unseen tasks varies with different selection of seen tasks, calling for more thorough investigation of the relationship between task similarity and transferability. as for q3, we find that enlarging the size of upstream data does not necessitate better cross-task generalization abilities. we envision cross-task generalization to be an integral component towards general linguistic intelligence, and we hope crossfit serves as a useful testbed for driving related progress. ## the crossfit challenge in this section, we present the crossfit challenge, a problem setting for acquiring and evaluating cross-task generalization. ideally, a strong crossfit system can capture cross-task generalization ability from a set of seen tasks and thus adapts to new unseen tasks efficiently. ## nlp few-shot gym towards learning to generalize across tasks in crossfit challenge, we need a resource that contains sufficient number of tasks, covering a wide range of nlp applications, and presented in a unified text-to-text format. herein, we introduce the nlp few-shot gym, a repository of 160 few-shot tasks gathered from existing open-access datasets. ## methods to crossfit we mainly use bart-base @xcite as the text-to-text transformer for our analysis in the crossfit setup. we leave confirmatory experiments with t5-v1.1-base and bart-large model in appendix c. ## empirical analysis in this section we look to interpret the results and answer our research questions. we summarize the arg scores in q1. can we teach pre-trained lms to generalize across tasks with existing methods? overall performance. from correlated performance gains. the performance gain obtained with different upstream learning methods are correlated with each otheri.e., tasks that benefit from multi-task learning is likely to also benefit from meta-learning. for the random partition, the spearman correlation between the relative improvement brought by mtl and maml is 0.66, with p value equals to 0.0015. this suggests that different upstream learning methods, while taking different optimization objectives, capture similar inductive bias from t train . mtl is a strong baseline. surprisingly, the most straight-forward multi-task learning method is hard to beat. this could be counter-intuitive, as meta-learning methods are specifically designed for rapid generalization to unseen tasks, sharing the same goal with our crossfit challenge. we think there are three possible reasons: (1) due to memory constraints, we limit the number of innerloop updates to be one, which may be insufficient. also, meta-learning methods are highly sensitive to hyper-parameters and even random seeds (antoniou et al., 2019), which we do not tune exhaustively for practical reasons. ## conclusion and future work in this paper, we study the problem of building better few-shot learners via acquiring cross-task generalization ability from diverse nlp tasks. towards our goal, we introduce the crossfit challenge, an task setup that standardizes the training pipeline, data access and evaluation protocol. we also present the nlp few-shot gym, a repository of 160 diverse few-shot nlp tasks, to support crossfit learning in different scenarios. we empirically demonstrated that cross-task generalization can be acquired via multi-task learning and meta-learning; confirmed that the selection of seen tasks would influence the few-shot performance on unseen tasks. we have highlighted several unexpected or undesired observations in our analysis, for which we invite future work in understanding and combating related issues. in addition, we envision the crossfit challenge and the nlp few-shot gym to serve as the testbed for many interesting "meta-problems", such as (1) learning to generate prompt for diverse task formats and further improve learning efficiency @xcite ; (2) learning to select appropriate source tasks to learn from during upstream learning @xcite , potentially with task2vec methods @xcite ; (3) applying task augmentation strategies to prevent over-fitting @xcite ; (4) learning to accumulate knowledge and avoid catastrophic forgetting in an continual learning setup @xcite ; (5) decomposing complex tasks into atomic tasks and exploring cross-task generalization through the lens of compositionality @xcite . engineering, doceng '11, page 259-262, new york, ny, usa. association for computing machinery. aida amini, saadia gabriel, shanchuan lin, rik koncel-kedziorski, yejin choi, and hannaneh hajishirzi. 2019. mathqa: towards interpretable math word problem solving with operation-based formalisms. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 2357-2367, minneapolis, minnesota. association for computational linguistics. jacob andreas, marcus rohrbach, trevor darrell, and dan klein. 2016. learning to compose neural networks for question answering. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1545-1554, san diego, california. association for computational linguistics. antreas antoniou, harrison edwards, and amos storkey. 2019. how to train your maml. in international conference on learning representations. trapit bansal, rishikesh jha, and andrew mccallum. 2020a. learning to few-shot learn across diverse natural language classification tasks. in proceedings of the 28th international conference on computational linguistics, pages 5108-5123, barcelona, spain (online). international committee on computational linguistics. trapit bansal, rishikesh jha, tsendsuren munkhdalai, and andrew mccallum. 2020b. self-supervised meta-learning for few-shot natural language classification tasks. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 522-534, online. association for computational linguistics. roy bar-haim, ido dagan, bill dolan, lisa ferro, danilo giampiccolo, bernardo magnini, and idan szpektor. 2006. the second pascal recognising textual entailment challenge. in proceedings of the second pascal challenges workshop on recognising textual entailment, volume 6, pages 6-4. venice. francesco barbieri, jose camacho-collados, luis espinosa anke, and leonardo neves. 2020. tweeteval: unified benchmark and comparative evaluation for tweet classification. in findings of the association for computational linguistics: emnlp 2020, pages 1644-1650, online. association for computational linguistics. max bartolo, alastair roberts, johannes welbl, sebastian riedel, and pontus stenetorp. 2020. beat the ai: investigating adversarial human annotation for reading comprehension. transactions of the association for computational linguistics, 8:662-678. luisa bentivogli, peter clark, ido dagan, and danilo giampiccolo. 2009. the fifth pascal recognizing textual entailment challenge. in tac. jonathan berant, andrew chou, roy frostig, and percy liang. 2013. semantic parsing on freebase from question-answer pairs. in proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544, seattle, washington, usa. association for computational linguistics. chandra bhagavatula, ronan le bras, chaitanya malaviya, keisuke sakaguchi, ari holtzman, hannah rashkin, doug downey, wen tau yih, and yejin choi. 2020. abductive commonsense reasoning. in international conference on learning representations. yonatan bisk, rowan zellers, ronan le bras, jianfeng gao, and yejin choi. 2020. piqa: reasoning about physical commonsense in natural language. in thirty-fourth aaai conference on artificial intelligence. michael boratko, xiang li, tim o'gorman, rajarshi das, dan le, and andrew mccallum. 2020. pro-toqa: a question answering dataset for prototypical common-sense reasoning. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 1122-1136, online. association for computational linguistics. jan a. botha, manaal faruqui, john alex, jason baldridge, and dipanjan das. 2018. learning to split and rephrase from wikipedia edit history. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 732-737, brussels, belgium. association for computational linguistics. jonathan bragg, arman cohan, kyle lo, and iz beltagy. 2021. flex: unifying evaluation for few-shot nlp. tom b brown, benjamin mann, nick ryder, melanie subbiah, jared kaplan, prafulla dhariwal, arvind neelakantan, pranav shyam, girish sastry, amanda askell, et al. 2020. language models are few-shot learners. arxiv preprint arxiv:2005.14165. ankush chatterjee, kedhar nath narahari, meghana joshi, and puneet agrawal. 2019. semeval-2019 task 3: emocontext contextual emotion detection in text. in proceedings of the 13th international workshop on semantic evaluation, pages 39-48, minneapolis, minnesota, usa. association for computational linguistics. anthony chen, gabriel stanovsky, sameer singh, and matt gardner. 2020a. mocha: a dataset for training and evaluating generative reading comprehension metrics. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 6521-6532, online. association for computational linguistics. annual meeting of the association for computational linguistics, pages 1074-1084, florence, italy. association for computational linguistics. angela fan, yacine jernite, ethan perez, david grangier, jason weston, and michael auli. 2019. eli5: long form question answering. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3558-3567, florence, italy. association for computational linguistics. manaal faruqui and dipanjan das. 2018. identifying well-formed natural language questions. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 798-803, brussels, belgium. association for computational linguistics. chelsea finn, pieter abbeel, and sergey levine. 2017. model-agnostic meta-learning for fast adaptation of deep networks. in proceedings of the 34th international conference on machine learning, volume 70 of proceedings of machine learning research, pages 1126-1135. pmlr. tianyu gao, a. fisch, and danqi chen. 2020. making pre-trained language models better few-shot learners. arxiv, abs/2012.15723. tianyu gao, xu han, hao zhu, zhiyuan liu, peng li, maosong sun, and jie zhou. 2019. fewrel 2.0: towards more challenging few-shot relation classification. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6250-6255, hong kong, china. association for computational linguistics. danilo giampiccolo, bernardo magnini, ido dagan, and bill dolan. 2007. the third pascal recognizing textual entailment challenge. in proceedings of the acl-pascal workshop on textual entailment and paraphrasing, pages 1-9. association for computational linguistics. ona de gibert, naiara perez, aitor garcía-pablos, and montse cuadros. 2018. hate speech dataset from a white supremacy forum. in proceedings of the 2nd workshop on abusive language online (alw2), pages 11-20, brussels, belgium. association for computational linguistics. bogdan gliwa, iwona mochol, maciej biesek, and aleksander wawer. 2019. samsum corpus: a human-annotated dialogue dataset for abstractive summarization. in proceedings of the 2nd workshop on new frontiers in summarization, pages 70-79, hong kong, china. association for computational linguistics. andrew gordon, zornitsa kozareva, and melissa roemmele. 2012. semeval-2012 task 7: choice of plausible alternatives: an evaluation of commonsense causal reasoning. in *sem 2012: the first joint conference on lexical and computational semantics -volume 1: proceedings of the main conference and the shared task, and volume 2: proceedings of the sixth international workshop on semantic evaluation (semeval 2012), pages 394-398, montréal, canada. association for computational linguistics. jiatao gu, yong wang, yun chen, victor o. k. li, and kyunghyun cho. 2018. meta-learning for lowresource neural machine translation. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 3622-3631, brussels, belgium. association for computational linguistics. harsha gurulingappa, abdul mateen rajput, angus roberts, juliane fluck, martin hofmann-apitius, and luca toldo. 2012. development of a benchmark corpus to support the automatic extraction of drugrelated adverse effects from medical case reports. journal of biomedical informatics, 45(5):885-892. text mining and natural language processing in pharmacogenomics. xu han, hao zhu, pengfei yu, ziyun wang, yuan yao, zhiyuan liu, and maosong sun. 2018. fewrel: a large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4803-4809, brussels, belgium. association for computational linguistics. luheng he, mike lewis, and luke zettlemoyer. 2015. question-answer driven semantic role labeling: using natural language to annotate natural language. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 643-653, lisbon, portugal. association for computational linguistics. johannes hoffart, mohamed amir yosef, ilaria bordino, hagen fürstenau, manfred pinkal, marc spaniol, bilyana taneva, stefan thater, and gerhard weikum. 2011. robust disambiguation of named entities in text. in proceedings of the 2011 conference on empirical methods in natural language processing, pages 782-792, edinburgh, scotland, uk. association for computational linguistics. eduard hovy, laurie gerber, ulf hermjakob, chin-yew lin, and deepak ravichandran. 2001. toward semantics-based answer pinpointing. in proceedings of the first international conference on human language technology research. lifu huang, ronan le bras, chandra bhagavatula, and yejin choi. 2019. cosmos qa: machine reading comprehension with contextual commonsense reasoning. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2391-2401, hong kong, china. association for computational linguistics.