Cross-Task Generalization Abilities of Large Language Models (2024)

Abstract: Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge and experience obtained when learning prior tasks. Enabling similar cross-task generalization abilities in NLP systems is fundamental for approaching the goal of general intelligence and expanding the reach of language technology in the future.In this thesis proposal, I will present my work on (1) benchmarking cross-task generalization abilities with diverse NLP tasks; (2) developing model architectures for improving cross-task generalization abilities; (3) analyzing and predicting the generalization landscape of current state-of-the-art large language models. Additionally, I will outline future research directions, along with preliminary thoughts on addressing them.

## introduction in recent years, large language models (llms) have greatly revolutionized natural language processing research, demonstrating remarkable capabilities in various natural language processing benchmarks @xcite @xcite . as their capabilities have expanded, there has been a corresponding increase in their adoption. llm-powered tools are now playing an essential role in daily activities, from translation and search engines, to personalized chatbots and tutors. looking ahead, we can expect llms to be applied to a wider spectrum of downstream applications with increasing complexity and intricacy. however, building these applications still requires extensive task-specific efforts. this involves data collection, model architecture modifications and training procedure design. even with the most powerful llms, manual selection of in-context examples or prompt engineering is often required to fully unlock their performance. from a practical perspective, these task-specific approaches lack scalability. every new application in the future will demand repeating these tedious and costly processes. from a research perspective, achieving human-level performance on individual tasks through extensive data collection and engineering efforts falls short of the ideal general intelligence. a truly intelligent system should be able to "reuse previously acquired knowledge about a language and adapt to a new task quickly" @xcite . evaluating these systems based on their "skill-acquisition efficiency" @xcite becomes crucial in this context. existing work has approached the problem of learning efficiency by developing better few-shot learning algorithms, e.g., re-formulating tasks into formats that resembles the pre-training objective @xcite . such progress primarily focus on improving instance-level generalization, i.e., how to better generalize from a few labeled instances to make predictions about new instances, within the scope of one individual task. from a broader perspective, human-like learning efficiency also benefits from task-level generaliza-tion, or cross-task generalization (fig. 1 ). humans accumulate their learning experience on previous seen tasks, so that when confronted with a novel task, we are able to grasp the essence of it quickly and learn it efficiently. my research goal is to enable human-like adaptability and learning efficiency in nlp systems. i argue that achieving cross-task generalization is an essential building block for this goal. in the following, i will first revisit the background and prior works ( §2). next, i will introduce my contributions in three areas: (1) benchmarking crosstask generalization with diverse nlp tasks ( §3.1); few-shot in-context learning. in-context learning (icl) is an alternative approach for fewshot learning by simply concatenating the few-shot examples and using them as a prompt before the inference example. popularized by more recent language models like gpt-3 @xcite and palm @xcite , icl allows models to learn from a few examples without any gradient updates and achieve competitive performance. while this approach works well for very large models, smaller models requires meta-training to gain similar capabilities @xcite . our research on cross-task generalization aligns more with the latter approach. however, the former approach remains relevant, as the next-token prediction objective during pre-training can be seen as a superset of language tasks, and icl can be viewed as generalizing to unseen tasks at inference time. ## future directions pushing the limit of in-context learning. as an alternative to model fine-tuning, in-context learning has shown to be effective in adapting an llm to perform novel tasks. existing works on in-context learning mostly focus on conditioning on demonstrations of one single task. it is possible to break this convention by conditioning on diverse and heterogeneous contexts. for example, @xcite ; @xcite highlight the benefits of intermediate task transfer in the fine-tuning paradigm. revisiting this technique with in-context learning may help improve end-task performance and also enhance our understanding of in-context learning. recent progresses on long-context lms open up new opportunities for scaling not only the length, but also the diversity and composition of "contexts" for in-context learning, which we plan to investigate in the future. from data-efficient learners to self-sufficient learners. so far in our efforts, the models are expected to perform few-shot learning when the fewshot training data are provided and fixed. a more ambitious goal will be to build intelligent systems that can acquire their own learning material and learn in the open-endedness. as the capabilities of llms continue to grow, they demonstrate agentic behaviors such as reasoning @xcite , planning @xcite , tool use @xcite , self-refinement @xcite , etc. all of these are also fundamental aspects of human learning processes. in the future, we plan to incorporate these latest advances into building an autonomous, self-sufficient learning agent capable of devising a learning plan, executing it, reflecting on its own limitations, and dynamically adjusting the plan throughout the course of learning.