Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models (2022)

Abstract: How to usefully encode compositional task structure has long been a core challenge in AI. Recent work in chain of thought prompting has shown that for very large neural language models (LMs), explicitly demonstrating the inferential steps involved in a target task may improve performance over end-to-end learning that focuses on the target task alone. However, chain of thought prompting has significant limitations due to its dependency on huge pretrained LMs. In this work, we present compositional fine-tuning (CFT): an approach based on explicitly decomposing a target task into component tasks, and then fine-tuning smaller LMs on a curriculum of such component tasks. We apply CFT to recommendation tasks in two domains, world travel and local dining, as well as a previously studied inferential task (sports understanding). We show that CFT outperforms end-to-end learning even with equal amounts of data, and gets consistently better as more component tasks are modeled via fine-tuning. Compared with chain of thought prompting, CFT performs at least as well using LMs only 7.4% of the size, and is moreover applicable to task domains for which data are not available during pretraining.

## introduction philosophy, linguistics, and computer science have long debated how and whether to explicitly encode the compositionality of task structure in models of language understanding and generation @xcite . the prevailing paradigm in today's nlp is end-to-end learning, in which the learning of compositional task structure is subsumed by the learning of a complex target task, with the support of increasingly powerful language models (lms) @xcite @xcite . recent work in compositionality in nlp has been mostly limited to semantic parsing and multihop reasoning for the purpose of q&a @xcite , 2021; @xcite . however, a series of recent works have proposed generating "chains of thought" as a means to expand an lm's ability to reason beyond a single forward pass @xcite @xcite . the success of chain of thought approaches suggests broader opportunities to study the use of compositional structure as a means to improve the learning of complex tasks, rather than as a byproduct of end-to-end learning. breaking down a complex task into sub-tasks is a ubiquitous construct in human problem-solving. in machine learning, it has inspired curriculum learning (cl) @xcite , which hypothesizes that a model should start learning from easier concepts and progress to harder ones, as humans do. in this work, we explore the idea of cl through the lens of incremental task complexity, which is fundamentally different from prior works in nlp centered on incremental example difficulty (e.g., organizing training data by increasing sequence length or decreasing word frequency). we propose compositional fine-tuning (cft), a fine-tuning strategy in which sub-tasks are organized as components of a curriculum that progressively teaches a target task, as shown visually in figure 1 . cft is novel in two ways: it is a cl approach in nlp that focuses on incremental task complexity instead of incremental example difficulty; and unlike chain of thought prompting, cft does not depend on huge, pretrained lms-it relies on smaller, fine-tuned lms instead. this is advantageous because the largest lms are hard to access and expensive, and their pretraining data, while vast, still fail to cover a wide range of domains. we focus on conversational recommendation, which is especially rich in complex tasks @xcite . as shown in figure 1 , a relatively short recommendation prompt may comprise component tasks as diverse as understanding a user preference-related to pragmatics-and finding an item that correctly matches the semantics of such a preference. despite this diversity in component tasks, recommendation tasks are still underexplored in the nlp community @xcite @xcite . we make the following contributions: when compared to chain of thought prompting, we further find that cft performs equally or better while requiring lms only 7.4% of the size (as seen in ## problem definition the recommendation task depicted in figure 1 takes as input a set of items (set i) and a set of user preferences (set p ), such that recommend(p, i) outputs the item that best matches the user preferences. in its simplest form, we have a pair of items @xmath0, i 2 } and a single preference @xmath1, such that recommend({p}, {i 1 , i 2 }). this form maps naturally to what we call a "decision template," composed of two sentences: one with a preference (e.g., "you don't like cold weather."), and another with a sufficiently different pair of items (e.g., "between london and lisbon, you should visit" → lisbon). we use the term "decision" because recommend(p, i) can be considered an instance of a decision task where i represents options and p expresses the criteria to be applied. breaking down recommend({p}, {i 1 , i 2 }) into component tasks, the first task consists of comparing two items along a given attribute. this can be defined as compare(a, o, {i 1 , i 2 }) that takes as input an attribute a (e.g., temperature), an order o (e.g., higher), and the two items, and then outputs the item that satisfies the comparison. we call this task a "factual comparison" (e.g., "between london and lisbon, the city with warmer weather is" → lisbon), which is further decomposed into "factual statements" that simply enunciate the attribute value of an item (e.g., "the average temperature in lisbon is" → 17.5c). with that, a domain d can be formalized as @xmath2, a) where i f ull is the full set of items and a the set of attributes. considering the world travel domain, for example, i f ull may represent a list of well-known cities and @xmath3, population} the average temperature and total population, respectively. we instantiate this schema in our experiments in §5, but it can be used to generate new recommendation datasets or repurposed for other decision tasks. ## compositional fine-tuning (cft) cft consists of three sequential steps: decompose, where we break the complex task into component tasks; demonstrate, where we generate examples for each of these component tasks; and fine-tune, where we organize the training data according to task-level compositionality. ## experiments considering our problem and cft, we pose the following questions: we conduct four experiments to answer rqs 1 and 2, leveraging data from two domains: world travel, and local dining. world travel represents a less challenging scenario, considering that the lm is more likely to have prior knowledge about world cities and their various attributes. local dining, conversely, represents a more challenging scenario, as the lm is less likely to exhibit any prior knowledge. our experiments are based on gpt-3's curie model (13b parameters), which was the largest lm available for fine-tuning at that time. 3 ## discussion across our experiments, both in our released dataset and in sports understanding, we found consistent evidence that lms may benefit from compositional structure when learning a complex task. although we obtain improvements from three very different types of component tasks-factual statements, factual comparisons, and negative preference interpretations-standard end-to-end learning schemes tend to overlook the explicit use of compositional structure or focus only on factual knowledge. we hope to encourage further research in other principled, task-agnostic methods for leveraging compositional structure in lm fine-tuning. compared to chain of thought prompting, methods based on fine-tuning have at least two advantages. first, 100+b parameter lms are hard to access and expensive. when using davinci with 8-shot chain of thought prompting, each of our examples costs usd 7.5 cents, foot_8 which is roughly 50 times more expensive than fine-tuning curie with cft. second, many domains are not within pretraining data (e.g., due to proprietary data), so it is necessary to consider fine-tuning methods that inject custom data and preserve the lm's ability to chain thought. this limitation of strictly promptbased methods has been recently noted by @xcite , and we emphasize it in light of our results in the local dining domain. while cft certainly requires more data than chain of thought prompting, interestingly, we found it to be remarkably more efficient w.r.t model size. works leading up to @xcite have hypothesized that generating intermediate steps expands an lm's ability to reason beyond a single forward pass @xcite ; however, cft suggests that we have not yet exhausted what can be done within one forward pass. considering this optimal use of smaller models, cft can be potentially used for "distilling" a complex multi-step workflow based on very large lms-as seen in @xcite -into one smaller lm. we believe that our findings motivate research in fully automating the steps behind cft. for the decompose step, prior nlp works in decomposition @xcite @xcite could be expanded to this context. @xcite , in particular, point to an interesting direction with "least-to-most prompting." we note that the automation of the decompose step is also warranted by chain of thought prompting, in which decomposition is also performed manually. for the demonstrate step, automation would entail a few sub-steps: (i) exploring the lexical space (e.g., the space of possible preferences); (ii) generating paraphrases to increase natural language variation (e.g., our phrasings); and (iii) populating these phrasings with data from a domain of interest (i.e., i f ull ). in contrast, automating the fine-tune step is straightforward. finally, any models generated with cft can be viewed as components themselves. for example, if a model is not able to handle larger sets of preferences or items (i.e., |p | > 1 or |i| > 2) in a decision template without losing performance, then one potential solution is to use an upstream agent to break a complex case into smaller ones (i.e., with |p | = 1 and |i| = 2) and combine their outputs. @xcite propose a framework than can be applied to this end. ## conclusion & future work in this work, we proposed cft as an improvement upon end-to-end learning. to enable research on this topic, we developed a new schema for generating recommendation datasets, which we instantiated in two domains. we showed that cft indeed consistently outperforms end-to-end learning, as much as 32% for local dining. furthermore, we found evidence suggesting that more component tasks can be beneficial for cft. finally, instantiating chain of thought prompting in our dataset and cft in sports understanding, we found cft to be as good or better with lms only 7.4% of the size. for future work, we plan to apply cft to tasks with even more depth and breadth as in figure 1 , as well as to conventional spatial navigation datasets (e.g., scan from lake and baroni ( 2018 )). at the same time, we encourage others to test cft on the large family of tasks that fit the inference types already covered in figure 1 : those including facts, comparisons, criteria interpretations, and decisions, as seen in the recommendation task; or those including facts and assertions, as seen in sports understanding. we also plan to explore ways for fully automating the decompose and demonstrate steps.