M ac G yver: Are Large Language Models Creative Problem Solvers? (2024)

Abstract: We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking.This work (1) introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and (2) provides insight into the constrained problem-solving capabilities of both humans and AI.

## introduction creativity has long been considered the driving force behind modern civilization, and one of the hallmarks of human intelligence @xcite . as large language models (llms) have become increasingly powerful, researchers have begun to investigate their reasoning ability in problem-solving tasks @xcite problem (a): i need to roll out dough evenly but don't have a rolling pin. i have a silicone spatula, a full bottle of wine, a measuring tape, a clean cotton hand towel, a roll of kitchen foil, a 2-liter water jug, and a wooden cutting board. how should i proceed? problem (b): i have a flat tire, but my lug wrench is broken. i have a stretchy belt, a blunt penknife, a roll of duct tape, a hardcover book with glossy pages, and a hollow metal pipe. the metal pipe is rusted, breaking under force. how to replace the flat tire? to bridge this gap, we curate macgyver, a novel unconventional problem-solving dataset consisting of 1,683 sets of verbal problems that require human-like creativity in the realm of physical reasoning. drawing inspiration from the cognitive science literature @xcite , we collect problem scenarios that deliberately push against functional fixedness-a cognitive bias that limits an agent from employing familiar tools in innovative ways. notably, leveraging the generative strength of llms and the verification strength of humans, we design a novel and labor-efficient pipeline to collect progressively more challenging scenarios ( §2). these scenarios are verified by humans as requiring unconventional usage of objects to find a solution. for example, solving @xcite in figure 1 requires using the wine bottle as a makeshift rolling pin. 1 each problem in our dataset is paired with at least one human-provided or verified solution. to the best of our knowledge, macgyver is the first dataset of unconventional everyday problems requiring two key elements of creativity @xcite : divergent thinking (to come up with creative or unconventional usage of objects) and convergent thinking (to accomplish a goal efficiently). next, we use the resulting dataset as a benchmark to evaluate the creative problem-solving abilities of both human participants and recent llms, including gpt-3.5, @xcite @xcite @xcite . our results in §4 reveal a substantial gap between most lms and human. while the best performing lm, gpt-4, complements the capability of an arbitrary human under certain domain-specific settings (e.g., fixing a hole on the wall), humans' collective wisdom is so far still invincible. additionally, llms struggle to identify unsolvable problems and either exhibit misleading helpfulness or are ultraconservative in inappropriate cases. in §5, we present detailed comparison between human and machine. finally, a qualitative analysis of llm responses reveals two common failure modes: (1) models propose physically infeasible, unnecessary, or wrong solution steps that deviate from the intended goal, and 2) models hallucinate unavailable tools or do not adhere to constraints specified. we propose two prompting strategies to mitigate these common error types: (1) a self-reflection based strategy to iteratively verify the feasibility of each generated step and then modify as necessary, and 2) a cognitive-science-inspired strategy of first divergently exploring the potential use of presented tools and then converging on the problem solution. experimental results show the efficacy of 1 if the problem is unsolvable given the presented tools and constraints (problem b in figure 1 ), we expect the agent to identify such infeasibility and provide a short justification. ## macgyver dataset llms have demonstrated utility for idea generation @xcite . therefore, instead of asking humans to come up with thousands of constrained scenarios from scratch, we design a progressive refinement pipeline to explore llms' potential to generate problem settings quickly and at scale ( §2.1). human annotators then verify that each problem is concrete and requires creativity ( §2.2). each instance in our dataset includes a constrained problem setting paired with at least one human-provided or verified solution ( §2.2, §c.2). ## assessing the task difficulty to gauge the challenge of our task posed to the most recent llms, we evaluate the zero-shot performance of @xcite . nevertheless, existing automatic evaluations fall short to assess h o ld in g it e m s 1 0 . 6 % ( 9 3 7 ) cove ring 9.5% (836 ) t y in g o r c o n n e c t in g 5 . 8 % ( 5 0 8 ) c l e a n i n g 5 . 3 % ( 4 6 5 ) p ro te c ti o n 4 .6 % (4 0 6 ) mea suri ng 3.1% (276 ) eate n as food 3.1% (271 ) se al in g 3 .0 % (2 6 5 ) m a k in g n o is e 2 .6 % ( 2 2 9 ) r e a c h i n g h i g h a r e a s 2 . 6 % ( 2 2 8 ) h e a t i n g 2 . 5 % ( 2 2 2 ) g r i p p i n g t h i n g s 2 . 5 % ( 2 1 9 ) c u t t in g 2 .0 % d ra w in g /w ri ti n g 2 .0 % de co ra ti on 1. 7% sit tin g/s tep pin g 1.5 % illumi nation 1.4% starting fire 1.4% sep ara ting 1.3 % po we rin g de vic e se pa ra tio n st re tc hi ng ## benchmarking humans and llms a natural follow-up question is how well modern llms perform on this task, as compared to humans. we thus evaluate the performance of several recent llms (i.e., palm2, claude2, llama2, gpt-3.5 and gpt-4) on a representative sample of the entire macgyver dataset which contains 323 problems. in addition, we gauge the capability of average humans on the same set of tasks. ## enhancing llms' problem solving here, we investigate whether different prompting strategies can enhance the problem-solving abilities of existing llms. in §6.1, we conduct a detailed error analysis on gpt-4, showing it is weakest at identifying the correct tool affordance and physical feasibility. in §6.2, we propose two new prompting strategies that effectively reduce its mistakes. ## discussion and conclusion significance of work we propose a new playground and the accompanying macgyver dataset for creative problem solving, which covers a broad range of topics for everyday innovation, such as household, training, and outdoor sports, which is orthogonal to the existing areas of reasoning and creativity, and adds to the spectrum of machine intelligence. the area of daily innovation, or "little-c" according to the creativity theory ( conclusion we present macgyver, a novel benchmark focusing on everyday innovation that is carefully collected with quality and diversity control. we evaluate and compare both llm and human performances, and highlight failure modes of llms in proposing physically feasible actions towards a goal. nonetheless, we find llm capabilities to be complementary to human capabilities under certain domain-specific settings. we propose two new prompting methods that effectively improve this reasoning ability in llms.