Creative Problem Solving in Large Language and Vision Models - What Would it Take? (2024)

Abstract: We advocate for a strong integration of Computational Creativity (CC) with research in large language and vision models (LLVMs) to address a key limitation of these models, i.e., creative problem solving. We present preliminary experiments showing how CC principles can be applied to address this limitation. Our goal is to foster discussions on creative problem solving in LLVMs and CC at prestigious ML venues.

## introduction creativity is "...the ability to come up with an idea which, relative to the pre-existing domainspace in one's mind, one could not have had before. whether any other person (or system) has already come up with it on an earlier occasion is irrelevant." @xcite , p.216. for artificial agents, computational creativity (cc) is a multi-disciplinary field (spanning philosophy, psychology, neuroscience, and computer science) that seeks to develop computational methods capable of generating creative outcomes reminiscent of creative processes in humans @xcite . within cc, creative problem solving is a sub-area that requires an agent to discover -from its perspective -novel and previously unseen ways to accomplish a task. for example, in the absence of a ladle to scoop ingredients, an agent might creatively choose to substitute a bowl in place of the ladle. in this sense, creative problem solving encompasses creativity that is specifically task-oriented, as opposed to the generation of creative artifacts e.g., music or images. while recent state-of-the-art large language models (llms) and vision-language models (vlms) have demonstrated competency in artistic endeavours @xcite , creative problem solving continues to be a shortcoming of these models (we use llvm to denote the umbrella of both llms and vlms). for instance, in @xcite , the authors point out that "discontinuous tasks" that require a certain "eureka" idea, i.e., creative problem solving, is currently a limitation of models like gpt-4. similar observations have been made in follow up work showing that state-of-the-art llms inherently possess poor creative problem solving capabilities compared to humans @xcite . given this obvious limitation, ongoing research in machine learning should seek to address the gap between llvms and creative problem solving, to further enhance the intelligent capabilities of these models. we believe that a discussion of computational creativity is essential to addressing this limitation. it is our position that machine learning and computational creativity should be strongly integrated in research to enable effective creative problem solving in llvms and push the frontiers of their ingenuity. creative problem solving can be a resourceful skill for artificial agents. as defined in prior work, "intelligence is the ability to work and adapt to the environment with insufficient knowledge and resources." @xcite , p.10. demonstrated in hallmark examples of human ingenuity, like the makeshift co 2 filter built onboard the apollo-13 @xcite , or the makeshift medical devices used to offset equipment shortages during covid-19 @xcite , creative prob-lem solving is especially important when dealing with resource-critical scenarios. since humans tend to "choke" under high pressure situations @xcite often limiting their problem solving skills, autonomous agents equipped with llvms that have similar capabilities would be highly assistive and transformative to humans in high-stake environments. these include situations like rescue missions @xcite or autonomous operation in human-inaccessible environments (e.g., space or underwater exploration) with limited resources @xcite . however, the exceptional degree of creative problem solving necessary for such assistance remains beyond the scope of llvms today, limiting their intelligence (see appx. b.1). even though creative problem solving (cps) is a shortcoming of existing llvms, computational creativity seldom finds its way into mainstream ml research. we believe this discrepancy aligns with the "two cultures" problem @xcite (also corroborated in van heerden and bas (2021); @xcite ), and is motivated by three aspects of cc literature as it relates to creative problem solving: a) the lack of a precise definition of cps makes it challenging to identify how existing approaches in llvms are deficient in cps skills; b) the somewhat "abstract" computational descriptions of cps in computational creativity is challenging to connect to practical algorithms in llvms; and c) the lack of standardized benchmarks make it harder to evaluate llvms for cps. in our discussions relating to a) in section 3.1, b) in section 4, and c) section 5, we hope to address these gaps and encourage the ml community to think about how llvms can be augmented with creative problem solving skills through a deeper discussion of computational creativity. to emphasize the applicability of principles from cc for creative problem solving in llvms, we discuss the seminal work of margaret a. boden from cc literature that introduces three forms of creativity, namely, "exploratory", "combinational", and "transformational" @xcite . prior work has discussed the extension of boden's forms of creativity to creative problem solving in ai @xcite , however, their work does not include recent advances in llvms nor how boden's principles can be extended to specific approaches for llvms. ongoing discussions by leading ml experts like dr. shane legg, co-founder of deepmind, have suggested that "search" could help such models perform creative problem solving, quote, "... these foundational models are world models of a kind, and to do really creative problem solving, you need to start searching" @xcite . there has also been speculation that openai's q * search (described as a "significant breakthrough" in popular media) could be targeting a similar approach @xcite . interestingly, we note that "search" as described here, can be linked to boden's proposed "exploratory" approach (section 4.1.1). however, in section 4, we posit that "combinational" and "transformational" modes should be equally emphasized to achieve creative problem solving in llvms. although we choose to expand on boden's work as the focal point to drive our arguments in the main paper, it is not the only theory in cc that is relevant to this discussion. for completeness, we elaborate on additional cc theories and their applicability to creative problem solving in llvms in appx. b. ## from task planning to creative problem solving creative problem solving can be broadly described as the process through which agents discover novel ways of accomplishing a task that was unsolvable prior to the discovery. computationally, creative problem solving can be achieved through planning, learning, or hybrid approaches @xcite . following a review of the different definitions of creative problem solving that have been proposed (appx. a), we believe the following most closely connects to existing formalisms in ml. @xcite define the notion of a concept, as a state (of the environment and/or agent) or action. more generally, the authors denote c x as the set of all concepts relating to x (x denotes environment states s or actions a). hence, c s denotes the set of all environmental states, and c a denotes the set of agent actions. formally, the authors state their definition as (page 7, @xcite ): ## evaluation of creativity an important discussion in the context of creative problem solving is how can creative problem solving be evaluated?. prior work has proposed that creativity necessitates both novelty and value @xcite , where the former guarantees that the generated outputs of a creative process are original, and the latter ensures that the generated outputs are useful. in the context of cps, novelty refers to the discovery of new con-cepts (as defined in section 3.1), while value insists that the newly discovered concepts successfully solve the task. hence, cps benchmarks should specifically evaluate how the task was solved (novelty and value) rather than the typical ml evaluation of whether the task was successful or not (value only). some existing approaches that make this distinction describe problem settings that can be used to measure cps skills of llms through the implicit integration of novelty and value measurements @xcite @xcite . in @xcite , the authors create a dataset of 1600 real-world problems that necessarily involve creative reasoning abilities. their proposed benchmark involves identifying novel approaches that can accomplish the given task (value). similarly, in @xcite , the authors introduce the only-connect-wall (ocw) dataset to measure cps capabilities of llms. the authors in @xcite explore physical commonsense reasoning that is more generally applicable, beyond object-based creative problems. the authors introduce physical interaction: question answering, or piqa consisting of 16,000 qa pairs where each question is paired with two possible common-sense solutions with a ground truth. in @xcite , the authors introduce commonsenseqa 2.0 (csqa2) dataset consisting of both object-based and nonobject based creative problems. the dataset consists of 14,343 questions distributed across 1,868 distinct topics. currently, to the best of our knowledge, there are no standard benchmarks available to measure cps skills of vlms, although our preliminary experiments show one way to measure this using the task of object substitution. ## conclusion and future work in this paper, we argued that an effective approach for enabling creative problem solving -currently a key limitation of llvms -should derive from computational creativity literature. to emphasize this at each juncture, we discussed the specific principles from cc that can be extended to achieve creative problem solving in llvms, describing the potential for further research with these insights. it is rare to see special tracks or workshops targeted at computational creativity within more prestigious ml conferences. these programs typically focus on creative artifact generation and art (such as the neurips workshop on machine learning for creativity and design (neurips, 2022) or the recent tutorial at emnlp on creative natural language generation @xcite ), but do not discuss cps, thus failing to bridge the gap between cc and ml. we hope to see a deeper integration of the cc communities at such strong ml venues. we hope to encourage the reader to view creative problem solving and ml holistically, through the lens of computational creativity.