Benchmarking Language Model Creativity: A Case Study on Code Generation (2025)

Abstract: As LLMs become increasingly prevalent, it is interesting to consider how “creative” these models can be. From cognitive science, creativity consists of at least two key characteristics: convergent thinking (purposefulness to achieve a given goal) and divergent thinking (adaptability to explore new environments or constraints) (CITATION). In this work, we introduce a framework for quantifying LLM creativity that incorporates the two design ingredients: (1) We introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions to a given problem by incrementally imposing new constraints on the previous solution, compelling LLMs to adopt new strategies. (2) We define NEOGAUGE, a metric that quantifies both convergent and divergent thinking in the generated creative responses by LLMs. We test the proposed framework on Codeforces problems, which serve as both a natural dataset for coding tasks and a collection of prior human solutions. We quantify NEOGAUGE for various proprietary and open-source models and find that even the most creative model, GPT-4, still falls short of demonstrating human-like creativity. We also experiment with advanced reasoning strategies (MCTS, self-correction, etc.) and observe no significant improvement in creativity. As a by-product of our analysis, we release NEOCODER dataset for reproducing our results on future models.

## introduction most recent works on llm creativity evaluation focus on open-ended generation tasks, such as storywriting @xcite @xcite , paper abstract generation @xcite , and role-play discussion @xcite . however, the degree to which llms possess and utilize creativity for problem-solving remains unclear. an ## constructing the neocoder dataset we present denial prompting to stimulate creative responses from llms. ## experiments and results we report the creativity of current llms ( §5.2) and evaluate different reasoning strategies ( §5.3) for creativity. ## conclusion we propose protocols for evaluating language model creativity in problem-solving and introduce the denial prompting framework and neogauge metric to provide a comprehensive creativity evaluation, measuring both convergent and divergent creativity, inspired by extensive research on human creativity. to facilitate future research, we release our neocoder dataset and shed light on the limitations of current reasoning strategies in improving llm creativity.