Creative Preference Optimization (2025)

Abstract: While Large Language Models (LLMs) have demonstrated impressive performance across natural language generation tasks, their ability to generate truly creative content—characterized by novelty, diversity, surprise, and quality—remains limited. Existing methods for enhancing LLM creativity often focus narrowly on diversity or specific tasks, failing to address creativity’s multifaceted nature in a generalizable way. In this work, we propose Creative Preference Optimization (CrPO), a novel alignment method that injects signals from multiple creativity dimensions into the preference optimization objective in a modular fashion. We train and evaluate creativity-augmented versions of several models using CrPO and MuCE, a new large-scale human preference dataset spanning over 200,000 human-generated responses and ratings from more than 30 psychological creativity assessments. Our models outperform strong baselines, including GPT-4o, on both automated and human evaluations, producing more novel, diverse, and surprising generations while maintaining high output quality. Additional evaluations on NoveltyBench further confirm the generalizability of our approach. Together, our results demonstrate that directly optimizing for creativity within preference frameworks is a promising direction for advancing the creative capabilities of LLMs without compromising output quality.

## introduction large language models (llms) have made significant progress across a broad range of natural language generation tasks @xcite @xcite @xcite . however, whether llms exhibit true human-like creativity, i.e, the ability to produce novel (i.e., original), high-quality (i.e., useful) and surprising (i.e, unexpected) ideas @xcite remains unclear. research on the creativity of llms has found mixed results, with some reporting that llms are more creative than humans @xcite @xcite , others reporting that they are less creative @xcite @xcite , and some finding their creativity to be on par with each other @xcite . however, past research has also found that the high llm performance can be attributed to the artificial nature of the creativity tasks @xcite ) commonly employed to evaluate llms, such as the alternative uses task @xcite or to the remarkable creativity of human-written texts on the web @xcite . consequently, llms have been shown to often lack novelty and surprise in their generations @xcite @xcite and produce significantly less diverse content compared to humans @xcite @xcite @xcite @xcite . these tendencies limit the utility of llms for creative tasks, such as story generation and creative problem solving that often require longer responses and "out-of-the-box" thinking @xcite @xcite . recent research has proposed some methods for improving the creativity of llms, often targeting the diversity aspect alone @xcite @xcite @xcite @xcite @xcite or focusing on a single creativity task @xcite @xcite . however, creativity is a multifaceted ability that also encompasses novelty, surprise, and quality and manifests itself in a wide range of tasks. consequently, it has been argued that methods promoting creativity improvements should consider multiple dimensions of creativity together across several creative tasks @xcite . hence, the broader challenge of enhancing overall creativity in llm outputs largely remains underexplored. to this end, we propose a novel approach to directly optimize for creativity in language model generation through preference learning @xcite . recent works targeting improvement in llm creativity have mainly focused on black-box techniques to elicit creative outputs through input-level (e.g., prompting) @xcite @xcite and outputlevel strategies (e.g., creative decoding) @xcite @xcite . however, these methods are inherently limited to the fixed creative capacity of language models and are not designed to optimize for fine-grained dimensions of creativity. recently, motivated by the negative impact of the preference alignment techniques on the diversity of llm outputs @xcite @xcite @xcite , few works have suggested directly modifying the preference optimization methods to promote output diversity @xcite . inspired by these approaches, we design a new optimization strategy that injects signals from multiple dimensions of creativity into the preference modeling objective in a modular fashion. specifically, we integrate the novelty, diversity, surprise, and quality dimensions of creativity into the training objective of direct preference optimization (dpo) @xcite , with weighted composition that allows balancing each dimension's contribution. we call this method creative preference optimization (crpo) and provide its conceptual illustration in figure 1 with full details in section 3. we test the efficacy of crpo using muce (multitask creativity evaluation), our newly curated large-scale dataset of prompt-response pairs annotated with human preferences across a diverse range of creative tasks in multiple languages. while previous work has largely evaluated creativity improvements on a narrow range of tasks like story generation @xcite @xcite or creative problem solving @xcite , muce enables us to test whether our methods truly generalize across a diverse range of creativity assessments. our results show that llama-3.1-8b-instruct (ai@meta, 2024) and mistral-7b-instruct-v0.3 @xcite trained using crpo outperform the same models trained using only supervised fine-tuning (sft) or dpo without any creativity injections, as well as existing llms such as gpt-4o, generating more novel, diverse, and surprising outputs than all the baselines while maintaining high quality. we publicly release our code, models, and data for future research foot_0 . our main contributions are as follows: ## creative preference optimization according to its three-criterion definition, creativity involves the generation of novel, high-quality, and surprising ideas @xcite @xcite . moreover, creative outputs tend to be highly diverse across individuals @xcite . therefore, to promote overall creativity in llm outputs, we propose to inject unsupervised metrics related to each dimension of creativity into the loss functions of standard preference optimization methods. we use direct preference optimization (dpo) @xcite to illustrate our modifications to the loss function. recall that in the standard formulation of dpo, a policy model (p θ ) is directly optimized on a dataset of (x, y w , y l ) where x, y w and y l refer to the model input (i.e. prompt), preferred (i.e. chosen) model response and dispreferred (i.e. rejected) model response, respectively. using the ratio between the policy model's likelihood and that of the reference sft model (p sf t ) as an implicit reward, the training objective of dpo is defined as follows @xcite : a challenge with standard preference optimization methods is that they may significantly reduce the diversity of the responses llms generate, as the loss function encourages models to generate preferred responses even if they are not very creative @xcite @xcite @xcite @xcite . existing approaches to address this in the preference optimization objective have centered around curating preference data based on various diversity metrics @xcite or incorporating extra regularization terms that encourage diverse generations while balancing quality @xcite . for example, the recently proposed diversified dpo (ddpo) method adds a scalar diversity term δ w (i.e. diversity score of the preferred response) into the dpo loss @xcite : while diversity is important for creativity, research in psychology has long established that truly creative responses also require novelty, surprise, and quality @xcite @xcite . therefore, we propose incorporating metrics for each of these, alongside diversity, into the preference loss in a modular structure, enabling the construction of different creativity models by combining these dimensions as needed. in our proposed creative dpo loss, δ w , ν w , ξ w and γ w correspond to diversity, novelty, surprise and quality scores of the preferred response respectively and λ d , λ n , λ s and λ q are hyperparameters that control the effect of each score (we call them injection weights). in particular, when λ @xmath0, λ @xmath1, λ @xmath2, we recover the ddpo loss. while there are multiple approaches for operationalizing δ w , ν w , ξ w and γ w , we propose to use the following metrics for each: ## the muce dataset to compile muce, we solicited data from the global creativity research community, specifically targeting researchers studying human creativity to obtain data from tasks known to be valid creativity measures. we specifically targeted datasets which contained complete metadata, including information about the task, language, and items that participants responded to. we gathered additional data by performing a manual search of the open science framework database foot_1 , and only retained data from peer-reviewed articles. in total, 43% of the data in muce has never been publicly released, making it unlikely that llms have seen the item-response combinations for the majority of our tasks. every response in muce was rated for creativity by at least two raters, and in some cases up to 75 employing a missing-raters design @xcite . while it is common practice to measure creativity using multiple independent raters, individual raters may deliver unhelpful or noisy ratings if they did not understand the task instructions, had a different understanding of the rating criteria, or for other reasons @xcite . to account for this, we followed best practices for subjective scoring tasks by employing judge response theory @xcite to check for raters whose ratings were uninformative in an information-theoretic sense. we fit jrt models to each task within muce, which gave us an information function for each rater across tasks. we then input the results from the jrt into a genetic algorithm @xcite which identified a subset of raters per dataset that maximized the per dataset rater information function. 3 this process dropped uninformative raters from each dataset, enhancing the quality of the final creativity ratings. the individual rater's scores were aggregated via factor scores, as is best practice in creativity assessment (silvia, 2011), and we rescaled the factor-transformed creativity scores into the integer range 10-50 as is done for prior work in automated creativity assessment @xcite . full details about the dataset construction are in appendix a. ## results figure 2 summarizes performance on our heldout evaluation suite across creativity dimensions for all baselines and crpo models using the llama-3.1-8b-instruct as a base. results for mistral-7b-instruct-v0.3 can be found in appendix figure 6 and follows the same trends. first, we observe a clear separation between existing instruction-tuned llms and our models: while the former cluster around high quality but low novelty, diversity, and surprise, our models achieve high scores across all four dimensions. second, for each creativity dimension, the model trained with that specific injection outperforms others on the same metric, confirming the effectiveness of targeted optimization, without a considerable drop in quality. models that combine a creativity signal with an external quality signal (crpo-{nov,div,sur}-qua) improve in quality but show reduced performance on the targeted dimension, illustrating a trade-off. the same pattern holds when comparing the crpo-nov-div-sur model to the full crpo-cre model, further highlighting the balance between quality and other facets of creativity. interestingly, the vanilla dpo model, without any creativity injections, already outperforms existing llm baselines, demonstrating the strength of our preference dataset. still, most of our creativity-optimized models significantly surpass dpo across all dimensions. finally, the sft model performs worst in quality and shows only comparable performance on other dimensions, reinforcing prior findings (chung et al., 2025) about the limited generalizability of supervised fine-tuning in creative tasks, where no single "correct" answer exists. we also compare our approach to creative prompting (brainstorm & select) and decoding (min-p sampling) strategies which we detail in appendix d.3 and results are reported in appendix tables 2 and 6 . we can see that while the creative prompting and decoding strategies improve baseline performance, our models with the standard prompting and decoding approaches still beat them across all dimensions with minimal drop in quality. overall, our results show that crpo enhances multiple aspects of creativity with minimal impact on quality, offering a flexible and effective framework for creativity alignment in llms. ## discussion in this work, we propose a modular preference alignment framework that uses signals from different dimensions of creativity to promote overall output creativity improvement. we show that small llms trained using our framework on a psychologically-valid datasets of human creativity outperform strong baselines such as gpt-4o on both automated and human evaluations. our formulation builds on well-established dimensions from creativity research-specifically novelty, sur-prise, diversity, and quality-which are frequently identified as core components in both cognitive psychology @xcite @xcite and computational creativity literature @xcite @xcite . we also employ creativity metrics that provide measurable signals aligning with key cognitive theories and enable practical optimization within language models. these signals have also been shown to provide a good feedback mechanism and increase human creativity in several tasks @xcite . similarly, our work shows that these signals can be used as an implicit feedback mechanism in the learning process to improve language model creativity. additionally, our framework is well-suited for downstream creative tasks where certain dimensions of creativity matter more than others. in creative writing, novelty and diversity ensure fresh, engaging content, while surprise drives plot twists. in advertising, novelty and surprise capture audience attention in crowded markets. for brainstorming, diversity is key to exploring a wide idea space. by enabling models to be trained with customizable mixes of creativity dimensions, our framework adapts to the needs of each domain. ## conclusion we introduce crpo, a flexible methodology for enhancing the creativity of llms. leveraging a novel large-scale human preference dataset focused on creativity, we show that models aligned with crpo produce generations that are not only novel, diverse, and surprising, but also high in quality --on both our held-out evaluation suite and the external noveltybench dataset. human evaluations further confirm that raters consistently judge our model's outputs to be more creative than those of several strong baselines, highlighting the potential of our approach to boost llm creativity. while our experiments focus on smaller models such as llama-3.1-8b and an english-only dataset, future work could explore the scalability of crpo to larger models, multilingual settings and other preference optimization methods.