B ayesian Optimization for Controlled Image Editing via LLM s (2025)

Abstract: In the rapidly evolving field of image generation, achieving precise control over generated content and maintaining semantic consistency remain significant limitations, particularly concerning grounding techniques and the necessity for model fine-tuning. To address these challenges, we propose BayesGenie, an off-the-shelf approach that integrates Large Language Models (LLMs) with Bayesian Optimization to facilitate precise and user-friendly image editing. Our method enables users to modify images through natural language descriptions without manual area marking, while preserving the original image’s semantic integrity. Unlike existing techniques that require extensive pre-training or fine-tuning, our approach demonstrates remarkable adaptability across various LLMs through its model-agnostic design. BayesGenie employs an adapted Bayesian optimization strategy to automatically refine the inference process parameters, achieving high-precision image editing with minimal user intervention. Through extensive experiments across diverse scenarios, we demonstrate that our framework outperforms existing methods in both editing accuracy and semantic preservation, as validated using different LLMs including Claude3 and GPT-4.

## introduction in the rapidly evolving field of visual content manipulation, image editing has gained significant attention due to its practical applications across various domains. unlike traditional image generation models such as stable diffusion @xcite and dall-e 3 @xcite , our work is specifically focused on improving control in the image editing process. recent advancements in controllable synthesis, such as those by @xcite and @xcite original image. controllable synthesis in generation technology @xcite @xcite @xcite has recently attracted significant attention due to its expanded range of applications. models such as pix2pix and cycle-gan have demonstrated the ability to transform images from one domain to another, effectively applying controllable synthesis to tasks like style transfer and image enhancement @xcite . recent advances like zone have enabled instruction-driven modifications without pre-defined training samples @xcite , while new frameworks have emerged for intuitive and localized image editing by manipulating internal attention mechanisms @xcite . this represents a shift towards more granular control over ai-generated content, enabling precise, region-specific alterations without additional input masks. however, despite these advancements, existing methods face several critical challenges. first, most state-of-the-art local editing methods heavily rely on mask priors to constrain the editing regions-either through manual input or derived from attention map analysis and semantic segmentation-making them less accessible for non-expert users. additionally, methods like cross-attention control and diffusion models often encounter challenges in fine-tuning model parameters to align with user requirements, resulting in a disparity between desired and actual outputs. these issues are particularly pronounced in applications that demand detailed modifications based on user instructions. with the llms-driven adaptation, recent various applications @xcite @xcite @xcite have advanced developed. to address these challenges, we propose bayesgenie: a novel framework that achieves precise localized editing without any form of mask guidance. our approach uniquely combines the semantic understanding capabilities of llms with the parameter optimization power of bayesian methods @xcite . bayesgenie leverages llms to generate detailed prompts from user requirements, which then guide a stable diffusion model to modify images accurately. the framework employs bayesian optimization to systematically explore the parameter space, particularly the image and text classifier free guidance (cfg) weights, to maximize output quality. our method provides an end-to-end solution, where users are only required to provide a textual description, eliminating the need for manual selection or marking of specific image regions. this approach streamlines user interaction, enhancing intuitiveness and accessibility. moreover, our method operates without pre-training or fine-tuning on specific datasets, instead leveraging the capabilities of multiple multimodal llms to produce high-quality outcomes. our experimental results demonstrate that the integration of llms and bayesian optimization enables more intuitive and accurate image editing. as shown in figure 1 , our method can effectively implement specific modifications while maintaining the scene's overall coherence and aesthetic integrity. in summary, our contributions are: ## methodology our system architecture integrates llms and bayesian optimization for image editing (figure 2 ). an llm processes the original image and modification requirements to generate a textual prompt capturing the desired changes. this prompt and the original image are then fed into a diffusion model to generate a modified image. bayesgenie enhances this process through dynamic prompt refinement and parameter optimization. the llm evaluates each generated image, scoring it based on requirement satisfaction and providing feedback for improvements. bayesian optimization then iteratively adjusts key parameters, specifically 'text_cfg_scale' and 'image_cfg_scale', which balance text and image components in the diffusion model. this optimization minimizes the negative llm score, maximizing alignment between generated images and desired outcomes. ## discussions cost and efficiency the algorithm requires relatively low computational resources and costs. the costs for gpt-4o are shown in the table below, with the main expense being the prompt tokens for bayesian loop evaluation. the total cost for running the algorithm once to generate a 512x512 image is 0.176 dollar. additionally, this experiment was conducted on a machine with a single rtx 4080, and each algorithm run takes approximately 2.5 minutes. the algorithm to more precisely converge on an optimal result. our bayesian optimization strikes a balance between accuracy and cost by using 20 iterations, as increasing beyond this number has shown diminishing returns in terms of accuracy improvements, while still incurring higher computational costs. this choice ensures that the algorithm remains both computationally efficient and capable of generating high-quality outputs without unnecessary resource consumption. generalizability and robustness in the experiments described above, we demonstrated the robustness of this approach across different llms by using gpt-4o and claude 3.5. additionally, due to the versatility of the classifier-free guidance (cfg) technique, this method serves as a trainingfree optimization solution that can be applied to various diffusion models. future directions may include implementing the text-cfg and image-cfg modules in other diffusion models and testing this method's generalizability. ## conclusion our work introduces bayesgenie, a novel and model-agnostic framework that combines bayesian optimization with large language models (llms) to enhance the fine-grained image editing process. by leveraging the power of llms to generate natural language prompts, bayesgenie simplifies the image creation process, making it more intuitive and accessible. the framework's robust performance across various scenarios highlights its versatility and adaptability, demonstrating its potential for broad applicability in both academic and practical contexts. ultimately, bayesgenie sets the stage for future advancements in ai-driven content creation, offering a powerful tool for precise and user-friendly image editing.