Evolutionary Reward Design and Optimization with Multimodal Large Language Models (2024)

Abstract: Designing reward functions is a pivotal yet challenging task for Reinforcement Learning (RL) practices, often demanding domain expertise and substantial effort. Recent studies have explored the utilization of Large Language Models (LLMs) to generate reward functions via evolutionary search techniques. However, these approaches overlook the potential of multimodal information, such as images and videos. In particular, prior methods predominantly rely on numerical feedback from the RL environment for doing evolution, neglecting the incorporation of visual data obtained during training. This study introduces a novel approach by employing Multimodal Large Language Models (MLLMs) to craft reward functions tailored for various RL tasks. The methodology involves providing MLLM with the RL environment’s code alongside its image as context and task information to generate reward candidates. Then, the chosen agent undergoes training, and the numerical feedback from the environment, along with the recorded video of the top-performing policy, is provided as feedback to the MLLM. By employing an iterative feedback mechanism through evolutionary search, MLLM consistently refines the reward function to maximize accuracy. Testing on two different agents points to the preeminence of our approach over previous methodology, which themselves outperformed 83% of reward functions designed by human experts.

## introduction large language models (llms) have shown remarkable success in distinct tasks. state-of-the-art models such as gemini @xcite , palm @xcite , and @xcite have achieved results comparable to human experts on different benchmarks. in this paper, we are specifically interested in their capabilities in designing reward functions for reinforcement learning practices. recent studies have shown that gpt-4 can autonomously generate reward functions for multiple agents in isaacgym by taking the environment code as context and employing evolutionary search @xcite . impressively, it achieved results similar to and sometimes even better than those of human experts. this result is very important for two reasons: firstly, the task of designing effective reward functions is notoriously challenging and timeconsuming, and this approach streamlines the process by creating an end-to-end pipeline; secondly, by requiring no additional task-specific modifications, it showcases the generalization capabilities of evolutionary search on reward design. however, a significant shortcoming of this approach, and llms in general, is that they can only operate on textual and numerical data. in contrast, when designing reinforcement learning strategies, human experts often leverage visual data to gain a deeper understanding of the problems that can be solved and improvements that can be made. it is our hypothesis that incorporating visual data could provide the model with enhanced comprehension, thus leading to improved accuracy. we introduce erom: "evolutionary reward design and optimization with multimodal large language models (mllms)" method as a novel way to generate reward functions. in the erom method, we utilize mllms' zero-shot coding abilities to generate reward functions. first, we provide the mllm with the environment as context by providing the source code; then, we give it the description of the task, guidelines for reward function generation, and an image of the idle agent. after it generates the first iteration of the reward function, we provide feedback from the environment both numerically and visually by providing the video of the agent . using evolutionary search, it generates a better set of reward functions, and this process iteratively continues. our contributions with the erom method are as follows: due to budget limitations, we mostly aimed to show a proof-of-concept of our approach. all the contributions listed above held true for our tests, but without more experiments, the (2) and (3)' rd contributions above should be approached tentatively. ## conclusion and discussion designing effective reward functions is a laborious task that requires expertise and time. recent researchers have sought to address this problem by utilizing large language models (llms) to generate reward functions by taking the environment as context, employing evolutionary search, and utilizing reward reflection @xcite . however, they have only used numerical feedback and textual information for reward sampling and the reward reflection process. in this work, we address this limitation by incorporating videos of agents in training and their idle images into the evolutionary process with the help of multimodal large language models (mllms). our aim is to enhance the success rate of previous methodology, which have already outperformed 83% @xcite of human experts in their focused tasks. experiments conducted with two agents across two distinct tasks have indicated that our approach is more effective than solely utilizing textual information.