Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models (2025)

Abstract: Fine-tuning Large Language Models (LLMs) with first-order methods like back-propagation is computationally intensive. Zeroth-Order (ZO) optimisation uses function evaluations instead of gradients, reducing memory usage, but suffers from slow convergence in high-dimensional models. As a result, ZO research in LLMs has mostly focused on classification, overlooking more complex generative tasks. In this paper, we introduce ZOPrO, a novel ZO algorithm designed for *Preference Optimisation* in LLMs. We begin by analysing the interplay between policy and reward models during traditional (first-order) Preference Optimisation, uncovering patterns in their relative updates. Guided by these insights, we adapt Simultaneous Perturbation Stochastic Approximation (SPSA) with a targeted sampling strategy to accelerate convergence. Through experiments on summarisation, machine translation, and conversational assistants, we demonstrate that our method consistently enhances reward signals while achieving convergence times comparable to first-order methods. While it falls short of some state-of-the-art methods, our work is the first to apply Zeroth-Order methods to Preference Optimisation in LLMs, going beyond classification tasks and paving the way for a largely unexplored research direction. Code and visualisations are available at https://github.com/alessioGalatolo/VisZOPrO.

## introduction fine-tuning large language models (llms) is a computationally demanding process, often incurring significant costs in terms of both time and memory. traditional fine-tuning relies on first-order optimisation techniques, such as backpropagation @xcite , which require substantial computational resources due to the calculation and storage of gradients. the memory footprint for storing model parameters, activations, and gradients can be prohibitive, especially for large-scale llms @xcite @xcite . this has spurred interest in alternative optimisation techniques, such as zeroth-order optimisation @xcite @xcite . zeroth-order optimisation (figure 1 , left) presents a compelling alternative by eliminating the need for gradient computation. instead, it approximates gradients using only function evaluations (i.e., forward passes), typically through finite difference methods. for example, in simultaneous perturbation stochastic approximation (spsa) @xcite , the function is evaluated two times, where each time noise is either added or subtracted from the weights and a 'step' is taken toward the most promising direction. this significantly reduces memory requirements, as the model needs to be loaded into memory only once. additionally, due to their very nature, these methods are also inherently fit for optimising non-differentiable objectives such as human preferences @xcite where a step can be taken directly towards the preferred model's generations. while promising, zeroth-order methods face challenges in terms of convergence speed, particularly for high-dimensional models like llms. recent research has explored various strategies to improve the convergence of zeroth-order meth-ods for llm fine-tuning. these include using strong fixed prompts @xcite , employing meta-learning techniques and combining zeroth-order with first-order methods @xcite . however, all empirical studies to date focus on classification or question-answering tasks, which may not fully capture the complexities associated with fine-tuning llms for more complex generative tasks. in contrast to previous work, we dive into the more challenging and practically relevant problem of preference optimisation for llms. preference optimisation involves training a policy model to generate desirable outputs, often guided by a reward model that learns to distinguish between highquality and low-quality outputs. this process, in many cases, entails iterative training of both models, generally in settings of online learning @xcite or as a way to refine the reward model to account for the distribution shift of the policy throughout training @xcite . to this end, we hypothesise that the evolving dynamics of the policy and reward models during preference optimisation can provide valuable information for guiding the search direction of zeroth-order methods. we thus begin our investigation by visualising and analysing how popular preference optimisation methods, such as proximal policy optimization (ppo) @xcite and direct preference optimization 1 (dpo) @xcite , act on the policy and reward model, with particular attention on their relation. whilst we do not find any statistical direct correlation, our visualisation and analysis suggest a significant influence of previous updates on upcoming ones. further, we find that the reward model's updates are consistently orthogonal to those of the policy. we integrate these findings into traditional spsa by influencing how the random vector is sampled, restricting the sample space to improve the convergence rate. we then show how our modifications do not hinder the theoretical guarantee of convergence. finally, we evaluate our method (sketched in figure 1 ) on various tasks and datasets to show its effectiveness. our contributions are as follows: ## visualising preference optimisation we begin our investigation by analysing how the policy and reward models are optimised using traditional methods. we simulate multiple training runs across different methods, collecting model checkpoints at various stages of optimisation. at the end of training, we visualise and analyse the weights as updated by each method with particular attention to the interplay between the reward and policy models. ## zopro: zeroth-order preference optimisation in this section, we introduce zeroth-order preference optimisation (zopro), a method for optimising policies in rlhf-like settings without explicit gradient information. we first formalise the iterative training process, describing our adaptation of simultaneous perturbation stochastic approximation (spsa) for policy optimisation, and finally present our novel sampling strategy that takes into account the insights gained with our visualisation analysis, in particular, we exploit the orthogonality between policy and reward updates and the relation between each policy update and its consequent. foot_1 ## experiments among the challenges of effective optimisation of zo methods is a careful balance of the hyperparameters η and ε: when these values are too high, the model rapidly degenerates into generating incoherent outputs; if they are too low, the model fails to improve. the challenge is further exacerbated in the context of preference optimisation, where improvements are subtle compared to classification tasks (previous works), in which performance gains are more easily quantifiable. for this, we sweep over multiple values of ε (the perturbation multiplier) and learning rates η using a standard grid search approach where we check 9 combinations of these values. we select the most promising ones based on their reward improvement and keep them fixed across tasks. for our experiments, we adopt a similar iterative training to that described in section 3.1 and algorithm 1. we initially only consider a single model, llama 3.2 1b (llama team, 2024), the previously used task of building a conversational assistant and add two new tasks: summarisation, through the dataset 'summarize from feedback' @xcite , and machine translation through the dataset wmt20 @xcite . ## conclusions in this work, we presented a comprehensive visual and quantitative analysis of preference optimisation dynamics by tracking the training evolution of both policy and reward models through popular methods. our visualisations using t-sne and umap revealed structured training trajectories, within each model (e.g., policy at iteration t compared to t + 1) and in relation to each other (i.e., policy compared to reward). these insights guided the development of zopro, the first zeroth-order method applied to the task of preference optimisation, where all previous methods only considered classification or question-answering. zopro uses our empirical findings to restrict its gradient search and thus improves its convergence rate. this is done by implementing a modified spsa approach with an adaptive perturbation strategy that balances a deterministic component (from prior policy and reward updates) and restricted stochastic exploration. we show how this choice preserves theoretical convergence guarantees. our experiments on the tasks of summarisation, machine translation and conversational assistants, show that zopro achieves consistent improvements in reward while maintaining a lower memory footprint and faster per-iteration runtime than first-order methods. this inherently comes with less overall improvement per step, putting our method just behind state-of-the-art performance. nevertheless, we demonstrate that gradient-free, zeroth-order optimisation can be both viable and competitive even when targeting preference optimisation.