Paper: A bi-objective perspective on controllable language models: reward dropout improves off-policy control performance (ICLR-2024)

Reviewer comment (Theory):
This work only focuses on a single "balanced" Pareto solution to the proposed bi-objective optimization, which weakens the motivation for using bi-objective formulation. According to the experimental results such as Fig. 3, if you only consider the reward metric and would like to relatively neglect the likelihood objective, why not consider the trade-off problem from the perspective of constrained optimization with the likelihood objective as the constraint?

Author response:
- Dear Reviewer #3, thank you for your thoughtful feedback. Personally, this comment really struck a chord with us - your question is the one that has the most implications for our research, which explores the "perspective" of training RLMs, as opposed to the majority of recent work that focuses on developing model, designing neural network structure, or implementing an algorithm. We are truly touched by your feedback.
- As this was a very important question with implications for our study, we wrote an answer for this as *Meta Response 3 (MR3) to be shared with all reviewers and ACs.* Please refer to our response in *MR3*. We have also included additional experiments related to your question in Appendix I. Please see the revised submission.
    - In the additional experiment, you can see that if the RLM is trained without *the transfer learning*, then the reward and likelihood rise together (i.e., maximizing both objectives simultaneously, which implies a bi-objective problem) until the reward touches RUBO, and then as training continues, the reward objective is sacrificed to maximize the likelihood objective (This is the theoretical Issue mentioned in MR3).
    - The trend of the reward and likelihood objective increasing together until the reward touches the RUBO shows that RLM training is a bi-objective problem, not a constrained single-objective optimization problem.
            
- Once again, thank you so much for your thoughtful feedback.

Reviewer comment (Evaluation):
One possible solution is that the authors can provide evidence reflecting some other trade-off solutions on the Pareto front are also important (e.g., plotting an approximate Pareto front and showing different behaviors of Pareto solutions).

Author response:
- Dear Reviewer #3, thank you for your thoughtful feedback. Personally, this comment really struck a chord with us - your question is the one that has the most implications for our research, which explores the "perspective" of training RLMs, as opposed to the majority of recent work that focuses on developing model, designing neural network structure, or implementing an algorithm. We are truly touched by your feedback.
- As this was a very important question with implications for our study, we wrote an answer for this as *Meta Response 3 (MR3) to be shared with all reviewers and ACs.* Please refer to our response in *MR3*. We have also included additional experiments related to your question in Appendix I. Please see the revised submission.
    - In the additional experiment, you can see that if the RLM is trained without *the transfer learning*, then the reward and likelihood rise together (i.e., maximizing both objectives simultaneously, which implies a bi-objective problem) until the reward touches RUBO, and then as training continues, the reward objective is sacrificed to maximize the likelihood objective (This is the theoretical Issue mentioned in MR3).
    - The trend of the reward and likelihood objective increasing together until the reward touches the RUBO shows that RLM training is a bi-objective problem, not a constrained single-objective optimization problem.
            
- Once again, thank you so much for your thoughtful feedback.

Reviewer comment (Experiments):
Moreover, the experiment only uses a relatively small language model (i.e., GPT-2). LLMs can weaken the influence of the reward-likelihood trade-off due to their larger model capacity.

Author response:
- Dear Reviewer #3, thank you for your thoughtful feedback. If you are asking *"Does using LLMs with greater capacity mitigate the reward-likelihood trade-off?"*, we would say *"yes"*.

- In fact, we have already discussed this in Fig 1(b) and Section 5. Looking at the first column (left-most column) of Fig 1(b), we can see that the target policy $\pi$ (green bars) is skewed to the right, where the higher reward exists from Case 0 to Case 2.

- Here, the green bar represents the position to which $\pi$ converges, and it implies the reward upper bound (RUBO) by which $\pi$ can maximize the reward subject to the behavior policy $\beta$ (blue bars) given for each case.

- To summarize, Fig 1(b) shows that the wider the coverage of $\beta$, i.e., the larger the capacity of the LLM, the higher the RUBO, and thus the position of the green bar shifts to the right. In other words, a larger capacity of LLMs mitigates the reward-likelihood trade-off and increases controllability.

- Meanwhile, thanks to your feedback, we realized that it would be very instructive to experiment with Reward Dropout for different LLMs and compare their effects directly. Therefore, we have conducted additional experiments. Please refer to Appendix J. This experiment compared reward dropout effects across GPT2 (117M), OPT (350M), XGLM (564M), and GPT2-Large (774M). It was your thoughtful feedback that inspired us to conduct this experiment. We would like to express our sincere gratitude.

Reviewer comment (Theory):
For eq. (5), if the behavior policy has already maximized the reward objective, will the bi-objective optimization reduce to a single objective optimization?

Author response:
- Dear Reviewer #3, thank you for your thoughtful feedback. our answer to your question is *"yes"*. It is correct that the bi-objective problem reduces to single-objective optimization. We think it can be easily explained with a 10-turn positioning game.
- Let us think of Fig 1 or Fig 2. If the behavior policy $\beta$ is already maximizing $R(\tau)$, then the distribution of $\beta(\tau)$ (the blue bars) will be skewed towards the highest rewarding position. This is because if the behavior trajectories $\tau \sim \beta$ are the optimal trajectories that maximize $R(\tau)$, there are no positions from which $\tau$ is sampled except for the highest reward positions.
- This means that the target policy $\pi$ will be trained with only trajectories of the highest reward position, where reward maximization is already guaranteed by $\beta(\tau)$. Accordingly, all that $\pi$ needs to optimize is the likelihood objective, i.e., $\pi$ only needs to converge to the behavior policy $\beta$.

Reviewer comment (Experiments):
For fig. 2, please provide additional results under a non-normal distribution behavior policy.

Author response:
- Dear Reviewer #3, thank you for your thoughtful feedback. We sincerely appreciate your request for additional experiments on the main content of our study. Due to a lack of space in the main text, the requested experiment has been included in Appendix C.2. Please refer to it.
- In addition to the results presented in Appendix C.2, we conducted more experiments with different behavior policies and reward distributions. However the message from the 10-turn positioning game is consistent: *“To maximize rewards, the coverage of a behavior policy should be broad enough to explore high rewarded areas, i.e., the use of LLMs is highly encouraged.”* Hence, we have included only a few of the experiments in Appendix C.2 as examples.

Reviewer comment (Theory):
What is the core idea of reward dropout? I think the core idea is to relax the distribution of rewards in order to achieve an easier Pareto improvement. From this perspective, I wonder why the final performance is improved by the quantile dropout that sharpens the distribution.

Author response:
- Dear Reviewer #3, thank you for your thoughtful feedback. You are correct that the core idea of Reward Dropout is to relax the reward distribution in order to achieve Pareto improvement. To be more precise, Reward Dropout is a technique to *"manipulate the reward distribution to force the update of $\pi$ toward a direction where Pareto improvement is guaranteed"*.
- Here we need to decide *"which way to manipulate"* the reward. You may be wondering at this point why it has to be a "quantile" dropout. To explain this, we first need to understand Theorems 4.1, 4.2, and 4.3, and how they lead to the Reward Dropout:
    - Based on Theorems 4.1 and 4.2, we proved that Eq (4) contains an inherent tradeoff between the likelihood objective $\beta(\tau)$ and the reward objective $R(\tau)$. Specifically, the RUBO in Theorem 4.1 and the log-negative relation in Theorem 4.2 clearly demonstrate that there exists a trade-off between $\beta(\tau)$ and $R(\tau)$ in Eq (4). Therefore, we can assure that the problem of minimizing Eq (4) is equivalent to a Pareto optimization problem that optimizes two conflicting objectives.
        - $\text{Eq (4)} \qquad \text{KL}\left[ \pi(\tau) || \beta(\tau) e^{R(\tau)} \right] = \sum_{\tau} \pi(\tau) \ln\frac{\pi(\tau)}{\beta(\tau)e^{R(\tau)}}$
        - $\text{(Theorem 4.1)} \qquad \text{KL}\left[ \pi(\tau) || \beta(\tau) e^{R(\tau)} \right] \Longrightarrow \mathbb{E}\_{\tau \sim \pi} \left[ R(\tau) \right] \leq \text{KL} \left[ \pi(\tau) || \beta(\tau) \right] = \text{RUBO}$
        - $\text{(Theorem 4.2)} \qquad \mathbb{E}\_{\tau \sim \pi} \left[ R(\tau) \right] \leq \text{KL} \left[ \pi(\tau) || \beta(\tau) \right] \ \overset{\pi \rightarrow \pi^{\*}}{\Longrightarrow} \ \mathbb{E}\_{\tau^{\*} \sim \pi^{\*}} [ R(\tau) ] = - \mathbb{E}\_{\tau^{\*} \sim \pi^{\*}} [ \ln \beta(\tau)] \ \Longleftrightarrow \ R(\tau)=-\ln\beta(\tau) , \ \ \forall{\tau^{\*}} \sim \pi^{\*}$
        - In Meta Response 2 (MR2), we have explained more about how RUBO implies a trade-off. Please refer to MR2.
                
    - **Both Eq (6) and Theorem 4.2 describe the optimal state of Eq (4)**, which implies a Pareto optimal solution $\tau^{\*} \sim \pi^{\*}$. This means that **Eq (6) and the result of Theorem 4.2 is just describing the same phenomenon,** i.e., the Pareto optimal state, **from the different perspective**, **one focusing on bi-objectiveness** (Eq (6)) **and the other on the trade-off** **between the two objectives** (Theorem 4.2).
        - $\text{Eq (6)} \qquad \pi^{\*}(\tau)=\frac{\beta(\tau)e^{R(\tau)}}{e^{1-\lambda}} = \beta(\tau)e^{R(\tau)} \quad \left( \because \lambda = 1 \right)$
        - As for why it should be $\lambda=1$, please refer to our response to Weakness 5 from reviewer #2.

  - **In a Pareto optimization problem,** it is obvious that **any policy that does not achieve Pareto optimality**, i.e., $\pi(\tau)\neq \beta(\tau)e^{R(\tau)}$, **always has room for Pareto improvement**. And we know that Pareto optimality presented in Theorem 4.2 is derived from RUBO presented in Theorem 4.1.
    - Here, through *reductio ad absurdum* (i.e., through the proof by contradiction), **Theorem 4.3 shows that** the contraposition of Theorem 4.1, i.e., $\mathbb{E}\_{\tau \sim \pi}[R(\tau)] > \text{KL}[\pi(\tau)||\beta(\tau)]$, leads to$\pi(\tau)\neq \beta(\tau)e^{R(\tau)}$, or **the negation of Theorem 4.1 leads to the negation of Eq (6), and Pareto optimality does not hold.** In other words, **Theorem 4.3 provides the condition for the Pareto improvement.**
        - $\text{(Theorem 4.3)} \qquad \mathbb{E}\_{\tau \sim \pi}[R(\tau)] > \text{KL}[\pi(\tau)||\beta(\tau)]  \ \overset{\text{Eq (14)}}{\Longrightarrow} \ \mathbb{E}\_{\tau \sim \pi}[R(\tau)] + \mathbb{E}\_{\tau \sim \pi}[\ln \beta(\tau)] > 0 \ \Longrightarrow \ \pi(\tau)\neq \beta(\tau)e^{R(\tau)}$
                
    - Therefore, **by forcing the Pareto improvement condition given by Theorem 4.3 to hold**, i.e.,  $\mathbb{E}\_{\tau \sim \pi}[R(\tau)] > \text{KL}[\pi(\tau)||\beta(\tau)] \ \Longrightarrow \ \mathbb{E}\_{\tau \sim \pi}[R(\tau)] + \mathbb{E}\_{\tau \sim \pi}[\ln \beta(\tau)] > 0$, **we can ensure that the target policy $\pi$ or the target LM $\pi\_{\theta}$ is updated only in the direction of Pareto improvement** that increases both the likelihood and reward objectives without unnecessary exploration, and as a result reach the Pareto optimal solution faster.
    - The message behind Theorem 4.3 is simple: *“the Pareto improvement condition is satisfied when $\mathbb{E}\_{\tau \sim \pi}[R(\tau)] + \mathbb{E}\_{\tau \sim \pi}[\ln \beta(\tau)] >0$, so manipulate either or both $R(\cdot)$ and $\beta(\cdot)$ to hold it.”* In other words, **having Eq (7) hold will always update the parameter $\theta$ of the target policy (target LM) $\pi\_{\theta}$ in the direction of Pareto Improvement.**
        - $\text{Eq (7)} \qquad \mathbb{E}\_{\tau \sim \pi}[R(\tau)] + \mathbb{E}\_{\tau \sim \pi}[\ln \beta(\tau)] > 0$
---
  - Given that the behavior policy is either a pre-defined distribution, i.e., $\beta(\cdot)$, or a pre-trained LM, i.e., $\beta_{\bar{\theta}}(\cdot)$, whose parameter is fixed as $\bar{\theta}$, **it is only $R(\cdot)$ that can be manipulated. Accordingly, we only need to focus on $R(\tau)$ to hold Eq (7).**
    - Specifically, we need to **manipulate $R(\tau)$ such that only a few high rewards are considered** because $\mathbb{E}\_{\tau \sim \pi} \left[ R(\tau) \right]$ refers to the average reward $\tilde{r}$ and the average is sensitive to bias caused by outliers. Extremely saying, **a single high reward is more effective at satisfying the Pareto improvement condition, i.e., $\mathbb{E}\_{\tau \sim \pi}[R(\tau)] + \mathbb{E}\_{\tau \sim \pi}[\ln \beta(\tau)] > 0$, than many average rewards.**
        - In fact, we have already written about this in the Reward Manipulation paragraph in Section 5. Please refer to the orange text in section 5 of the revised submission.
                
    - This is where we can explain why quantile dropout is effective. **Quantile dropout is literally a way to discard all samples with rewards below a predefined quantile and consider only a few samples with higher rewards**. In other words, **it's a technique that reinforces the bias toward high-reward samples by focusing on highly rewarding outliers to achieve Pareto Improvement.** This is why Quantile Dropout cannot help but drive Pareto Improvement.
---
- Please refer to Meta Response 2 (MR2) and Appendix B for more details related to Theorems 4.1, 4.2, and 4.3.

Reviewer comment (Theory):
The CML and RLM are still different problems. The paper should not claim that they analysis CLM by RLM because they are "intrinsically analogous".

Author response:
- Dear Reviewer #2, thank you for your thoughtful feedback. As we mention in *“MR1) of the Meta responses to All Reviewers,”* we initially defined the scope of our work as research on controllable language models (CLMs) because we believed that our work offered an original perspective on CLMs, i.e., controlling LMs can be viewed from a bi-objective perspective, and that this perspective contributes to the field of language model control research in general. However, your comment made us realize that this generalization could be misleading. We have therefore removed the sentence that includes the expression “*intrinsically analogous*“. We thank you for your rigorous comments.

Reviewer comment (Theory):
Eq. 3 is a constrained optimization problem, and Eq. 4 is a KL divergence, I do not see why the paper claims that "Eq. 3 can be expressed as Eq. 4".

Author response:
- Dear Reviewer #2, thank you for your thoughtful feedback.
- If you look carefully, you'll notice that Eq (3) does not represent a constrained optimization problem; it's just a mathematical representation of RLM as one of many ways to model $\ln p\_{\text{clm}}(\tau|c)$ (just like Eq (1) and Eq (2)).
- In other words, it's just a mathematical way of saying "RLM is a fine-tuning approach that updates the parameters $\theta\_{\text{lm}}$ so that the pre-trained language model $p\_{\text{lm}}$ is maximized for a reward function $R(\tau)$".
- Perhaps the notation $s.t.$ is causing some misunderstanding, so I'll change it to notation $\text{where}$.
    - $\text{Eq (3)} \qquad \ln p\_{\text{clm}}(\tau|c) \approx \ln p\_{\text{lm}}(\tau|\theta\_{\text{clm}}) \qquad          \text{where} \qquad \theta\_{\text{clm}} = \argmax\_{\theta\_{\text{lm}}} \mathbb{E}\_{\tau \sim p\_{\text{lm}}(\hat{\tau}|\theta\_{\text{lm}})} \left[ R(\tau) \right] \ .$
- Meanwhile, Eq (4) only defines the representative objective function of RLM. Therefore, the claim *“Eq. (3) can be expressed as Eq. (4)”* is simply a statement to explain that Eq (4) is a more refined mathematical fomulation of Eq (3).
    - $\text{Eq (4)} \qquad \text{KL}\left[ \pi(\tau) \big|\big| \beta(\tau)e^{ R(\tau) } \right] = \sum\_{\tau} \pi(\tau) \ln \frac{ \pi(\tau) }{ \beta(\tau) e^{ R(\tau) } } = - \ \mathbb{E}\_{\tau \sim \pi}[R(\tau)] + \text{KL}[\pi(\tau)||\beta(\tau)]$
    - Representative objective function of RLM : $\mathbb{E}\_{\tau \sim \pi}[R(\tau)] - \text{KL}[\pi(\tau)||\beta(\tau)]$
        
- For evidence of the use of Eq (4) as a representative objective function in RLM, please refer to the following references:
    - [Stiennon et al. (2020)](https://proceedings.neurips.cc/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf), page 6, Section 3.4, Human Feedback Policies paragraph
    - [Korbak et al. (2022)](https://aclanthology.org/2022.findings-emnlp.77.pdf), Section 3, Equation (4)
    - [Perez et al. (2022)](https://arxiv.org/pdf/2202.03286.pdf), Section 2.2, Reinforcement Learning paragraph
    - [Korbak et al. (2022)](https://proceedings.neurips.cc/paper_files/paper/2022/file/67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf), page 3, Equation (2)
    - [Ouyang et al. (2022)](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)’s [Supplementary Material](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Supplemental-Conference.pdf), page 31, Section D.4, Equation (2)
    - [Bai et al. (2022)](https://arxiv.org/pdf/2204.05862.pdf), page 17, Equation (4.1)

Reviewer comment (Writing):
$\pi$ is never defined. Also, $\Pi$ function class is never defined.

Author response:
- Dear Reviewer #2, thank you for your thoughtful feedback. But I'm sorry that I don't understand in what context you pointed out that $\pi$ and $\Pi$ are not defined. Is Section 3.2, i.e., the Terms and Notation section, insufficient to provide enough information about the definition of $\pi$? If so, I would appreciate some clarity on this feedback.

Reviewer comment (Theory):
In Eq. 6, why would taking $\lambda=1$ results in the optimal policy? It seems arbitrary to me unless I missed some important derivation proving that $\lambda=1$ is the optimal Lagrange multiplier.

Author response:
- Dear Reviewer #2, thank you for your thoughtful feedback. I understand what your point is. However, my explicit choice of $\lambda = 1$ in Eq (6) (a seemingly arbitrary choice) was intentional, and not arbitrary.

    - $\text{Eq (6)} \qquad \pi^{\*}(\tau) = \frac{\beta(\tau)e^{R(\tau)}}{e^{1-\lambda}} = \beta(\tau)e^{R(\tau)} \quad \left( \lambda=1 \right)$
            
- To understand that $\lambda=1$ is an optimal choice, not an arbitrary choice, we first need to know that from a probabilistic perspective, the optimal policy $\pi^{\*}$ is always defined as the Boltzmann distribution $\frac{\beta(\tau)e^{R(\tau)}}{\sum\_{\tau}\beta(\tau)e^{R(\tau)}}$ with regardless of $\lambda$.
- Specifically, given that Eq (6) is the solution of Eq (5) and that solving Eq (5) is equivalent to minimizing Eq (4), we can derive some properties underlying Eq (6) from the perspective of minimizing Eq (4).
    - $\text{Eq (4)} \qquad \text{KL}\left[ \pi(\tau) || \beta(\tau) e^{R(\tau)} \right] = \sum\_{\tau} \pi(\tau) \ln\frac{\pi(\tau)}{\beta(\tau)e^{R(\tau)}}$
    - $\text{Eq (5)} \quad \argmax\_{\pi} \mathbb{E}\_{\tau\ \sim \pi} \left[ R(\tau) + \ln \beta(\tau) \right] + \mathcal{H}[\pi] \quad s.t. \quad \sum\_{\tau}\pi(\tau)=1$
            
- We know that minimizing Eq (4) always results in the same optimal policy $\pi^{\*}$ regardless of $\lambda$, because the denominator becomes a normalization constant due to the probability condition $\sum\_{\tau}\pi(\tau)=1$ (see Eq (10) in Appendix A.1):
    - Eq (10)    $\pi^{\*}(\tau)=\frac{\beta(\tau)e^{R(\tau)}}{e^{1-\lambda}} =
            \frac{\beta(\tau)e^{R(\tau)}}{\sum\_{\tau}\beta(\tau)e^{R(\tau)}}  \qquad \left(\ \because \sum\_{\tau}\pi(\tau)=1 \right)$
            
- We also know that Eq (4) is the KL divergence, a metric that quantifies the distance between two probability distributions, where the minimum value is always equal to 0.

- Accordingly, the condition $\sum\_{\tau} \beta(\tau)e^{R(\tau)}=1$ is necessarily required for Eq (4) to be 0.
    - $\text{Eq (4)} \qquad \text{KL}[\pi(\tau)||\beta(\tau)e^{R(\tau)}] = -\sum\_{\tau} \pi(\tau)\ln \frac{\beta(\tau)e^{R(\tau)}}{\pi(\tau)} \geq -\ln \sum\_{\tau} \pi(\tau) \frac{\beta(\tau)e^{R(\tau)}}{\pi(\tau)} = - \ln \sum\_{\tau} \beta(\tau)e^{R(\tau)}=0 \quad \bigg( \ \text{if} \ \sum\_{\tau} \beta(\tau)e^{R(\tau)}=1 \bigg)$
            
- That is,  $\sum\_{\tau} \beta(\tau)e^{R(\tau)}=1$ is the necessary condition for minimizing Eq (4), i.e., solving Eq (5), and therefore this condition must be considered in Eq (10).

- Let us plug $\sum\_{\tau} \beta(\tau)e^{R(\tau)}=1$ into Eq (10). As a result, we can obtain $\pi^{\*}(\tau) = \frac{\beta(\tau)e^{R(\tau)}}{e^{1-\lambda}} = \frac{\beta(\tau)e^{R(\tau)}}{1}$, which implies $e^{1-\lambda}=1 \Leftrightarrow \lambda =1$  must hold.

- In conclusion, given the condition $\sum\_{\tau} \beta(\tau)e^{R(\tau)}=1$ that guarantees the minimum value of Eq (4), it is obvious $\lambda=1$ is the optimal Lagrange multiplier and thus $\pi^{\*} = \beta(\tau)e^{R(\tau)}$ will always hold.

Reviewer comment (Evaluation):
The experimental results in Section 5 does not provide surprising or insightful results. Even the translation to LLM concepts doesn’t provide insight. The takeaways are that we do better when the two rewards have more overlap (so have a wide span of vocabulary and use an LLM over a smaller model) and when the two rewards have similar maxes (so line up your two rewards by training on data that fulfills both objectives). The visualization is cute, but the takeaways are already standard knowledge (bigger models do better, OOD tasks are harder). If the intent is to empirically test your proofs, then can you write this section in terms of when a policy becomes a uniform policy (4.2), or show the pareto frontier across policies with the same reward etc. etc. The connection between 4 and 5 is weak.

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback. Your feedback, especially *"Section 5 does not provide surprising or insightful results"* and *"the takeaways are already standard knowledge"*, was very thought-provoking for us. We have had a lot of fun preparing this response and would like to express our sincere gratitude for giving us the opportunity to respond.
- We believe that the goal of Section 5 is best captured by the last sentence of the first paragraph in Section 5: *"The goal of this experiment is to confirm the theoretical results and analyze them in the CLM context."* That is, we wrote Section 5 to support the validity of the theoretical outcomes presented in Section 4, including theorems and propositions.
- The most important contents of our work are Theorems 4.1, 4.2, and 4.3 and the simple yet powerful technique called Reward Dropout that derives from them. **Therefore, I sincerely request that you read *“MR2) of the Meta Responses”* first.**
---
- As explained in *“MR2) of the Meta Responses”*, **Reward Dropout is built on Theorems 4.1, 4.2, and 4.3**, and thus we need to verify how reliable these theorems are before examining the effectiveness of Reward Dropout. More precisely, we need to verify whether our theorems can derive *“obvious facts”*, *"trivial properties"* or, to use your words, *"unsurprising takeaways."* **If our theorems fit well with obvious facts** (known as *"standard knowledge"*) **without contradiction,** then we can say that the theorems are reliable, and by extension, **we can say that the theoretical background for Reward Dropout is reliable as well.**
- **Accordingly, we have chosen two specific questions** that can be applied to these theorems. **Our aim is to find answers to these questions and ensure that these answers are aligned with *“standard knowledge”***. The two questions we have selected are as follows:
    - *“Can behavioral policy improve the Pareto optimum of target policy (auxiliary condition; Proposition 4.1)?”*
    - *“What is the Pareto optimum of the target policy if the behavioral policy does not have a feasible region (violation condition; Proposition 4.2)?”*
    - The reasoning behind the above questions is straightforward. We consider RLM as a bi-objective optimization problem, and it is important for optimization research to understand the conditions under which a better local optimum can be obtained (auxiliary conditions) and the conditions under which a solution cannot exist (violation conditions).
            
- **Proposition 4.1 follows from Theorem 4.2 and states that the expected reward of the optimal target policy $\mathbb{E}\_{\tau^{} \sim \pi^{}}[R(\tau)]$ can be improved**. This provides the answer to *“Can behavioral policies improve the Pareto optimum of a target policy?”,* which implies that *“the larger the capability of the behavioral policy, the better the expected reward of the optimal target policy.”* **This aligns well with our standard knowledge *"bigger models do better"*** without contradiction.
    - $\text{(Theorem 4.2)} \qquad  \mathbb{E}\_{\tau^{*} \sim \pi^{\*}} [ R(\tau) ] = - \mathbb{E}\_{\tau^{\*} \sim \pi^{\*}} [ \ln \beta(\tau)]$
    - $\text{(Proposition 4.1)} \qquad \mathbb{E}\_{\tau^{*} \sim \pi^{\*}} [ R(\tau) ] = - \mathbb{E}\_{\tau^{\*} \sim \pi^{\*}} [ \ln \beta(\tau)] \quad \Longrightarrow \quad \mathbb{E}\_{\tau^{\*} \sim \pi^{\*}}[R(\tau)] \propto \mathcal{H}[\beta(\tau)]$

Reviewer comment (Experiments):
Also this set-up described in Appendix D.2 doesn’t really allow you to modify the first part of β(τ), and I feel that to keep with the spirit of this being a bi-objective RL problem, it should be possible to get different rewards from that first part of β(τ).

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback. We understand your concerns.
- However, as the mathematical notation throughout the paper and explicit references in Section 2.2 and Appendix D.1 indicate, it is the trajectory or sentence $\tau$ that we consider as a variable in this study. The reason we followed the mathematical framework of probabilistic inference described in Section 2.2 rather than the traditional RL framework is that we wanted to treat *“the entire trajectory as a random variable,”* not just the actions, states, or parts of the trajectory.
- In this sense, fixing the initial part of $\tau$ is not a problem. Of course, we agree that the fact that the initial part of $\tau$, defined as a random variable, depends on a fixed prefix, i.e., *“a constant variable,”* may fade the meaning of *"random variable"*. However, what is important to us is the complete sentence (complete trajectory) sampled starting from the prefix, and since there are theoretically an infinite number of possible complete sentences, I don't see a big problem with interpreting $\tau$ as a random variable.
- To summarize,
    1. Eq (6), which formalizes the bi-objective spirit, originates from the "probabilistic inference" problem of minimizing Eq (4).
    2. And the random variable defined in this paper is the complete trajectory $\tau$, not a part of it. 
    3. Therefore, if there is no problem with interpreting $\tau$ as a random variable as a complete sentence, then fixing a part of $\tau$ does not go against the spirit of the bi-objective

Reviewer comment (Presentation):
[PRESENTATION] The decoding method is quite expensive, as it involves updating the params of the entire LLM for each target objective (finetuning to be polite is one entire fine-tuning, finetuning to be a negative sentiment is one entire fine-tuning). A lot of CLM methods have the same model be able to output language under different control codes. Can you call this out more explicitly in the limitations section (because it doesn't come across until you read the Appendix).

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback. Your feedback is certainly valid. Unlike other approaches such as CCLM and BCLM that allow *"a single model to generate text from different control codes,"* what you point out is a common problem with RLM-based approaches. We will discuss this in the limitations section as page limits allow.

Reviewer comment (Novelty):
[SOUNDNESS/NOVELTY] The last section (the NLP experiment) is strikingly similar to: https://aclanthology.org/2022.naacl-main.57.pdf I'm actually not sure there are any differences except that they don't then re-train the policy (please highlight the other differences for me if they exist). Do you do better than this paper (them only using the method as a controlled decoding solution, and your version fine-tuning based on the controlled decodes)? (A positive answer to this question would also help motivate your RL framing where the policy is typically updated).

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback. we have skimmed the paper you referenced. Unfortunately, we are not sure that your reference and our work are appropriate to compare, for the following reasons:
    - Our work belongs to the class of RLMs.
    - The reference work, NeuroLogic A*esque paper, seems to belong to the class of BCLMs.
            
- Nevertheless, we believe that there are some similarities between the two papers in terms of finding paths, or trajectories, that maximize future (expected) value. Therefore, we will contrast the differences between the two papers to the best of our knowledge, to make the contribution of our work more explicit.
    - **Our work is a train-time approach**, considering the language model as a policy and optimizing the policy for a control objective.
        - On the other hand, **the reference work is an inference-time approach,** predicting tokens’ future value subject to the control objective and steering the decoding direction accordingly.
    - **Our work proposes an RL technique.**
        - On the other hand, **the reference work proposes a decoding algorithm.**
    - **The Reward Dropout** proposed in our paper intervenes in the **"training process"** and improves **"training performance"** based on **"theoretical grounds."**
        - On the other hand, **the NeuroLogic Aesque Decoding** proposed in the reference paper intervenes in the **"inference process"** and improves **"decoding efficiency"** based on **"heuristic grounds.".**

Reviewer comment (Theory):
In Eq 4, why is the reward function exponentiated? Why does this make sense in context of the problem (and not only because it leads to nice cancellations in successive equations).

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback. You might have asked this question lightly, but for us, your question had a very big implication. Eq (4) is taken for granted in RLM research as the objective function of RLM, and it was very refreshing for me to see that Eq (4) itself could be questioned. It was also a learning experience for me. We have answered this question in *“MR3) of the Meta responses to All Reviewers.”* Please refer to MR3.

Reviewer comment (Writing):
How is R computed for each sentence?

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback.  As is usual in RLM research, we pretrained the reward model to compute the rewards. The pre-trained reward model predicts the probability that the generated sentence has the intended attribute. Please refer to the Pseudo algorithm table in Appendix D.4.

Reviewer comment (Experiments):
Are there other RL tricks you believe would be useful? Adding those here (and beating CLM baselines AND comparing the effect of updating policy with the trick vs only controlled decoding with the trick) would better help make the case for the utility of the RL-perspective empirically. As is, I'm not convinced this framing is marginally more empirically useful.

Author response:
- Dear Reviewer #1, thank you for your feedback. Your question about other RL 'tricks' that we believe useful sounded like a 'tricky' question, and to be honest, it is because we didn't understand what you meant by the term 'RL trick'.

- Of course, there are many “tricky techniques “ that help improve RL performance. For example, adding a baseline to the reward is one of the simplest techniques you can use. This technique fixes the reward near the baseline, making the objective function that optimizes the reward an unbiased estimator, thereby reducing the variance of trajectory sampling and improving the stability of model training.

- Another example of a simple and tricky technique is reward clipping, which is a common technique used in DQNs. Reward clipping was designed to improve the robustness of a model by reducing the impact of extreme values, and is widely used as an alternative to gradient clipping; in fact, Reward Dropout is the philosophical opposite of reward clipping in that it focuses on amplifying the impact of extreme values.

- If the term RL trick you mentioned means the “reward-driven” RL techniques as simple and tricky as the examples above, as far as we know, there are not many such simple & tricky techniques. In particular, most of them focus on training stability, like reward baseline and reward clipping introduced above, and we have not yet seen any techniques that focus on improving control performance, like Reward Dropout. In this sense, Reward Dropout can be said to be very original and novel since it aims to improve control performance while being simple and tricky.

- **we are strongly confident that Reward Dropout will become a very influential technique in the future because it not only definitely improves the performance of the control, but also can be applied to any method, algorithm, or task** as long as it deals with RL problems. Considering the high applicability of the RL framework despite its high computational complexity, **I believe that more and more research will be based on RL in the future, and the importance of Reward Dropout will increase accordingly.**

Reviewer comment (Theory):
My own main weakness as a reviewer is that I may be undervaluing the theoretical contributions here. Other works have already referred to controlled decoding as an RL problem, but they do not have your proofs. Either discuss why the theory alone is a solid contribution (Why is treating CLM as a biobjective RL problem -- as past papers do -- is an unsafe assumption without these proofs established) OR make the case that the RL perspective is useful empirically (See Question 3).

Author response:
- Dear Reviewer #1, thank you for your thoughtful feedback. We think our response to *“weakness #1”*, and *“MR2) and MR3) of the Meta Responses”* might be the answer to this question.