Paper: Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR-2024)

Reviewer comment (Novelty):
The primary distinction from other Reinforcement Learning from AI Feedback (RLAIF) works seems to be the mapping of the LLM's reward output from text space to scalar reward space, raising questions about the approach's uniqueness.

Author response:
Thanks so much for the valuable comment. We would like to further elaborate the approach’s uniqueness, which is in two-fold:
1. The novel framework with a tunable language model plugins for LLM-based dialogue systems.
2. The design of goal-oriented AI feedback.
 
As for your concern regarding the design of goal-oriented AI feedback, we would like to highlight two distinct features from common RLAIF works:
- As you mentioned, mapping of the LLM's reward output from text space to scalar reward space for RL algorithm.
- We consider the long-term goal-oriented reward, instead of AI preference on single-turn responses. The long-term goal-oriented reward requires multi-turn interactions with a dynamic environment, i.e., the user.

**Revision**: In the revision, we elaborate the approach’s uniqueness more clearly in **Section 2** by comparing with the related works.

Reviewer comment (Experiments):
The need for training different dialog policies for each dialog domain. This makes this system less generalizable.

Author response:
We apologize for this confusion. In fact, the generality of the proposed method lies on the pluggable dialogue policy planner. With this design, we do not need to train the whole LLM-based dialogue system for each dialog domain, while we just need to train a small language model plugin. This not only makes the LLM-based dialogue system more generalizable to the new case in each dialogue domain than prompt-based approaches, but also enables the LLM-based dialogue system can be adapted to different dialogue domain by just substituting the dialogue policy planner plugin without affecting the LLM’s exceptional capabilities of response generation and context understanding.

**Revision**: In order to better elaborate the generality of the proposed method, we add more descriptions in the **Introduction**.

Reviewer comment (Reproducibility):
How is the reward LLM utilized during inference at each turn of dialogue?

Author response:
As mentioned in Section 3 “LLM as Reward Model”, the reward LLM has two functions : (1) to determine the goal completion during the conversation; (2) to evaluate the policy outcome with scalar rewards.

As shown in Figure 1(b), during the inference phase, the reward LLM will not output the reward value, which is only used for the RL training phase. However, the reward LLM will still be used to determine whether the goal has been completed during the conversation, which is used for evaluation.

**Revision**: In order to better present the inference process, we add more details in the caption of **Figure 1** regarding the usage of the reward LLM.

Reviewer comment (Reproducibility):
Could you clarify the process of mapping the reward LLM's output to scalar values and its integration into the PPDPP during each dialogue turn?

Author response:
Sorry about the confusion regarding the reward mapping process. Take emotional support dialogues as an example. To assess whether the patient' emotional issue has been solved, we prompt the reward model to answer a multi-choice question "Has the patient's issue been solved?", and then generate the goal-oriented AI feedback at temperature $\tau>0$ to sample the  responses for $l$ times. We define a mapping  $\mathcal{M}_r(\cdot)$ to transform verbal feedback to scalar rewards, such as "the patient feels worse", "the patient feels the same", "the patient feels better", "the patient's issue has been solved" as -1.0, -0.5, 0.5, and 1.0, respectively.

When we obtain the $l=5$ sampled responses as ["the patient feels better", "the patient's issue has been solved", "the patient feels better", "the patient's issue has been solved", "the patient feels better"], we can compute a continuous scalar value $v_t=\frac{1}{5}(0.5+1.0+0.5+1.0+0.5)=0.7$. If $v_t>\epsilon$, we regard the state as GOAL-COMPLETED and set the reward $r_t=v_t$. If not, we assign a small negative reward, \textit{e.g.}, $r_t=-0.1$, to penalize the lengthy conversation for promoting efficient goal completion.

**Revision**: In order to make the description of the reward mapping process more clear, we reorganize the part of reward mapping to the same part in **Appendix F.4**, which presents the whole process with specific examples.

Reviewer comment (Reproducibility):
Can you elaborate more on supervised fine-tuning used to PPDPP?

Author response:
Sorry about the confusion regarding the supervised fine-tuning (SFT) process. The SFT process is described in the second part of Section 3. The SFT exactly follows the traditional corpus-based fine-tuning process for dialogue action prediction. In specific, we have the human-annotated dialogue corpus with dialogue action labels for each turn. Given the dialogue history $\{u_1^\text{sys}, u_1^\text{usr}, ..., u_{t-1}^\text{sys}, u_{t-1}^\text{usr}\}$, the PPDPP is fine-tuned to predict the dialogue action $a_t$ for the next turn.

**Revision**: In order to make the description of the SFT process more clear, we add more details in the part of "Plug-and-Play Dialogue Policy Planner" in **Section 3**.

Reviewer comment (Evaluation):
I try to understand why there is negative relative success rate in Figure 2?

Author response:
**Response**: We apologize for the confusion. The relative success rate is calculated by subtracting the actual success rate of the Standard prompting method from that of the concerned method. Therefore, the negative relative success rate means that this method achieves a lower success rate than the Standard prompting method. This result shows that in some cases, these methods perform even worse than the vanilla ChatGPT with Standard prompts.
 
**Revision**: In the revision, we added the above descriptions on the caption of **Figure 2** to better present the results.