Paper: Tool-Augmented Reward Modeling (ICLR-2024)

Reviewer comment (Experiments):
One of my main concerns is lack of experiments on standard reward modeling datasets. There are many datasets not included in the paper such as the Anthropic HH dataset, Stack Overflow, OpenAI WebGPT, and ChatGPT comparisons datasets. They do conduct analysis on a small portion of the HH dataset, but not on the provided testing set. In addition, they show worst test accuracy than is reported in some other papers that only use conventional reward modeling [1]. Since the main claim of the paper is that by using tools they can improve the accuracy of reward models, I think their method should be validated on these popular datasets.  

[1] Dong, Hanze, et al. "Raft: Reward ranked finetuning for generative foundation model alignment." arXiv preprint arXiv:2304.06767 (2023).

Author response:
Thank you for your valuable feedback. In response to your concern, we have conducted experiments on two standard reward datasets: WebGPT Comparison and HH-RLHF. The results are as follows: We partitioned the WebGPT Comparison dataset into 13.6K training samples and 2.4K test samples. Additionally, we randomly extracted 50K samples from HH-RLHF as training samples, combining them with our TARA dataset. The results reveal that our Themis outperforms other vanilla reward models, with the exception of RAFT (LLaMA-7B). It's noteworthy that RAFT (LLaMA-7B) performs SFT on the 112K positive training samples of the HH-RLHF dataset and then executes reward modeling training on 100K pairwise samples.
In contrast, Themis does not perform SFT and is exclusively trained on 50K samples from HH-RLHF (vs. RAFT trained on 100k samples on RMs as well as SFT on 112K samples) in a multi-task learning setting (WebGPT Comparison + HH-RLHF + TARA), achieving comparable performance with RAFT (LLaMA-7B). We believe these additional experiments address your concerns and provide valuable insights into the effectiveness of our approach on standard reward modeling datasets. Thank you for bringing this to our attention, and we appreciate your thoughtful review.

| Model | WebGPT Comparisons | HH-RLHF |
| --- | --- | --- |
| Deberta [1] | - | 69.25 |
| GPT-Neo-2.7B [2] | - | 68.27 |
| RAFT (LLaMA-7B) [2] | - | **79.52** |
| RM (Bert-Large) | 66.57 | 64.20 |
| RM (Vicuna-7B) | 66.68 | 72.34 |
| Themis (Vicuna-7B) | **69.53** | 77.59 |

Reviewer comment (Evaluation):
How to trust tools is an important aspect to consider here. At least in the examples, it looks like there is a risk of biasing the reward model and generative model to outputs of specific tools being used. This could be concerning as tools are not necessarily unbiased.

Author response:
**For W1**. Your concern about the potential bias introduced by the tools is indeed crucial.  As you said, the tools may produce biased aspects or wrong aspects here, such as invalid tool invocations, incorrect tool-invoked results, biased content, and unexpected errors such as network failure. To avoid the reward model interference by these biases, we have implemented several measures:
1. **Strict Quality Control**: We rigorously control the quality of the training set, filtering out invalid tool invocations generated by GPT-4 to ensure the reliability of the training data.
2. **Noise Injection**: Recognizing the possibility of tools not always providing accurate outputs, we intentionally introduce "noise" instances in the training data. This strategy helps train the reward model to produce correct rewards even in scenarios where the tools may not work optimally. An example is shown in the following. Our Themis model encounters an error during the invocation of the *Wiki Search tool* and no results are returned. Importantly, our model demonstrates a capacity for reasoning based on its internal understanding, unaffected by the absence of tool output, which highlights the ability of our model to engage in self-reflection and form independent judgments when faced with unexpected tool behavior. 

    > Question: What do you call the chinese writing system?
    >
    > Answer: Standard Mandarin
    >
    > **Thought**: I need to invoke the Wiki Search tool to search Chinese writing system
    >
    > **Action**: Wiki Search
    >
    > **Action Input**: Chinese writing system
    >
    > **Observation**: An error occurred during the tool invoke, so no result was returned.
    >
    > **Rationale**: The answer provided is "Standard Mandarin," which is incorrect in the context of the question. The question is asking about the Chinese writing system, and "Standard Mandarin" refers to a spoken language, not a writing system. The tool's execution, however, resulted in an error, and no relevant information was retrieved. Based on this, the answer is already incorrect, and the tool's failure to provide information doesn't impact the correctness of the answer.

3. **Rationale Generation**: A key aspect of our Themis approach is the Rationale generation stage. Here, the tool-augmented reward model aggregates and synthesizes previously acquired information and undergoes a reasonable reasoning process, which involves RM's self-reflection instead of listening to the results of the tool.  The emphasis on rationale generation encourages Themis to form reasoned decisions, relying on a process of synthesis and self-reflection rather than solely relying on the output of the tools. This approach enhances the model's ability to reason independently and avoid undue influence from tool biases. This is an example in which the model generates incorrect tool invocations, leading to biased observations. However, it is crucial to note that our model identifies the errors in tool results through the Rationale module. Importantly, even in the presence of biased information, our model maintains its capacity to provide reasonable answers, showcasing its robustness and ability to navigate through biases introduced by tool invocations.

    > Question: Why is the flight/fight our defence mechanism when remaining calm and composed may serve better outcomes?
    >
    > Answer: The fight or flight response is an evolutionary response that happens in stressful situations. It can be helpful in situations where there is a perceived threat to survival. The fight or flight response is characterized by feeling bodily sensations of stress, such as an increased heart rate and faster breathing. This readies the body to either fight the danger or flee from it. The body’s sympathetic nervous system is responsible for this response, which triggers the release of adrenaline and noradrenaline...
    >
    > **Thought**: The information given in the answer is well-established knowledge within the field of psychology and does not pertain to a current event. Therefore, I don't think I need to use a tool to verify this.
    > 
    > **Action**: Google Search
    >
    > **Action Input**: None
    >
    > **Observation**: 1 of 4. pronoun · ˈnən. singular or plural in construction. Synonyms of none. 1. : not any. 2. : not one : nobody. 3...
    > 
    > **Rationale**: Although the executor's tool results did not provide any specific results or contradictions, the question asked why remaining calm and composed may serve better outcomes as a defense mechanism compared to the flight/fight response. However, the answer only explained what the fight or flight response is and how it works. The question about the potential advantage of remaining calm and composed in stressful situations wasn't addressed. Thus, it's reasonable to reason that the answer is incorrect.

Reviewer comment (Experiments):
It is not entirely clear how GPT-4 is used to generate RM training data. Note that GPT-4 itself is a system if the proposal is to use GPT-4 to train RM, one can argue why not directly train RM on GPT-4 data or use GPT-4 directly as reward model.

Author response:
**For W2**. GPT-4 is a closed-source model that poses significant challenges in terms of cost and accessibility. Additionally, the practicality of conducting reward model training on GPT-4 is hindered by the necessity to add extra fully connected layers for reward prediction since OpenAI API did not support this feature. Given its high cost and limited availability for public use, employing GPT-4 directly as a reward model becomes impractical for wider applications. Therefore, we just utilize the strong in-context learning ability of GPT-4 and prompt GPT-4 to generate partial data, including tool-invoking processes (*Thought* and *Action* in Sec 2.2) and rationales (*Rationale*). Subsequently, we meticulously filter out invalid tool invocations and incorrect rationales to ensure the quality and reliability of the generated training data.

Reviewer comment (Theory):
When alpha in eq. 2 is set to zero, we converge to typical tool use via simple prompting right? the RM is still a pretrained model and can be prompted to use tools even without explicit loss terms on tool use. Is this understanding correct?

Author response:
**For W3**.  Your question is insightful, and we appreciate the opportunity to clarify. When alpha is set to zero, the model transforms into a vanilla reward model, specifically corresponding to RM (Vicuna-7B/Bert-Large) as indicated in Table 1 and Equation 1. In this configuration, the model generates a scalar reward solely through a fully connected layer, devoid of any tool-invoking or explanation-generating processes.
With alpha set to zero, it's crucial to note that the reward model cannot spontaneously produce a tool use process without explicit loss terms on tool use. While we can resort to in-context learning (ICL) by prompting the model with a few tool-use examples, it is important to highlight that this approach may still lag behind in finetuning on different tools (with $\alpha=1$). Additionally, it is constrained by the maximum context length, as the contextual examples of ICL can consume significant length.

Reviewer comment (Experiments):
Model size of 7B is quite small to capture knowledge to compete with the tools used in this paper. I think a more realistic setup would be to take a larger model and the gap between say Wiki tool and that result could look very different.

Author response:
**For W4**. In Sec 4.2 (Scaling Trends in Themis), we explored the correlation between the performance and the scale of models. To enhance the training efficiency and reduce the resource cost, we utilized the parameter-efficient fine-tuning method LoRA which only trains specific modules, as shown in Table 1. The results elucidated a positive correlation between the scale of the model and its overall performance, and highlighted a substantial gap in the *WikiSearch* tool, as you rightly pointed out. 
Taking your valuable advice into consideration, we have conducted full-parameter fine-tuning with Themis (Vicuna-33B), and reported the results as follows.  As illustrated in the table below, Themis with 33B demonstrates superior performance compared to its 7B counterparts, exhibiting an approximately 2% absolute improvement in preference prediction. Notably, the larger models showcase optimal capabilities in harnessing *Wiki Search*, *Google Search*, and *Translator* tools. This trend aligns with the findings from our LoRA experiments, which we have incorporated into Table 1. We value your thoughtful review and trust that these additional experimental results adequately address your concerns.
| Model | Calendar | Calculator | Weather | Code | Translator | WikiSearch | Google Search | Multi | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Themis (Vicuna-7B) | 100.00 | 98.05 | 100.00 | 99.47 | 90.91 | 93.31 | 64.92 | 99.31 | 93.31 |
| Themis (Vicuna-33B) | 100.00 | 97.40 | 100.00 | 99.47 | 93.54 | 96.55 | 73.72 | 99.31 | 95.21 |

Reviewer comment (Writing):
I think the write-up could improve, especially for Sec 3.1, I had difficulty understanding exactly how GPT-4 was used and how training data was prepared.

Author response:
**For W5**. Sorry for the confusion about the data collection steps. Due to the page limits, we have revised Section 3.1 and added detailed prompts for each agent in Appendix B in the submitted version for a more comprehensive understanding. Furthermore, we have also revised Section 3.1 in the draft version, hope this can address some of your concerns. We simplify the dataset construction steps as follows:
* Step 1: Initially, we collect a reward dataset featuring each instance comprising a question, a positive answer, and a negative answer. To construct this dataset, we employ two distinct approaches: resume open-source, high-quality datasets, and generation from scratch with some heuristic methods. However, the above methods usually only produce positive answers. To address this concern, we leverage a GPT-4 as a negative generation agent to generate antagonistic negative answers, which will be described in Step 3.
* Step 2: Subsequently, we define the name, and API calls of some tools including *Calculator, Code Interpreter, Translator, Google, Weather, Calendar Search*, and *Wiki Search*.
* Step 3: In this step, we leverage the capabilities of  GPT-4 to generate the tool invocation processes and negative samples. To achieve this, we design a simulated environment where GPT-4 plays three different agents: *the negative generation agent*, *the tool agent*, and *the rationale agent*. Taking the Calculator tool as an example, we extract some instances from GSM-8K. However, the data only contain questions and positive answers. To address this limitation, we prompt the negative generation agent to generate the negative answer, constructing the pairwise preference samples. Subsequently, we query the tool agent to generate the tool invocation process, encompassing the *Thought*, *Action*, and *Action Input* stages. Then humans execute the tools based on the generated tool information, yielding the *Observation*. Finally, we amalgamate the above information (inputted question-answer, tool invocation process, observations) and prompt the rationale agent to produce the *Rationale* stage. For a more comprehensive understanding of this process, additional details are available in Appendix B.2.
* Step 4: We exclude tool-invoked processes acquired in Step 3 that exhibit invalid formats. Subsequently, we discard processes exceeding three interaction steps, lacking relevant function calls, or manifesting parsing errors in their execution results.

We hope the above can address some of your concerns.  And we sincerely look forward to further discussion. Thank you for your time and consideration!

Reviewer comment (Writing):
The description of the method is not very clear. My understanding is that the reward model first generates some explanations based on the inputted question and answer, and then connects a fully connected layer to the final hidden state to produce a scalar reward.

Author response:
You accurately captured the essence of our approach. Our reward model initiates tool-invoking processes and generates explanations based on the inputted question and answer. Subsequently, we append an ``<eos>`` token, and the scalar reward is produced by a fully connected layer using the last hidden states corresponding to the ``<eos>`` token. To enhance clarity, we have revised the method description in Section 3, providing more details. We trust that these modifications improve the overall clarity of our method description.

Reviewer comment (Evaluation):
Why is there a significant difference between the results of Themis (Vicuna-7B + LoRA) and Themis in table1?

Author response:
**For Q1**: In Themis (Vicuna-7B + LoRA), we adopt a strategy where specific parameter-efficient modules (using **LoRA**) are trained, while the **pretrained weights of Vicuna are frozen**. This approach significantly reduces training resource costs but may lead to a performance decrease compared to the full-parameter fine-tuning setting.

Reviewer comment (Presentation):
Does beta=0, w=0 in table1 mean RM (Vicuna-7B)?

Author response:
**For Q2**: RM (Vicuna-7B) represents a **vanilla reward model**. In this configuration, the model generates a **scalar reward solely through a fully connected layer**, without engaging in any tool-invoking or explanation-generating processes. Therefore, in the context of our methodology, RM (Vicuna-7B) aligns with the setting where $α=0$.

Reviewer comment (Evaluation):
I understand that all the values in table2 are accuracies, so why is the binary classification accuracy lower than 50 for RM (Vicuna-7B, Zero-shot)?

Author response:
**For Q3**: The observed lower performance in the **zero-shot setting** is primarily attributed to the fact that it was evaluated **without training on HH-RLHF datasets**. In contrast, the **fine-tuning** results demonstrate performance **exceeding 50%**. More specifically, the lower accuracy for RM (Vicuna-7B, Zero-shot) is a result of the specific design choices made in the RM model, particularly in the zero-shot setting. The evaluation on the HH-RLHF* dataset in the zero-shot scenario revealed unsatisfying performance, as indicated by an accuracy below 50%) As outlined in response to a previous question (Q2), RM (Vicuna-7B) employs a scalar reward generated solely through a fully connected layer. This approach, while chosen for its simplicity and efficiency, has inherent limitations, especially in the context of a shallow and weak signal. The model relies on the input question and answer, which may result in overfitting to training samples, particularly when evaluating with the complexities of the hh-rlhf* dataset in the zero-shot setting. In contrast, the proposed method outperforms vanilla RMs in both zero-shot and fine-tuning settings (as shown in Table 2). Moreover, it's essential to note that the task is not a traditional binary classification problem. RM (Vicuna-7B, Zero-shot) is designed to predict the preference score for both negative and positive pairs, then compute the correctness (acuracy) of the ranking order. We hope our response can address your questions.  

**Re: Q3 Follow-Up**. We appreciate the reviewer's continued engagement and would like to address the additional concern regarding the accuracy of RM using Vicuna-7B in a zero-shot setting.

The observed lower performance is **primarily due to the challenges associated with domain generalization in vanilla RM**. Specifically, the **RM (Vicuna-7B) trained on tool-related data** using the vanilla RM objective (without the ability of tool use) may encounter **overfitting to the domain-specific data** (including *Calendar*, *Calculator*, *Weather*, *Code*, *Translator*, *Wikipedia*, and *Google Search*). Consequently, it achieves less than 50% accuracy (worse than random) on the HH dataset concerning human alignment, highlighting the vulnerability of vanilla RM to overfitting within the in-domain data. On the other hand, **Themis, which is also trained on tool-related data, attains a 55% accuracy** on the HH dataset (human alignment domain) in the zero-shot setting. This **underscores the exceptional out-of-domain generalization** in preference prediction, credited to the effective integration of tool use.

Additionally, we would like to emphasize two key factors influencing zero-shot performance:
1. **Dataset Distribution and Domain Differences**: It is crucial to emphasize the substantial disparity in data distribution between our dataset, TARA, which is tool-related, and the HH-RLHF dataset, which focuses on human alignment with helpful and harmless content. The distinct nature of these datasets introduces challenges in achieving optimal zero-shot performance. The inherent differences in domain and data characteristics contribute to the observed lower accuracy in the zero-shot setting.
2. **Impact of Model Parameter Size**: The size of the model plays a significant role in zero-shot performance. Larger models, such as RM(Vicuna-7B), may exhibit a susceptibility to overfitting (easy to converge), as evidenced by the rapid convergence in just one training epoch (See Figure 3 Left). In contrast, a smaller model like RM (Bert-Large) requires more epochs to fit the reward data. The phenomenon of overfitting in a large model could explain the lower accuracy in the zero-shot setting, where the model may struggle to generalize effectively to unseen datasets. It's noteworthy that RM (Vicuna-7B) shows improved performance by +30% when finetuned on a smaller set of HH-RLHF instances (500), demonstrating the adaptability of the large model to relevant (in-domain) training data.

To sum up, the challenges observed in zero-shot accuracy can be attributed to the significant differences in dataset distribution and the impact of model parameter size. We are committed to providing a more comprehensive understanding of these challenges and implementing necessary refinements to address them in the final version of our work.