Paper: OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (ICLR-2024)

Reviewer comment (Novelty):
My biggest concern is that while the paper claims their approach to be based on single-stage RL-free supervised learning, they do use data generated by API models (GPT-4 & GPT-3.5), and these models are believed to have gone through multi-stage training with RLHF. If the C-RLFT method leverages these kinds of data for their training procedure, it's not very convincing to me that the method really works in a fundamentally different way from prior approaches (SFT + RLHF). It also makes the comparisons to baselines not fully fair, since many baselines (under the 13b model scale) being compared to have not leveraged training data generated by (much larger) API models.

Author response:
- Thanks for your question. Here we summarized the fine-tuning data sources of OpenChat and other popular open-source language models. Actually, most open-source SFT models leverage training data generated by (much larger) API models, except for `llama-2-chat-13b`. It should be noted many existing models have more stringent requirements on the finetuning data, either data quantity or data quality. By contrast OpenChat enables to use the minimal amount of potentially expensive expert data as well as more easily obtainable medium-quality data to achieve improved LLM finetuning performance.

 
| Model                     | Base Model    | Finetuning | Data                                             |
| ------------------------- | ------------- | ---------- | ------------------------------------------------ |
| **Equal to 13b**          |               |            |                                                  |
| `vicuna-v1.1-13b`         | `llama-13b`   | SFT        | ~70k ShareGPT (GPT-4 + GPT-3.5)                  |
| `wizardlm-v1.0-13b`       | `llama-13b`   | SFT        | ~70k GPT-3.5                                     |
| `ultralm-13b`             | `llama-13b`   | SFT        | ~1.5M UltraChat (GPT-3.5)                        |
| `llama-2-chat-13b`        | `llama-2-13b` | SFT + RLFT | ~27k high-quality SFT data + ~2.9M preference    |
| `vicuna-v1.5-13b`         | `llama-2-13b` | SFT        | ~125k ShareGPT (GPT-4 + GPT-3.5)                 |
| `wizardlm-v1.2-13b`       | `llama-2-13b` | SFT        | ~250k Evol-Instruct + ShareGPT (GPT-4 + GPT-3.5) |
| **`openchat-13b (ours)`** | `llama-2-13b` | C-RLFT     | ~70k ShareGPT (GPT-4 + GPT-3.5)

Reviewer comment (Experiments):
Would the proposed method work if the mixed-quality data are generated by open-sourced (and potentially smaller) models, such as LLaMA-13b vs LLaMA-7b?

Author response:
- Thanks for your comment. It's not appropriate to train the base model (`llama-2-13b`) on datasets generated with models weaker than the base model (`LLaMA-7b`, `LLaMA-13b`), as they have worse capabilities and are unlikely to provide useful information for model fine-tuning.
- However, if the reviewer wants to look into the performance of C-RLFT on datasets other than ShareGPT (GPT-4 + GPT-3.5), we have added an experiment using the OpenOrca dataset [1]. OpenOrca is a reasoning dataset that is generated by human-written and GPT-4. We treat human data as $\mathcal{D}_{exp}$ and GPT-4 data as $\mathcal{D}_{sub}$, respectively. The results are shown in the following Table, where C-RLFT outperforms SFT on both datasets as well as the combined dataset.

| Type                   | BBH      |
| ---------------------- | -------- |
| C-RLFT (Human + GPT-4) | **49.4** |
| SFT (Human + GPT-4)    | 47.9     |
| SFT (Human)            | 47.1     |
| SFT (GPT-4)            | 46.3     |

Reviewer comment (Novelty):
I'd like to see a discussion of Xu et al. (I understand that Xu et al. came out probably after this work, but it seems to me that there are some similarities in methodology between the two papers).

Author response:
Thanks for your question. Our C-RLFT is fundamentally different from Xu et al., as their proposed method is still a preference-based method, like RLHF, while our method simply uses two different quality datasets.

- Our C-RLFT method has extremely relaxed requirements for the training dataset. It only requires to have a small set of expert data and a larger set of medium-quality data, without the need for preference labels. Meanwhile, C-RLFT reformulates the RL objective to a weighted supervised loss, which is more stable compared to the RL training in existing RLFT methods.
- Xu et al., like all RLHF methods, require pairwise preference labels for every question $x_i \mapsto (y_i^+, y_i^-)$, i.e., for every question $x_i$, it needs a preferred answer $y_i^+$ that is better than the worse answer $y_i^-$. Moreover, Xu et al. need to use two different quality LLMs to generate positive and negative answers, and then run DPO for final policy fine-tuning. The entire learning process is very heavy and inherits the potential instability and hyperparameter sensitivity of DPO-style RL training. Moreover, the positive and negative answers generated by LLMs could also introduce potential model-specific biases to the preference data. Nevertheless, we have cited this paper in our revision. We thank the reviewer for providing this reference.

**References:**

[1] Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv 2023.

Reviewer comment (Experiments):
The paper claims the superiority of C-RLFT over RLFT. But the claim is only supported by openchat-13b, which distilled its knowledge from two RLFT models: GPT-4 and GPT-3.5. So, it is only fair to say C-RLFT is better than SFT when distilling from GPT models, which is based on the fact the openchat-13b is better than vicuna-13b-1.5. However, one cannot say C-RLFT is better than RLFT, because openchat-13b learns from models trained by RLFT.

Author response:
We thank the reviewer for this thoughtful comment. Regarding the reviewer's concern, we provide detailed responses as follows:

- First, we'd like to clarify that our model is not simply distilling from GPT models. We have added the following experiments to compare C-RLFT vs SFT on different GPT data.
As the results show, if we simply distill a pretrained model by performing SFT on GPT-4 or GPT-3.5 data, their performances are much worse as compared to our proposed C-RLFT. Moreover, SFT with both GPT-4 and GPT-3.5 (\~70k) performs even worse than SFT tuned with only GPT-4 (\~6k), suggesting that indiscriminatively training with mixed quality data is likely to negatively impact learning (also discussed in our paper). By contrast, C-RLFT can smartly leverage the quality contrast of two datasets and achieve much better performance, this cannot be explained by distillation from better GPT models.

Table R1: Comparative performance of C-RLFT and SFT tuned on different data sources.

| Type                     | Average  | AlpacaEval | MT-bench | Vicuna-bench |
|--------------------------|----------|------------|----------|--------------|
| C-RLFT (GPT-4 + GPT-3.5) | **77.3** | **89.5**   | **57.5** | **85.0**     |
| SFT (GPT-4)              | 67.9     | 85.8       | 33.4     | 84.4         |
| SFT (GPT-4 + GPT-3.5)    | 52.7     | 78.6       | 33.1     | 46.3         |
| SFT (GPT-3.5)            | 42.8     | 76.5       | 16.9     | 35.0         |

- Second, note that `vicuna-v1.5-13b` and our `openchat-13b` uses the same base model (`llama-2-13b`), as well as ShareGPT datasets (both with GPT-4 + GPT-3.5 data) for finetuning, and `vicuna-v1.5-13b` uses even more data than ours (125k ShareGPT conversations vs 70k in ours), but our `openchat-13b` has achieved much better performance. 
- We have also compared with LLM models that are trained with SFT+RLFT as shown in the following table. Specifically, `llama-2-chat-13b` has the same base model as ours, but is tuned using SFT+RLFT with ~27k high-quality SFT data + ~2.9M preference, but its average score is still worse than our `openchat-13b`. This shows the strength of our proposed C-RLFT.

Table R2: Comparing openchat with our C-RLFT to a series of fine-tuned LLMs with SFT + RLFT.
| Models              | Base Models   | Method     | Dataset                                       | AlpacaEval | MT-bench | Vicuna-bench | Average  |
| ------------------- | ------------- | ---------- | --------------------------------------------- | ---------- | -------- | ------------ | -------- |
| **Larger than 13b** |               |            |                                               |            |          |              |          |
| `gpt-4`             | -             | SFT + RLFT | -                                             | 95.3       | 82.5     | 90.0         | 89.3     |
| `llama-2-chat-70b`  | `llama-2-70b` | SFT + RLFT | -                                             | 92.7       | 60.0     | 87.5         | 80.1     |
| `claude`            | -             | SFT + RLFT | -                                             | 88.4       | 65.0     | 76.3         | 76.6     |
| `gpt-3.5-turbo`     | -             | SFT + RLFT | -                                             | 86.1       | 50.0     | 50.0         | 62.0     |
| **Equal to 13b**    |               |            |                                               |            |          |              |          |
| `llama-2-chat-13b`  | `llama-2-13b` | SFT + RLFT | ~27k high-quality SFT data + ~2.9M preference | 81.1       | 55.3     | **86.9**     | 74.4     |
| `openchat-13b`      | `llama-2-13b` | C-RLFT     | ~70k ShareGPT                                 | **89.5**   | **57.5** | 85.0         | **77.3** | 

- Lastly, our main focus of this paper is to provide a much simpler and extremely lightweight alternative solution to the existing costly RLFT methods for open-source LLM fine-tuning, rather than developing the most powerful LLM. Our proposed method can be very useful for developers with budget constraints or unable to collect lots of human preference data.

Reviewer comment (Experiments):
It is not clear if the "low quality data" is even useful at all. (see my question below)

Author response:
- The "lower quality data" is important in our C-RLFT method, as it needs to provide contrast signals to guide policy learning. As shown in the previous Table R1 in our response to **W1**, simply performing SFT on GPT-4 data cannot guarantee the best performance. Naively tuning with mixed GPT-4 and GPT-3.5 data will lead to even worse results. In contrast, C-RLFT greatly outperforms SFT on the same datasets. This is primarily because C-RLFT is performing RL rather than supervised learning on data. It uses the quality difference between two datasets as an implicit reward signal to guide policy learning, making the LLM model capable of discriminating good vs bad samples during learning, thus leading to improved performance. Without the lower-quality data, such contrast learning signals may no longer exist.
- The performance `llama-2-13b` model finetuned on GPT-4 data using SFT is reported in the previous Table R1. We will also add the new results and related discussion in our revised paper.

Reviewer comment (Experiments):
What is the performance if the llama model is only trained on GPT-4 data? I feel it should be another ablated model to demonstrate the value of "low quality data".

Author response:
- The "lower quality data" is important in our C-RLFT method, as it needs to provide contrast signals to guide policy learning. As shown in the previous Table R1 in our response to **W1**, simply performing SFT on GPT-4 data cannot guarantee the best performance. Naively tuning with mixed GPT-4 and GPT-3.5 data will lead to even worse results. In contrast, C-RLFT greatly outperforms SFT on the same datasets. This is primarily because C-RLFT is performing RL rather than supervised learning on data. It uses the quality difference between two datasets as an implicit reward signal to guide policy learning, making the LLM model capable of discriminating good vs bad samples during learning, thus leading to improved performance. Without the lower-quality data, such contrast learning signals may no longer exist.
- The performance `llama-2-13b` model finetuned on GPT-4 data using SFT is reported in the previous Table R1. We will also add the new results and related discussion in our revised paper.

Reviewer comment (Theory):
It requires the data quality to be divisible into two sets: good quality and average quality, which might potentially limit its applications.

Author response:
- Thanks for your comment. It is worth noting that our method actually has extremely relaxed requirements on datasets and is very friendly for applications. It only requires to have a small set of expert data and a larger set of medium-quality data, without the need for detailed quality evaluation for each sample nor costly human preference labels. In our experiments, we use the widely adopted ShareGPT dataset (~70k conversations), with only 6k expert conversations generated by GPT-4. This finetuning dataset essentially is the same one used to finetune `vicuna-v1.1-13b`, but as shown in our paper, our `openchat-13b` achieved significantly better performance.
- We have also summarized the finetuning data sources of OpenChat and other popular open-source language models. It should be noted many existing models have more stringent requirements on the finetuning data, either data quantity or data quality. By contrast OpenChat enables to use the minimal amount of potentially expensive expert data as well as more easily obtainable medium-quality data to achieve improved LLM finetuning performance.
- Lastly, we'd like to mention that the expert vs sub-optimal datasets division in C-RLFT can also be conducted according to the downstream tasks, e.g., safe vs not-so-safe data, preferred role vs. less preferred role in role-playing tasks, etc. Our proposed C-RLFT framework provides a very flexible and performant approach to address a wide range of applications.

Table: The data source of the proposed OpenChat and other popular language models.

| Model                     | Base Model    | Finetuning | Data                                             |
| ------------------------- | ------------- | ---------- | ------------------------------------------------ |
| `vicuna-v1.1-13b`         | `llama-13b`   | SFT        | ~70k ShareGPT (GPT-4 + GPT-3.5)                  |
| `wizardlm-v1.0-13b`       | `llama-13b`   | SFT        | ~70k GPT-3.5                                     |
| `ultralm-13b`             | `llama-13b`   | SFT        | ~1.5M UltraChat (GPT-3.5)                        |
| `llama-2-chat-13b`        | `llama-2-13b` | SFT + RLFT | ~27k high-quality SFT data + ~2.9M preference    |
| `vicuna-v1.5-13b`         | `llama-2-13b` | SFT        | ~125k ShareGPT (GPT-4 + GPT-3.5)                 |
| `wizardlm-v1.2-13b`       | `llama-2-13b` | SFT        | ~250k Evol-Instruct + ShareGPT (GPT-4 + GPT-3.5) |
| **`openchat-13b (ours)`** | `llama-2-13b` | C-RLFT     | ~70k ShareGPT (GPT-4 + GPT-3.5)                  |

Reviewer comment (Experiments):
How sensitive is the results to the division of "good" data and "average" data? For example, how about if both sets of the data is of the same quality?

Author response:
- Thanks for the reviewer's constructive comment. It should be noted that if the "good" data and "average" data in C-RLFT are of the same quality, it actually reduces performing SFT on a single source of data. To fully address the reviewer's concern, we conducted additional experiments to compare C-RLFT on GPT-4 + GPT-3.5 against SFT methods tuned on GPT-4 and GPT-3.5, as well as their combination. The results are presented in the following table. 
- The results show that SFT tuned with only GPT-4 (\~6k) even performs better than SFT with both GPT-4 and GPT-3.5 (\~70k). This phenomenon confirms our statement in the paper that "it is not advisable to indiscriminatively feed all these mixed conversations to the base model, as the low-quality data are likely to negatively impact learning" in our Introduction. Furthermore, the performance of C-RLFT is still higher than SFT with GPT-4, proving the effectiveness of our proposed method. In our revised version, we will add this experiment to the ablation study.

Table: Comparative performance of C-RLFT and SFT tuned on different data sources.

| Type                     | Average  | AlpacaEval | MT-bench | Vicuna-bench |
|--------------------------|----------|------------|----------|--------------|
| C-RLFT (GPT-4 + GPT-3.5) | **77.3** | **89.5**   | **57.5** | **85.0**     |
| SFT (GPT-4)              | 67.9     | 85.8       | 33.4     | 84.4         |
| SFT (GPT-4 + GPT-3.5)    | 52.7     | 78.6       | 33.1     | 46.3         |
| SFT (GPT-3.5)            | 42.8     | 76.5       | 16.9     | 35.0         |

Reviewer comment (Experiments):
When training with Reinforcement Learning, how stable is the training process, since RL is usually different to train in most cases?

Author response:
Thanks for your comment. 
- It should be noted that although C-RLFT optimizes the KL-regularized RL objective, its policy optimization can be equivalently transformed into a simple weighted regression objective. This actually allows us to solve an RL problem instead of using a single-stage, supervised learning scheme to learn the optimized policy. 
- Moreover, it should be noted that in our proposed C-RLFT, we **no longer** need to learn and optimize with a reward model, no need to involve the value networks as in PPO-based RLFT, and even not need to maintain the reference policy $\pi_0$ for policy constraint $D_{KL}(\pi_{\theta} \| \pi_0)$ as in most RLFT methods (also avoid related policy constraint weight parameter tuning). Everything we need is single-staged supervised learning, and thus is extremely lightweight and stable.