Paper: The False Promise of Imitating Proprietary Language Models (ICLR-2024)

Reviewer comment (Evaluation):
(Definition of Capability) The coverage of the capability evaluations is somewhat limited. Currently, authors evaluate mostly on factuality tests like MMLU/NQ/HumanEval and also show results around toxicity but how about other capabilities? Could it be that the imitation data leads LMs to reason better? Or, could it be that imitation data drives better calibration? While I do understand that there is a finite compute budget and there should be a limit in evaluation, the capability definition here is rather to limit the conclusions to draw.

Author response:
We agree that there are many additional questions that one could potentially ask about model imitation, and many of these could be a great subject for future work. In our paper we focused on a representative set of LM evaluations targeting factuality, coding ability, toxicity, and human preference. However, we agree that one notable exception from this list is reasoning. For that reason, we will add evaluations on gsm8k to our camera ready paper.

Reviewer comment (Writing):
(Minor, Title) Given my concern in 2, I’m personally slightly skeptical to call this a `false promise`. It is unclear if it is broadly a false promise, or if there are capabilities that are improved – it’s rather there exists capabilities that this process even hurts, and we should be mindful about trusting crowdsourcing or other automated evaluations to understand the impact of a process.

Author response:
We agree with the sentiment of this comment and plan to adjust the framing of the title and abstract for the camera ready version of the paper.

Reviewer comment (Experiments):
Do human raters have low quality? The incentive design and crowdworker filtering seem lacking.

Author response:
We will include further details on our human evaluation setup in the camera ready version of the paper. While we agree that many more sophisticated and costly human rater setups are possible, we and many others in the community use this setup or setups similar to ours to obtain human ratings for model outputs. Our point is not that humans in general can't distinguish between outputs from models of different capability levels but that many existing crowd worker evaluations are unable to distinguish these. This underscores our point that LM evaluation is a challenging open problem for future work to address.

Reviewer comment (Evaluation):
If crowdworkers have low quality (thus their annotations unreliable), then it doesn't seem prudent to use Figure 1(c) (crowdworker preference vs. number of model parameters) to reach the conclusion that we should improve base LLMs.

Author response:
We are not relying on Figure 1(c) to reach this conclusion, we are relying on an ensemble of evidence including the quantitative results on several benchmarks across several model sizes.

Reviewer comment (Experiments):
The authors claim that matching ChatGPT using imitation would require an “enormous” amount of imitation examples. Is this supported anywhere in the paper?

Author response:
We believe that this is supported by our scaling curves, but we will make this statement more precise in the camera ready version of our paper.

Reviewer comment (Experiments):
I wonder if practitioners may interleave ChatGPT imitation data with actual pretraining data and fine-tuning data. It’s unclear if this setting would lead to the same problems.

Author response:
Continuing to train on many more tokens of pre-training data is indeed a way to improve the base model, which would be inline with our conclusions about improvising base pretrained models. Therefore, we believe that this is out of scope for our paper, but could be interesting research for subsequent work.

Reviewer comment (Experiments):
The author claimed that it is far more feasible to distill a specific behavior from ChatGPT as opposed to broadly matching its capabilities. However, the paper only conducted experiments on NQ-synthetic data.

Author response:
We will add an additional experiment on task-specific distillation for abstractive text summarization to the camera ready version of our paper.

Reviewer comment (Experiments):
The paper claimed that imitation models are adept at mimicking ChatGPT's style but not its factuality and become far better at following instructions. But the other important ability of the model, that is, the ability to reason, has not been well studied.

Author response:
We agree that the effectiveness of model imitation for learning reasoning is indeed an interesting question! Some recent work has studied this [1]. However, we will also add evaluations on gsm8k to test this question ourselves.

[1]: Orca: Progressive Learning from Complex Explanation Traces of GPT-4

Reviewer comment (Theory):
Is model imitation still a good solution if the model's factual performance is decoupled to the retrieval model?

Author response:
Yes! To answer the question:
```
Is model imitation still a good solution if the model's factual performance is decoupled to the retrieval model?
```

We believe that this is a very interesting question to investigate! It is indeed very plausible that model imitation may behave differently when paired with retrieval. Though we believe this question is out of scope for our work. In general, there are many additional questions that one could potentially ask about model imitation, and many of these could be a great subject for future work. In our paper we focused on studying the behavior of model imitation in a setting similar to prior work [1,2,3], so as to better understand model imitation's impact on a representative set of LM evaluations targeting factuality, coding ability, toxicity, and human preference.

[1]: Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality

[2]: Koala: A Dialogue Model for Academic Research

[3]: Alpaca: A Strong, Replicable Instruction-Following Model

Reviewer comment (Novelty):
The implicit assumption of this work (revealed in the title) is that there exists a claim or understanding that imitating proprietary language models by sampling their outputs for training is all that is needed to achieve performance parity - however, I contend that this isn't the prevalent understanding. It is understood that proprietary model output is a good source of finetuning data but not necessarily the only source. See for example the use of FLAN alongside imitation datasets like Alpaca for SFT.

Author response:
We agree that there exist many alternative sources of high quality finetuning data which do not happen to be generated by proprietary language models, and that these sources may imbue different properties in the source model than the datasets that we consider in our paper.

Additionally, we agree that the prevailing understanding of model imitation and more broadly finetuning has quickly evolved in recent months and that this may not be the prevalent understanding today. We will therefore reframe the abstract and title to better reflect this for the camera ready.

Reviewer comment (Writing):
The authors present style imitation of proprietary models as a negative aspect of training on imitation data (at least in the abstract), however the right amount of style imitation can be a definite source of improvement for open-source models - as the authors point out in Section 4.4. My suggestion would be to revise the abstract to reflect this more accurately.

Author response:
We agree that there are many benefits to model imitation and will update our abstract and title to more clearly reflect this in the camera ready version.

Reviewer comment (Experiments):
In Figure 4, the 5-shot MMLU performance of the 13B imitation model is quite low for a model of that size - can the authors describe the setup in more detail?

Author response:
We will add more details about our evaluation procedures to the appendix, but to answer your question, we use a standard MMLU evaluation procedure based on the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) repo with one exception. For our imitation models we abalated evaluations using both the log-probs of answer letters to select the answer choice and using the log-probs of the answer text. We found that using answer-letters consistently yielded higher MMLU performance on our imitation models, so we report the results using answer-letters in the paper.