Paper: Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR-2024)

Reviewer comment (Evaluation):
The PathUntil seems to be very expensive in the early stage, because of the low probability of sampling the correct answer.

Author response:
Fortunately, the computational cost remains acceptable across model sizes. For large models, we sample a few times, with each generation being slow, and for small language model, we sample a lot of times while each generation is fast. For instance, in our HumanEval experiments, we used a batch size of 200 across 8 GPUs for smaller models, resulting in a processing time of approximately 200 seconds per test example (with a sample number of 10,000).  For 0.03b model, the whole test time for human eval is 40 min. For 2.4b model, it is 2h. Both run in parallelization on 8 A100 GPUs.

Reviewer comment (Evaluation):
The smoothness of PathUntil highly depends on the output length. For HumanEval it may be okay because the code is simple and short. However, it would be very hard to make it very smooth for the long answers.

Author response:
We think that as long as we do not define ''pass'' as generate a sequence exactly the same as the ground truth answer, the smoothness of PassUntil won't be very hard for long generations. Suppose we are now predict the performance on AlpacaEval, who answer is known to be long (> 1000 words). We define ''pass''  as the generation from our model is better than GPT-4, judged by GPT-4. Thus, the probability of generating a ``pass'' answer is not exponentially decrease with the length.

Reviewer comment (Writing):
It would be helpful to provide a more detailed discussion between "ppl on task data" and "passuntil on the task data". I can understand these two are different, but this may be helpful to let more readers to understand the insight of this work.

Author response:
Thank you for highlighting this issue! A significant concern of ours is the common misunderstanding among readers, especially those without access to much pre-training experience, about the difference between 'low perplexity/loss' and 'good task performance.' We are addressing this in our revised paper by adding a detailed discussion on why tracking direct task performance, rather than perplexity/loss, is crucial:

1. PPL (Perplexity) evaluates aspects different from task metrics. It measures the likelihood of an answer being a natural continuation of a sentence, which is distinct from 'answering a question or do one task successfully'
2. PPL cannot accurately inform us of the concrete value of task performance, particularly in practical evaluations.
3. While we use loss on ground-truth of the small model to aid in predicting PassUntil, PPL alone is inadequate for predicting performance. This is because, for each sample, the accuracy is binary (0 or 1), which complicates establishing a linear relationship between model performance and PPL

Reviewer comment (Experiments):
Can the PassUntil curve be more smooth if we select the "top-K" candidates before sampling? Such as setting a value for top_k in huggingface model.generate API.

Author response:
In our experiments, we set the top-p value to 0.95. Your suggestion to improve the smoothness of the PassUntil curve by further adjusting top-p or top-k is valid. A concern, however, is setting the top-k or top-p value too low, potentially causing the correct answer to be permanently excluded from the candidate pool for small models.

Reviewer comment (Theory):
Motivation vs. actual design of the task scaling law: The paper states that “Despite the predictable decrement in LLM loss, task performance improvements are twisted during scaling” and "First, these works concentrate on training and validation loss metrics, which do not reliably predict task performance." This suggests that the loss metrics may not be highly dependable for predicting task performance. However, the task scaling law design still focuses on the correlation between PU and test loss. This may raise questions about its alignment with the initial motivation because the design of the task scaling law does not go beyond the scope of loss.

Author response:
Sorry for the confusion, reviewer 5dKH weakness-3 indicates that it's our fault that we do not make the point clear. Let us make the point clearer. 
- It's important to distinguish between '**loss is not predictive of task performance**' and '**loss can help predict task performance.**' The former suggests that loss is a not sufficient statistic for estimating task performance without other measurement, while the latter indicates that loss is one of useful factors in improving prediction accuracy. In our paper, we clearly verify both statements. Without utilizing the PassUntil method, one cannot deduce actual performance (accuracy) solely from loss values. For example, a loss of 1.0 does not directly translate to an accuracy of 0.2 for a task. And **actual performance must be empirically measured**. Furthermore, as shown in our pre-revision Figure 4, the loss of an individual sample does not have a one-to-one correlation with PassUntil results, much less with discrete accuracy. 
- However, loss does provide useful information. Once we measure PassUntil across a large sample set, we can establish a statistical relationship between loss and PassUntil (not possible if we only rely on loss data). This relationship can enhance our prediction accuracy.
- The incorporation of loss for improved predictions is driven by practical considerations, such as limited computational resources, rather than being a necessity. Figure 3 demonstrates that even without loss data, we can accurately predict task performance. Imagine a scenario where we can measure every sample with sufficient resolution to ensure each is passed at least once; in such a case, loss data would not be necessary.

Reviewer comment (Experiments):
Design of the experiment to predict performance: In the experiments for performance prediction, only the method designed in this paper was implemented, without conducting a comparison with other related methods previously proposed.

Author response:
As the first open-sourced project attempting to predict the scaling curve of task performance, we are not aware of any other existing methods capable of such prediction. If the reviewer is aware of related methods, we would appreciate references to enhance our understanding.

Reviewer comment (Theory):
PU vs. test loss in the task scaling law design: This work seems to give a vague definition of "pass" in PASSUNTIL. Can PU represent all performance metrics of tasks when required? In the task scaling law, can this relationship between PU and test loss be applicable to all tasks? In the paper "Are emergent abilities of large language models a mirage?" that this work cited, when the task performance metric is changed to Token Edit Distance, the relationship with loss does not align with the formula provided in this work.

Author response:
- Our methodology currently applies only to generative tasks where the success rate is zero for random generation. This limitation is outlined in Section 4.1's comments and the Limitations section in the Appendix. However, it appears that most tasks inherently fit this generative model with some pass criteria if the test format is adapted accordingly. We will expand on these limitations in the body of our revised manuscript.
- You mentioned 'Token Edit Distance,' which is predominantly used in multiple-choice tasks. While this metric offers more continuity than accuracy for such tasks, establishing a direct link between 'Token Edit Distance' and accuracy remains challenging. Therefore, we have deferred the exploration of these types of tasks to subsequent work.

Reviewer comment (Novelty):
The paper appears to misstate Schaeffer et al. (2023). Their abstract states, “we provide evidence that alleged emergent abilities evaporate with different metrics or _with better statistics_”. The “better statistics” refers to increased resolution. Specifically, their Figure 4 shows that increasing resolution by sampling significant amounts of additional data improves predictability on the original metric (Accuracy). One of Schaeffer et al. (2023)’s key takeaways is that predicting capabilities with scaling can, in certain cases, demand significantly higher resolution.

Author response:
- We appreciate your observation regarding the misattribution, and we apologize for any confusion caused. Our emphasis on distinguishing our work from Schaeffer et al. (2023) arose from our initial feedback after share our work with colleagues, where some perceived our approach as another version of ‘smooth metrics’, which led to doubts about the practicality of our method for predicting actual metrics of interest. Our contrast was specifically directed only at this aspect of their work, not their other conclusions, which we believe may have led to misunderstandings among readers.

- We also acknowledge our oversight in not sufficiently recognizing Schaeffer et al.'s discussion on resolution. In our revised paper, we will duly credit their insights on this aspect and state that our approach delves deeper into enhancing resolution based on the observation of Schaeffer et al. In fact, unlike Schaeffer et al., who increase resolution by expanding test data (an impractical approach for real test sets), our method improves resolution without the need for additional test data. This innovation allows for the prediction of more challenging tasks and represents the first open-sourced success to use this approach for predictable scaling.

Reviewer comment (Evaluation):
The authors assume unbounded computational resources in their PassUntil evaluation strategy, but in experiments they set a finite upper bound on the number of samples. Consequently, the Pass Until (PU) score they discuss is **not** the PU score they numerically compute. They should either use PU as defined or offer a correction if a limited compute budget exists.

Author response:
Regarding the second and third questions about the PassUntil method, we will address them together:
- We apologize for the confusion caused by our erroneous use of the term 'unbiased' estimate. During our method's design, we were aware of the 'biased' statement on Wikipedia. Our initial solution was to move the expectation to the denominator. With this remedy, the math itself is correct: $E[K] - r = E[f] = r(1-P(s))/P(s) = r/P(s)$, thus $P(s) = r/E[K]$.  Practically, if we repeat experiments, we first take expection of $K$, then get $P(s)$.  However, upon further reflection, we realize that this make $r/E[K]$ as a constant, rather than a random variable for estimating $P(s)$, makes 'unbiased estimate' an inappropriate term.
- Regarding the MVUE estimate you proposed, we find it theoretically sound but encountered an issue when setting $r$ to 1, resulting in an MVUE of $0$ for $P(s)$ as calculated by $(r−1)/(K−1)$. **We seek your guidance for a suitable MVUE when r=1**. (Negative Binomial Distribution itself doesn’t require $r>1$). Once that is solved, we would be happy to migrate the whole method to (r-1)/(K-1).
- Our current solution for the statement in the paper involves **shifting from 'unbiased estimate' to 'maximum likelihood estimate' (MLE).** The MLE, particularly $r/K$, suffices for most scenarios, including ours. 
- Notably, fixing the sample number $K$ to a large constant value and make $r$ a random variable also renders $r/K$ an effective MLE. This might partly resolves your concern about the case of PU score when computation limit exists. i.e., if we set all runs to sample for a fixed times $K$, and observe the number of success $r$, the MLE we get is still $r/K$. I wonder if this explanation might be satisfying to you.

Reviewer comment (Evaluation):
Table 2 should be converted to a figure and should include confidence intervals or standard errors.

Author response:
Yes, we highly agree. After we have submitted our paper, we also thought that the word ''emergent'' has been used to describe the sudden rise in performance and it's not very appropriate to re-define it and causing misunderstandings. We consider to name the phenominon of ''super-scaling law behavior'' as ''**accelerated emergence**'' in the revised version. With that definition, emergence can still be empirically understood as ''exponential-like growth'' in task performance. And among which,  a part of emergent task still obeys the task scaling law. While we are discovering the new case of emergence that grows faster than ``scaling law-induced emergence''.

Reviewer comment (Novelty):
On a higher level, the Figure 6 contribution is questionable. Previous emergent papers did not emphasize this BIG-Bench task, and it is not well known as an emergent ability.

Author response:
- Yes, it’s an assumption, we will make this clear as you suggested.

Reviewer comment (Experiments):
The task specifies 8 subtasks, but only 4 are considered in the manuscript, with no explanation of selection criteria.

Author response:
- Actually, the trade-off between actual sampling numbers and time cost for each sample is interesting. For small models, they need more sampling iterations, but each sample is super fast. For large models, their generation is slow, but they success within a few samples. For 0.03b model, the whole test time for HumanEval is 40 min. For 2.4b model, it is 2h.  All evaluations run parallelly on 8 A100 GPUs.

Reviewer comment (Evaluation):
The scaling curves are fit with no held-out data to test generalization, raising concerns about overfitting.

Author response:
We have mentioned it in limitation section. We will make it clearer in revision.

Reviewer comment (Experiments):
Limiting the number of samples ($10^5$) may make the PU estimator biased. Since PassUntil’s promise is unlimited sampling, this limitation needs correction or consideration.

Author response:
Regarding the second and third questions about the PassUntil method, we will address them together:
- We apologize for the confusion caused by our erroneous use of the term 'unbiased' estimate. During our method's design, we were aware of the 'biased' statement on Wikipedia. Our initial solution was to move the expectation to the denominator. With this remedy, the math itself is correct: $E[K] - r = E[f] = r(1-P(s))/P(s) = r/P(s)$, thus $P(s) = r/E[K]$.  Practically, if we repeat experiments, we first take expection of $K$, then get $P(s)$.  However, upon further reflection, we realize that this make $r/E[K]$ as a constant, rather than a random variable for estimating $P(s)$, makes 'unbiased estimate' an inappropriate term.
- Regarding the MVUE estimate you proposed, we find it theoretically sound but encountered an issue when setting $r$ to 1, resulting in an MVUE of $0$ for $P(s)$ as calculated by $(r−1)/(K−1)$. **We seek your guidance for a suitable MVUE when r=1**. (Negative Binomial Distribution itself doesn’t require $r>1$). Once that is solved, we would be happy to migrate the whole method to (r-1)/(K-1).
- Our current solution for the statement in the paper involves **shifting from 'unbiased estimate' to 'maximum likelihood estimate' (MLE).** The MLE, particularly $r/K$, suffices for most scenarios, including ours. 
- Notably, fixing the sample number $K$ to a large constant value and make $r$ a random variable also renders $r/K$ an effective MLE. This might partly resolves your concern about the case of PU score when computation limit exists. i.e., if we set all runs to sample for a fixed times $K$, and observe the number of success $r$, the MLE we get is still $r/K$. I wonder if this explanation might be satisfying to you.

Reviewer comment (Writing):
The definition "Emergence Still Exists But Becomes Continuous." may conflict with Wei et al. 2022’s definition. It may be clearer to title this section "An Alternative Definition of Emergence" or "A Quantitative Definition of Emergence".

Author response:
Yes, we highly agree. After we have submitted our paper, we also thought that the word ''emergent'' has been used to describe the sudden rise in performance and it's not very appropriate to re-define it and causing misunderstandings. We consider to name the phenominon of ''super-scaling law behavior'' as ''**accelerated emergence**'' in the revised version. With that definition, emergence can still be empirically understood as ''exponential-like growth'' in task performance. And among which,  a part of emergent task still obeys the task scaling law. While we are discovering the new case of emergence that grows faster than ``scaling law-induced emergence''.

Reviewer comment (Theory):
Regarding Theorem 2, the assumption that each reasoning step’s success rate obeys scaling law growth may not fit reality. LLMs have conditional dependencies such that correctness of early tokens increases the likelihood of later correctness. This assumption leads to geometric decay not observed in practice, as seen in Schaeffer et al. 2023 Figure 4.

Author response:
It is not contradict to our theoreom 2. The misunderstanding is indeed caused by our attempt to overwrite an exstablished concept. Multiple step reasoning, or complex skills for solving a task, will make the task scaling law's slope steeper in the log space, which results in a steeper  ''exponential form'' in the task performance. However, our defined ''accelerated emergence'' (originally named as emergence), is **not about the speed**, but about the **acceleration**. The acceleration is only distinguishable after the mathematical form has been established. And we actually means that ''multistep reasoning'' does not lead to super-scaling law behavior, a.k.a., accelerated emergence. We will make it cristal clear in the revision.

Reviewer comment (Theory):
Theorem 3’s statement that success rate is the majority vote of circuits is unclear. If majority voting is an assumption, the language should specify "if". If not, a justification for this decision-making process is needed.

Author response:
- Yes, it’s an assumption, we will make this clear as you suggested.

Reviewer comment (Evaluation):
Some LLM evaluation processes (e.g., MMLU or temperature=0 greedy decoding) do not involve sampling. In such cases, does PassUntil make sense? If not, a limitation section should be added.

Author response:
We have mentioned it in limitation section. We will make it clearer in revision.

Reviewer comment (Writing):
The title’s preposition “from” in "Unlock Predictable Scaling from Emergent Abilities" may be unclear. A clearer, more engaging alternative could be "Predicting Emergent Abilities of LLMs with Infinite Resolution Evaluations".

Author response:
We buy it! Thanks a lot for the excellent title!

Reviewer comment (Experiments):
The number of evaluation datasets used in this study is somewhat limited in comparison to previous works on scaling laws. (Please see Question 2)

Author response:
Our choice of evaluation tasks can be attributed to two primary considerations:
- 1. Our method is currently designed for tasks where a random guess would typically result in a zero score, particularly in smaller models. This limitation is detailed in Section 4.1, Comment 2.  However this does not compromise the general applicability of our approach. Generative tasks are inherently versatile, making our method suitable for a variety of applications. Moreover, other metrics could be build upon our method. For example, a possible solution for multiple choice grade metric is to track each choice’s PassUntil with task scaling law or just track the Brier Scores. 
- 2. Another consideration of our task selection from practical perspective is that the task can be evaluated on a simple pass/fail basis. While our method is capable of processing generative tasks that requiring complex human evaluation, these would necessitate the support of an answer-judging model, which will complicate our initial attempt for predictable scaling.

We must point out that **observing** scaling behavior (what Wei et al.2022a do) is easier than **predicting** it.  Despite that, we are currently expanding our scope of prediction!

Reviewer comment (Theory):
The task scaling law seems somewhat arbitrary, as some tasks require standard scaling laws while others necessitate super scaling laws. (Please refer to Question 3)

Author response:
Equation (4) was build upon very strong assumption (each token’s scaling speed is the same, and there is one circuit to get the answer passed in terms of neural circuit), without which we can not achieve any meaningful formulation of the scaling properties of task performance. We can see there are practical tasks (HumanEval, Date Understanding, etc.) do show such scaling properties.  However, since the assumption is so strong, we are not surprised to see the cases (in figure 6) that does not obey the assumption.  In fact, the whole section 6 is to discuss the more general case of task scaling law. We can see that if each token’s scaling speed is not the same, it must induce a sub-scaling law growth instead of super-scaling law growth. Which is not what we observed in experiments. So we propose another hypothesis based on neural circuits. The methodology we have adopted is not arbitrary or ad-hoc. Instead, it represents a systematic and gradual approach to addressing a complex scientific problem.

Reviewer comment (Theory):
Scaling laws are generally in an average sense over a dataset but not for individual tokens. For example, in a single generation, an error incurred in earlier tokens is likely to cause errors in subsequent tokens.

Author response:
- A2). While scaling laws are typically applied in an average sense, this does not preclude their adaptation to individual token levels. Theoretically, a scaling law implies linearity in the logarithmic space of model parameters/compute and the logarithmic space of loss. If single-token losses were not linear in this log space (There must be more random noise at instance-level, though), it would be challenging to understand the observed linearity on average. This is because numerous non-linear structures would dominate the scaling process, leading to non-linearity in aggregate. To provide a more rigorous analysis, prompted by your question, we are examining the scaling relationship of single-token loss. We do linear regression `log(loss) ~ log(modelsize)` on each token of  HumanEval's ground-truth solution. And we count the R^2's distribution. which is 

|     R^2    | 0-0.2 | 0.2-0.4 | 0.4-0.6 | 0.6-0.8 | 0.8-1.0 |
|---------|-------|---------|---------|---------|---------|
| Frequency | 0.17  | 0.12    | 0.15    | 0.25    | 0.29    |

So, at token level, randomness indeed dominate the loss, but there are still a considerable tokens obeying scaling law. Therefore, we think the assumption is reasonable at the deduction stage. A careful dive into the R^2 interval provide us the observationn that R^2=0-0.2 contains the tokens that are either too easy or too hard to predict.

Reviewer comment (Experiments):
This paper evaluates the "emoji," "date," and "identity" tasks. However, the number of evaluation datasets seems relatively small compared to existing works like Wei et al., 2022a. As a result, it is uncertain whether the conclusions drawn in this paper are broadly applicable. Could you please discuss the generalizability of your findings?

Author response:
Our choice of evaluation tasks can be attributed to two primary considerations:
- 1. Our method is currently designed for tasks where a random guess would typically result in a zero score, particularly in smaller models. This limitation is detailed in Section 4.1, Comment 2.  However this does not compromise the general applicability of our approach. Generative tasks are inherently versatile, making our method suitable for a variety of applications. Moreover, other metrics could be build upon our method. For example, a possible solution for multiple choice grade metric is to track each choice’s PassUntil with task scaling law or just track the Brier Scores. 
- 2. Another consideration of our task selection from practical perspective is that the task can be evaluated on a simple pass/fail basis. While our method is capable of processing generative tasks that requiring complex human evaluation, these would necessitate the support of an answer-judging model, which will complicate our initial attempt for predictable scaling.

We must point out that **observing** scaling behavior (what Wei et al.2022a do) is easier than **predicting** it.  Despite that, we are currently expanding our scope of prediction!

Reviewer comment (Theory):
The scaling laws in Equation (4) are for general tasks, irrespective of whether they involve multi-step reasoning. Ideally, they should be applicable to all tasks. However, Figure 6 demonstrates that it does not work well for multi-step reasoning tasks, and the authors propose another super scaling law to address this issue. It makes this paper a bit ad-hoc.

Author response:
Equation (4) was build upon very strong assumption (each token’s scaling speed is the same, and there is one circuit to get the answer passed in terms of neural circuit), without which we can not achieve any meaningful formulation of the scaling properties of task performance. We can see there are practical tasks (HumanEval, Date Understanding, etc.) do show such scaling properties.  However, since the assumption is so strong, we are not surprised to see the cases (in figure 6) that does not obey the assumption.  In fact, the whole section 6 is to discuss the more general case of task scaling law. We can see that if each token’s scaling speed is not the same, it must induce a sub-scaling law growth instead of super-scaling law growth. Which is not what we observed in experiments. So we propose another hypothesis based on neural circuits. The methodology we have adopted is not arbitrary or ad-hoc. Instead, it represents a systematic and gradual approach to addressing a complex scientific problem.

Reviewer comment (Theory):
Theorem 2 is based on the premise that emergent abilities are induced by multi-step reasoning. However, Appendix E of Wei et al., 2022a includes numerous non-reasoning tasks that exhibit emergent abilities.

Author response:
That ``emergent abilities are induced by multi-step reasoning'' is not an assumption made by this work, it is made by Wei et al.2022a in Section 5.1. We are refuting such assumption by formalizing it into a theorem to check it correctness. The non-reasoning tasks in Appendix E of Wei et al. indeed support our view against multi-step reasoning assumption.

Reviewer comment (Experiments):
In the evaluation of PU, both beam search and random sampling decoding are utilized. Does the temperature setting influence the performance, and if so, how is this factor taken into account?

Author response:
Temperature affects the performance. However, currently we didn’t take into account the variation of temperature during model scaling, we use the same temperature for the small LLMs as the large LLMs, which is a natural choice considering the current prevalent use of random sampling in LLMs' generation.