Paper: Implicit Intermediate Supervision for Learning Complex Functions (ICLR-2024)

Reviewer comment (Theory):
The theoretical results on the Parity/Sum task rely on some strong assumptions: bilinear parameterization, some initialization (for example, v = 0). Under these assumptions, the gradient over the parity distribution samples is zero. The assumption looks a little strong.

Author response:
Indeed, our architecture and initialization are somewhat non-standard. We use this setting in order to simplify our theoretical analysis, but we believe that similar results also hold for more “standard” networks, at the cost of making the analysis much more involved. Specifically, we would like to note that while our initialization ensures that the gradient over the parity distribution (at initialization) is exactly zero, it can be easily shown (e.g., using an analysis similar to [1]) that for any choice of architecture and initialization, the gradient over the parity is extremely small (that is, exponentially close to zero). Also note that we also report empirical experiments with “standard” choices of network architectures, which validate our theoretical findings.

Reviewer comment (Experiments):
It would be good to show more experimental results for some settings, for example, performance on the Sum task when training with a mixture distribution, or more Sum task samples and fewer Parity task samples (p range from 0.5 to 1). In Figure 1, results with p ranging from 0.1 to 0.5 are shown.

Author response:
Thank you for your suggestions. These experiments can indeed contribute to the paper, and we will add them to the final version of the manuscript.

Reviewer comment (Presentation):
There is something wrong with the structure of this paper, for example, section 1.2 is named as Related Work, which is usually a separate chapter. Moreover, this paper losses the Conclusion part.

Author response:
We will restructure the paper to have a separate Related Works section, and extend the discussion section to include conclusions. We do note that it is very common in our field to have the Related Works as a subsection of the introduction, and to have a combined Discussion and Conclusions section.

Reviewer comment (Presentation):
This paper does not contain an example to show the advantages of implicit supervision over explicit supervision, or an example that demonstrates how the implicit supervision work. So can the authors provide a figure in Section 1 that can make readers get your innovation quickly?

Author response:
The main claim in the paper is that intermediate supervision, which is required for solving some hard tasks, can happen naturally by training on a mixture of tasks. For example, computing the parity function over some subset of bits is hard to learn by itself, but when we train on a mixture of the parity and the sum task (where the goal is to compute the sum of the bits in the subset), learning becomes easy. We call this type of supervision implicit, because nowhere in the training process did we “force” the network to use its ability to compute the sum in order to compute the parity. In contrast, to provide explicit supervision, we could for example construct a dataset which includes “chain-of-thought” reasoning that forces the model to first compute the sum, and then use this result to compute the parity. We claim that implicit supervision appears more naturally in data “in-the-wild” (for example, data randomly scraped from the internet). Such data is very likely to contain many different tasks which can positively contribute to each other, but might not include clean step-by-step supervision for particular problems of interest. Generating such step-by-step supervision usually requires costly human annotations. In this sense, we claim that implicit supervision is “better”. We will update the manuscript to clarify this distinction between implicit and explicit intermediate supervision.

Reviewer comment (Experiments):
Assuming that there are now n tasks, task 1, task 2, …, task n. Will the following scenario occur: training task 1 with data from task 1 to task i yields excellent results, but when training task 1 with data from task 1 to task i+1, the performance significantly deteriorates?

Author response:
Regarding your question: note that we do not argue that all tasks help each other, and it is indeed possible that training on one task harms the performance of another task. Our focus is mainly on showing that in some cases, one task can dramatically improve the performance of another task.

Reviewer comment (Experiments):
The tasks they study are a little bit toy, which makes them easy to study, but it is a little bit unclear if these findings transfer cleanly to the LM pretraining setting.

Author response:
**Choice of Synthetic Datasets:**

In our study, we opted for synthetic datasets primarily due to their simplicity and clarity in defining tasks. While we recognize that this choice might limit the direct applicability of our findings to more complex, real-world scenarios, we believed it was necessary for examining the contribution of implicit intermediate supervision in a methodical way.

We acknowledge the importance of demonstrating that our findings are applicable in real-world LM pretraining settings. To this end, we refer to the recent model, OpenHermes-2.5-Mistral-7B, a state-of-the-art 7B parameter Language Model. During its development, the authors found that mixing a natural language instructions dataset with a code instruction dataset in a good ratio (around 10% of the total dataset) boosts model performance on several non-code benchmarks, including TruthfulQA, AGIEval, and GPT4All suite [1]. This aligns directly with our findings, suggesting that our conclusions are not only theoretically sound but also practically applicable in advanced LLM pre training scenarios.

Further substantiating our findings, existing academic literature, such as [2], also aligns with our conclusions. While the focus of [2] is primarily on the impact of pretraining with code to enhance reasoning capabilities in LLMs, it inadvertently supports our theory regarding implicit intermediate supervision. It is important to note that our work complements these existing studies by providing a theoretical underpinning for the observed empirical benefits, thereby filling a crucial gap in the current understanding of LLM training methodologies.

Reviewer comment (Novelty):
They frame this as a replacement for chain of thought, and state that it saves on collecting full reasoning chains for supervision. However chain of thought prompting (the predominate way to get LM to output intermediate reasoning) only requires a handful of examples that can usually be written by a single person in a few minutes. Whereas their method would require collecting or synthesizing large datasets. I feel that a potentially more interesting framing could be around understanding how large scale pretraining (multitask) can enable LMs to learn tasks which are typically challenging for neural networks to learn on their own.

Author response:
**Framing:**

We agree with the reviewer that the primary contribution of our work lies in settings like pre-training, where intermediate supervision is implicit and does not require the collection or synthesis of additional datasets.

Reviewer comment (Experiments):
Hoes does model scale impact the results? Will larger models be able to learn more effectively from fewer examples of the simpler tasks?

Author response:
1. We agree with the reviewer that the question of how scaling affects implicit intermediate supervision is an interesting question that should be addressed in future work. Note that both our theory and the conducted experiments are in the (relevant) overparameterized regime