Paper: Does GPT-4 have good intuition about functions? (ICLR-2024)

Reviewer comment (Experiments):
It is not clear whether the performance is due to memorization in the training data, generalization, or in-context learning. Since transformers can learn functions in-context performing a form of meta-learning (see references below), it will be important to compare with this baseline. Furthermore, the data contamination prevention by perturbing some words does not seem to be adequate. The authors are encouraged to try out more rigorous techniques such as those described in “Proving Test Set Contamination in Black Box Language Models” (Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, Tatsunori B. Hashimoto).

Author response:
**Response:** Thanks for raising this very interesting question. There can be a multitude of driving factors for the performance in our case. Firstly, considering the training instances, we believe that it is very unlikely for the model to have seen the exact same points since we are sampling from a noisy distribution of the underlying function (using additive Gaussian noise, renormalization, as well as feature renaming and merging), therefore, it is likely to be model generalization. On the other hand, for domain knowledge, it should most likely be just memorization as it underlies the rules of the domain. Also, in some cases like direct prediction for income as well as CO_2 datasets, in-context learning might be playing a crucial role, which might even just be task recognition instead of learning the task itself via in-context learning as argued in prior work (Pan et al. 2023). However, investigating the exact nature of this seems beyond the scope of our current work, which instead attempts to just evaluate the capabilities of GPT-4 on general function modeling tasks. The reviewer’s suggestion might be a great avenue for future work.

*Pan, J., 2023. What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (Doctoral dissertation, Princeton University).*

**Response:** We acknowledge the importance of developing rigorous approaches for evaluating test set contamination. However, we find that the methodology suggested in the reference is not a necessary indicator of dataset contamination in our context. In particular, the reference focuses on the impact of dataset ordering on perplexity as an indicator of contamination. While significant perplexity variations may indeed signal contamination, the absence of such variations does not guarantee the absence of contamination.  Also, we want to note that order of the sequence is particularly important in our applications as we consider time-series sequences. We highlight that our efforts to mitigate data leakage were comprehensive and may be the best one can do. Specifically, we have introduced several techniques, ranging from adding random noises, feature name obfuscation, feature merging, normalization, and beyond. These are much more comprehensive than the reference mentioned.  Finally, we would like to highlight that the reference mentioned by the reviewer came out last month i.e., after the submission deadline.

Reviewer comment (Experiments):
The paper only focuses on GPT-4. It will be important to study open source models such as LLaMA2. Comparing and contrasting LLaMA2 can provide more insights on the origin of this capability.

Author response:
**Response:** We agree that our evaluations are particularly focused on GPT-4, and studying open-source models such as LLaMA-2 can be interesting. Instead of drawing generalizable insights about capabilities of LLMs in modeling functions, our aim was instead to identify the progress that we have made in terms of function modeling with the most advanced LLM to date. Therefore, we just focused on GPT-4. While evaluating other models is valuable and can provide information about the open-source alternatives when function modeling capabilities are desired, we consider this to be beyond the scope of our current work.

Reviewer comment (Evaluation):
The evaluation takes a qualitative approach and yet the authors draw general conclusions.

Author response:
**Response:** We believe that we have both qualitative as well as quantitative results (see e.g. Table 1 and 3 for Income prediction, and Table 2, 4 for CO2 and Currency, respectively). Furthermore, we draw conjectures rather than broadly applicable conclusions about the capabilities of GPT-4 based on our experiments. It is impossible to know the test distribution when considering natural language interfaces, and hence reliably predict model performance.

Reviewer comment (Experiments):
The paper is motivated by asking whether GPT4 has some knowledge of simple functions (an example is given of the trajectory of a cannonball), akin to human capabilities. In my opinion this is a solid research question, but I am concerned that many of the experiments have only a tenuous connection to this question. In particular, I am thinking of feature selection, income classification, and currency prediction tasks. These seem to be essentially classic ML tasks, without much connection to function learning per se (beyond the vacuous connection that any supervised learning problem can be cast as learning a function from inputs to outputs – but this is quite different from, e.g. the motivating example of the cannonball trajectory). Considering that the authors motivate their work by comparing to human abilities this is a bit jarring – is there any evidence that humans are especially good at these kinds of tasks? Note that for the synthetic function prediction tasks, there actually is fairly detailed psychological data available on peoples’ performance.

Author response:
**Response:** We thank the reviewer for the interesting comment. We believe that tasks like income classification are still highly correlated to our core research question. This is because any relationship y = f(x) can be regarded as a function, both for regression (like the one used to model ball trajectory) as well as classification (like the one in the income task). The only difference is whether the target y is continuous or discrete. (b) In terms of tasks like feature selection, it is useful in reflecting one’s understanding about function. For example, if the underlying function is almost linear, then knowing which features are important are equivalent to knowing which weights are large in the linear function. Similarly, in the currency prediction task we use kernels designed by GPT-4, which can also show how the model understands the relationship between data points. In our research, we find methods like feature selection often provide additional insights about how LLMs understand a particular function. We believe direct prediction just covers one facet of the full function modeling capabilities that we intend to capture.


**Response:** Although we motivated our study from human ability, we intentionally avoid any direct comparisons, as we focus on the difference between function modeling with and without domain knowledge for machine learning models. An apples to apples comparison against human subjects is beyond the scope of our work.

Reviewer comment (Novelty):
This is a bit more subjective, but I can't help but feel that, even absent the issues described elsewhere in this review, that this paper doesn't make much of an original contribution beyond answering "how well can GPT do X" for yet another value of X. We already know that GPT can do a lot of things quite well, and considering it was trained on data generated by humans, it is not too surprising that it can recapitulate some human abilities (such as understanding simple functions). Similarly, is it really surprising that GPT can make better predictions when it is given domain knowledge about the function than when it is not? Now, I certainly don't mean to denigrate the field of "GPT studies" here – carefully cataloguing and understanding the capabilities of this system is surely of great interest. But by the standards of making "new, relevant, impactful knowledge", I unfortunately don't think the paper meets this bar, as it doesn't seem as though there are any especially novel prompting techniques, analysis methods, or tasks introduced in the paper.

Author response:
**Response:** It is true that there is already a plethora of work investigating ‘how GPT can do X’ on different tasks X. However, our work is substantially different in that (a) we try to investigate a more fundamental problem, i.e., which kinds of function modeling problems can be tackled by GPT, specifically when taking into account the capability to seamlessly integrate domain knowledge (see e.g. the synthetic task in our experiment). This provides a new perspective to understand these ‘can do X’ works; and (b) we are the first to study the function modeling ability of GPT from a Bayesian perspective, where we explicitly decouple the ability to recognize patterns from raw data and the ability to integrate with prior knowledge learned from the internet. Importantly, our work highlights the potential of LLM in modern data science, specifically its potential in cases where data is scarce or domain knowledge is important. Finally, we would like to point out that our evaluation framework is also new: we evaluate GPT’s understanding about function from different angles, using both direct methods (checking output accuracy) and indirect methods (feature selection/kernel design). The insights revealed by these different methods are complementary and cover a broad range of different applications.

Reviewer comment (Experiments):
From Pg 7. For the feature selection task, it is not fair to Yamada et. al’s method to claim that GPT4 is “more data efficient” when you did not actually test Yamada’s method on the smaller dataset! If you want to make this claim, you would need to control the dataset size for both methods, for example evaluate both GPT4 and Yamada on one dataset of size 10^4 and another on 10^2 (or whatever other numbers seem appropriate).

Author response:
**Response:** We thank the reviewer for pointing this out. We have now also provided the results for Yamada’s method in the small data regime. The results show that Yamada’s method is inferior to GPT-4 when using 10^3 samples, achieving an accuracy of ~80% compared to ~83% for our method. When considering 10^2 samples, which is the same case as our method, the method from Yamada et al. fails to perform sensible feature selection (it does not select features at all). We note that this is expected as Yamada’s method works by training a neural network, which is typically data hungry.

Reviewer comment (Evaluation):
On this same task, it needs to be clarified exactly how the evaluation was done, simply saying “Accuracy is measured by an MLP trained using 10^4 samples” is not sufficient. Was the data used to select the features separate from the data used to train the MLP? Is the “accuracy” here the training accuracy or test accuracy? Why are there error bars on the accuracy, but only a single set of features reported for each method? Does this mean that you only ran each feature selection method once, if so why?

Author response:
**Response:** The data used in feature selection and training classifier is the same. “Accuracy” in the table means test accuracy. The error bars in the table correspond to different runs of ‘feature selection + subsequent classifier training based on the selected features’. In this process, the feature selection results are often stable, however the performance of the trained classifiers can vary across different runs due to e.g. different initialization/optimization dynamics. This is why there are error bars even if the same set of features are selected.

Reviewer comment (Theory):
When asking GPT to describe its decision rules for the CO2 task, how much credence should we place in this answer and why? After all, the propensity of GPT to hallucinate is so well-known that it is often written about in the NY Times. For example, the "increasing rate of change" does not appear to even be present in the data, as the authors point out. But the more fundamental issue is that, even for the features that are actually present in the data, GPT may possibly be misrepresenting its own decision process (i.e. its predictions might actually depend on some other feature of the data than what it says). Is there reason to think that GPT can accurately introspect into its own decision process?

Author response:
**Response:** That’s a great question. We completely agree that the model can misrepresent the actual decision rule that it is using. Therefore, this evaluation just validates that GPT-4 has knowledge about the domain, rather than validating what GPT-4 is doing under the hood (Huang et al. 2023 attempts to do a systematic comparison against classical interpretability techniques). We updated the paper to make this clear.

*Huang, S., Mamidanna, S., Jangam, S., Zhou, Y. and Gilpin, L.H., 2023. Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations. arXiv preprint arXiv:2310.11207.*

Reviewer comment (Experiments):
In the currency exchange rate task, the authors state “there is a risk of data contamination in this case”. Why didn’t they apply the same mitigation as in the CO2 task (i.e. add random noise), or seemingly any mitigation at all?

Author response:
**Response:** We appreciate your attention to the issue of data contamination in our currency exchange rate task. To mitigate this, we implemented a two-pronged approach similar to our CO2 task methodology. First, we introduced random noise into the raw values to obscure direct data correlations. Second, we removed specific identifiers such as currency and metal names that could directly link the data to our training set (we mistakenly included the different version of the prompt, which was not used in the experiment in Appendix c.4, which we have fixed in the recent version – we apologize for the confusion that it created). This dual approach ensures the integrity of our data while maintaining its usefulness for training. We acknowledge that this was not explicitly stated in our initial submission, and we will clarify this methodology in the revised manuscript.

Reviewer comment (Reproducibility):
“I cannot use normal GP directly since there are missing values (section c.4)”: A GP model can seamlessly handle arbitrary patterns of missing data so I am not sure what is meant here. There are many other examples of important technical details in the experimental descriptions which are not described – specifics are given in the Questions section.

Author response:
**Response:** We are thankful to the reviewer for highlighting this, and apologize for the confusion. We highlighted the deficiency of normal GP in this prompt as we would also like to also see whether GPT-4 can come up with solutions beyond default missing data impulation techniques. For example, a good solution will consider the correlation between different dimensions rather than treating them separately during missing data imputation. The corresponding result was not included in the final manuscript for conciseness.

Reviewer comment (Writing):
Overall I found the writing quality and organization of the paper to be poor, here are specific examples:
- “in contrast to conventional machine learning models (pg.2)” – This claim is way too strong relative to the actual results; as far as I can tell, the only other model that was systematically evaluated was a small MLP.
- “three precious metals (pg.4)” – Precious metals are not mentioned in any preceding text.
- “venturing into diverse domains extending from physics (pg.1)” – There is no data presented on a physics task.
- “In this work, we ask a more fundamental question of how capable these models are in terms of representing functions at a general level (pg.9)” – What does it mean to represent a function at a “general level”, and how did the previously-mentioned papers fall short at this?
- Section 2 seems superfluous to me – as the formal Bayesian framework is never used thereafter, and the math is well-known enough that I don’t think this section adds anything to the understanding.
- While “Predicting missing parts” is listed as a task in section 3.1, I do not see any actual results for this task presented anywhere in the paper or appendix (many of the prompts however allude to missing parts). The closest thing seems to be the currency task, where the LLM predicts a kernel function which is used to impute missing values via GP inference, but this is not really a direct prediction of missing values.
- Similarly, in Figure 2, there appear to be “ghost tasks” of Ball trajectory prediction and Stock market prediction, which are not actually studied in the paper.
- “(this is relevant for the ‘output dependency modeling‘ task)” – There is no “output dependency modeling” task (ctrl-F “output dependency”).

Author response:
**Response:** We are thankful to the reviewer for the detailed comments regarding the writing and organization of the paper. We addressed all the organizational and writing related issues mentioned by the reviewer, and did another pass over the entire paper, attempting to improve clarity. All recent changes are highlighted in RED in order to make them clear.