Paper: GPT Is Becoming a Turing Machine: Here Are Some Ways to Program It (ICLR-2024)

Reviewer comment (Writing):
The methodology can be elaborated for better clarity.

Author response:
Thank you for the notes about font size, clarifying evaluation metrics, and more clearly identifying the research gap. We will address those notes as well as elaborate on methodology in our final version.

Reviewer comment (Novelty):
The research gaps can be further highlighted and discussed.

Author response:
Thank you for the notes about font size, clarifying evaluation metrics, and more clearly identifying the research gap. We will address those notes as well as elaborate on methodology in our final version.

Reviewer comment (Evaluation):
Section 2.1: “a variant of Bubble Sort algorithm adapted to this problem and shown in Prompt 2 can be used to solve 76% of these puzzles”, the author(s) should provide evaluation results to support this statement.

Author response:
The section 2.1 note on adapting the Bubble Sort algorithm to solve logical puzzles describes that Prompt 2, which was used in evaluating the Logical deduction task, solves 76% of the logical puzzles, and this result is shown in Table 1 in the Logical deduction task under IRSA.

Reviewer comment (Experiments):
I wonder whether the author(s) consider applying IRSA to GPT-4 and test the result accuracy.

Author response:
We have applied IRSA to GPT-4 in a couple instances, described in Section A.3.2 and A.3.3. In section A.3.3, we discuss that GPT-4 is not better than GPT-3 with IRSA on the LCS task. In Section A.3.2, the newly added experiment evaluating expressions of the form (a +- b) % 10 + c, averaging entropy and correctness across many trials to compare two different prompts on both GPT-3 and GPT-4, as we increase the temperature, providing a more direct comparison using GPT-4 with IRSA. The first prompt is a “guess then check” style prompt, where the answer to the expression is given first, followed by a step-by-step reasoning to achieve the answer, and the second prompt is a “reason then answer” style prompt, where the first line has a question mark instead of the answer, followed by the same step-by-step reasoning. 

The guess then check prompt has two examples of the following form:

(24-41)%10+2=5

24-41=-17

-17%10=?

-17 is negative so the answer is 10-7=3

-17%10=3

(24-41)%10+2=5

(The additional example has a positive result in a+-b),
And the “reason than answer” style that IRSA would follow, differs only in replacing the first 5 with a question mark.

Figure A.1 illustrates how the “guess then check” style prompt has just over 60% accuracy with GPT-4 and just less than 40% accuracy with GPT-3, while the IRSA style prompt has 100% accuracy for GPT-4 and near 100% accuracy for GPT-3. Interestingly, comparing the average entropy across 20 trials of 20 expressions shows that as the temperature increases, the entropy increases for both GPT-3 and GPT-4 using the “guess then check” prompt but with the “reason then answer” prompt, the entropy sits at zero for GPT-4 and near zero for GPT-3 regardless of temperature.

Reviewer comment (Experiments):
The author(s) generate random non-repeating digit sequences in the Bubble Sort problem, I wonder whether repeating digits can affect the results.

Author response:
On the repeating digits front, it seems that adjusting the prompt (A.4) just slightly to use <= as the comparison rather than < enables GPT-3 and GPT-4 to correctly execute bubble sort with repeating digits. We did a cursory trial with [3, 6, 8, 8, 7], [8, 8, 8, 8, 8], and [8, 8, 8, 8, 7] and there is no issue with execution. Importantly, though it did require the adjustment to the prompt with <= so that the swap_flag was not set to true for switching repeating digits, leading to an infinite loop. Trying with [3, 8, 6, 8, 7] led to an interesting issue with GPT-3 where once the execution got to <state> a=[3 6 8 8 7] i=3 P=4 n_swaps=1 swap_flag=true </state>, it proceeded to correctly identify the last 8 and 7 to swap, but instead created the state <state> a=[3 6 8 7 7] i=4 P=4 n_swaps=2 swap_flag=true </state>, where it ended up repeating the 7 instead of shaping the 7 and 8. Using skip attention, rerunning the prompt but adding the last correct state right after EXECUTION (not allowing GPT-3 to attend to how it generated that state), fixed the problem. GPT-4 did not run into this issue with the same problem.

Reviewer comment (Presentation):
I understand the conference has page limits, the authors can consider adjusting the font of prompts to save more space to elaborate the solution description, especially Section 2.4. The font of Prompt 1 is larger than the following prompts.

Author response:
Thank you for the notes about font size, clarifying evaluation metrics, and more clearly identifying the research gap. We will address those notes as well as elaborate on methodology in our final version.

Reviewer comment (Presentation):
The authors should clarify the evaluation metric is accuracy in the caption of Table 1-3.

Author response:
Thank you for the notes about font size, clarifying evaluation metrics, and more clearly identifying the research gap. We will address those notes as well as elaborate on methodology in our final version.

Reviewer comment (Novelty):
The author(s) can consider comprehensively comparing the proposed models with related work to clearly identify the research gap.

Author response:
Thank you for the notes about font size, clarifying evaluation metrics, and more clearly identifying the research gap. We will address those notes as well as elaborate on methodology in our final version.

Reviewer comment (Reproducibility):
The example and its execution path play the critical role in IRSA. Isn’t the availability of an execution path too strong an assumption for enabling GPT to execute a program? There are many things not discussed, including how to select the example and how to achieve the execution path. It is also unclear whether there is one example or multiple examples used in IRSA.

Author response:
In terms of prompts for more complex programs, such as those used for bubble sort, LCS, etc., there need not be an already available execution path if the model can create one itself.

The LLM itself can be leveraged to create IRSA style prompts, as we did to create the LCS execution path using the interpreter/compiler Prompt A.2. Otherwise, it would also be feasible to create an execution trace of a program that adds in an explanation between states for each step of the program. The most important features of IRSA are to describe the conditions for continuing and stopping execution, including explanations for state changes in the form “why then what”, and including full state descriptions. Though as skip attention and fragmented prompting shows, these state descriptions need not amount to a full execution path either — they need only describe the possible state transitions.

Our experiments have a variation in number of examples used. With bubble sort, LCS, logical puzzles, and the parentheses tasks shown in Table 1, there are 1-shot prompts. The fragmented prompts on the bubble sort execution have multiple fragments shown in Table 2 (single path and then 7, 13, 19, and 25 fragments), and the ablations in A.3.2 include 4-shot prompts for the if-else experiment and 2-shot prompts for the newly added expressions experiment (showing entropy and correctness as temperature increases for GPT-3 and GPT-4). Table 3 shows the interpreter prompts for synthetic programs with 1-shot, 2-shot, and 3-shot prompts.

Reviewer comment (Experiments):
It is not clear why IRSA is not used together with GPT4, which casts doubts on the applicability of the IRSA approach.

Author response:
IRSA can certainly be used with LLMs other than GPT-3, and we provide comparison against GPT-4 in a now added experiment on evaluating expressions in section A.3.2 in the appendix. Fig. A.1 studies evaluation of expressions of the form (a +- b) % 10 + c, which are hard due to the potential of a negative mod computation and the need to combine multiple computations. But prompting with worked-out examples may not work perfectly without following the IRSA recipe. We look at the difference between prompting in “guess then check” style,

(24-41)%10+2=5

24-41=-17

-17%10=?

-17 is negative so the answer is 10-7=3

-17%10=3

(24-41)%10+2=5

(plus an additional example with a positive result in a+-b),
vs the “reason than answer” style that IRSA would follow, differing only in replacing the first 5 with a question mark.

The only difference is one character per example (5 vs ? in the first line), but as the fig shows, the performance is very different (evaluating 20 randomly generated expressions in 20 trials each). One might assume that with the first prompt, when working on a new expression, the model should guess correctly following the examples, and then even if wrong, it would have a chance of correcting itself at the end, as its generated explanation may change its mind. But, the model often does not guess correctly as the reasoning requires iterative attention that it cannot do without unrolling the reasoning step by step. Then, even after generating a correct follow up step-by-step reasoning and computing correctly (a+b)%10, it has trouble deciding if it should just add c, or follow its initial guess. The second prompt does not create this uncertainty as the model follows the recipe and generates a placeholder ? instead of an initial guess, which later does not conflict with the generated reasoning. The answer entropy quickly rises with temperature (even for GPT-4) for the first prompt, but not for the second. Furthermore, the reduced entropy of the “reason then answer” prompt does not compromise accuracy, with the accuracy of both GPT-3 and GPT-4 at near 100% (sitting at 100% for GPT-4) regardless of temperature, while the “guess then check” style has less than 40% accuracy for GPT-3 and just over 60% accuracy for GPT-4.

A cursory trial of the bubble sort prompts with GPT-4 suggest that the IRSA style prompts triggers the same iterative behavior by GPT-4 as well for more complex algorithms.

Reviewer comment (Reproducibility):
Is there an algorithmic formulation of IRSA? For example, interested people can refer to this formulation to implement IRSA and execute programs using LLMs.

Author response:
The LLM itself can be leveraged to create IRSA style prompts, as we did to create the LCS execution path using the interpreter/compiler Prompt A.2. Otherwise, it would also be feasible to create an execution trace of a program that adds in an explanation between states for each step of the program. The most important features of IRSA are to describe the conditions for continuing and stopping execution, including explanations for state changes in the form “why then what”, and including full state descriptions. Though as skip attention and fragmented prompting shows, these state descriptions need not amount to a full execution path either — they need only describe the possible state transitions.

Reviewer comment (Theory):
Is the execution path assumption in IRSA too strong? How many examples are used in IRSA?

Author response:
Our experiments have a variation in number of examples used. With bubble sort, LCS, logical puzzles, and the parentheses tasks shown in Table 1, there are 1-shot prompts. The fragmented prompts on the bubble sort execution have multiple fragments shown in Table 2 (single path and then 7, 13, 19, and 25 fragments), and the ablations in A.3.2 include 4-shot prompts for the if-else experiment and 2-shot prompts for the newly added expressions experiment (showing entropy and correctness as temperature increases for GPT-3 and GPT-4). Table 3 shows the interpreter prompts for synthetic programs with 1-shot, 2-shot, and 3-shot prompts.

Reviewer comment (Experiments):
Can IRSA be used with LLMs other than GPT-3?

Author response:
IRSA can certainly be used with LLMs other than GPT-3, and we provide comparison against GPT-4 in a now added experiment on evaluating expressions in section A.3.2 in the appendix. Fig. A.1 studies evaluation of expressions of the form (a +- b) % 10 + c, which are hard due to the potential of a negative mod computation and the need to combine multiple computations. But prompting with worked-out examples may not work perfectly without following the IRSA recipe. We look at the difference between prompting in “guess then check” style,

(24-41)%10+2=5

24-41=-17

-17%10=?

-17 is negative so the answer is 10-7=3

-17%10=3

(24-41)%10+2=5

(plus an additional example with a positive result in a+-b),
vs the “reason than answer” style that IRSA would follow, differing only in replacing the first 5 with a question mark.

The only difference is one character per example (5 vs ? in the first line), but as the fig shows, the performance is very different (evaluating 20 randomly generated expressions in 20 trials each). One might assume that with the first prompt, when working on a new expression, the model should guess correctly following the examples, and then even if wrong, it would have a chance of correcting itself at the end, as its generated explanation may change its mind. But, the model often does not guess correctly as the reasoning requires iterative attention that it cannot do without unrolling the reasoning step by step. Then, even after generating a correct follow up step-by-step reasoning and computing correctly (a+b)%10, it has trouble deciding if it should just add c, or follow its initial guess. The second prompt does not create this uncertainty as the model follows the recipe and generates a placeholder ? instead of an initial guess, which later does not conflict with the generated reasoning. The answer entropy quickly rises with temperature (even for GPT-4) for the first prompt, but not for the second. Furthermore, the reduced entropy of the “reason then answer” prompt does not compromise accuracy, with the accuracy of both GPT-3 and GPT-4 at near 100% (sitting at 100% for GPT-4) regardless of temperature, while the “guess then check” style has less than 40% accuracy for GPT-3 and just over 60% accuracy for GPT-4.

A cursory trial of the bubble sort prompts with GPT-4 suggest that the IRSA style prompts triggers the same iterative behavior by GPT-4 as well for more complex algorithms.

Reviewer comment (Experiments):
The use of a trace in the prompt raises a few questions, which are not addressed. A classic sorting algorithm can correctly sort very long lists using a relatively short specification of the algorithm. Can IRSA do the same (sort long input lists with a short trace)? The experiments seem restricted to short inputs length 5 in sorting, for example). The scaling, generalization and robustness of the IRSA prompt to different inputs aren't well explored in the paper.

Author response:
For longer traces, such as the ones in LCS, we introduced skip attention, where the model is forced to ignore all generated tokens except the last generated state. This approach is likely to work better for the purpose of algorithm execution than the techniques for extending the attention span generally in the models themselves. With skip attention, LCS prompts generated much longer token streams.

Re. scaling, generalization, and robustness, a lot of it can be found in (now updated) Sec A.3.2, with an additional experiment in Figure A.1. IRSA style is meant to override model’s own “(over)thinking” and uncertainty, and force it to follow a recipe closely. Otherwise, any generated token that strays from the recipe can lead to at best unnecessary comments as in your experiment, and at worst “hallucination” leading to the wrong answer. In the main we illustrated the importance of IRSA with experiments with different bubble sort prompts, as well as on comparison with scratchpad traces. The appendix further studies the details, including in the new experiment in Fig. A.1. as well as the previous experiment in the section.
Fig. A.1 studies evaluation of expressions of the form (a+-b)%10+c, which are hard for 2 reasons, impeding scratchpad prompts on random programs. 1st, the model often incorrectly computes modulo of a negative number, thinking that -47 % 10 equals -7. 2nd, multiple computations must be combined. But prompting with worked-out examples may not work perfectly without following the IRSA recipe. We look at the difference between prompting in “guess then check” style

(24-41)%10+2=5

24-41=-17

-17%10=?

-17 is negative so the answer is 10-7=3

-17%10=3

(24-41)%10+2=5

(plus another example with a positive result in a+-b)
vs the “reason than answer” style that IRSA would follow, differing only in replacing the first 5 with a question mark.
The only difference is one character per example (5 vs ? in the first line), but as the fig shows, the performance is very different. One might assume that with the first prompt, when working on a new expression, the model should guess correctly following the examples, and then even if wrong, it would have a chance of correcting itself at the end, as its generated explanation may change its mind. But the model often does not guess correctly as the reasoning requires iterative attention that it can't do without unrolling the reasoning step by step. Even after generating a correct follow up step-by-step reasoning and computing correctly (a+b)%10, it has trouble deciding if it should add c or follow its initial guess. The 2nd prompt doesn't create this uncertainty as the model follows the recipe and generates a placeholder ? instead of an initial guess, which later doesn't conflict with the generated reasoning. The answer entropy quickly rises with temperature (even for GPT-4) for the 1st prompt, but not for the 2nd. Temperature is a way to artificially increase uncertainty of generation, but due to the size of the training set and possible contradictions in it, uncertainty (epistemic) is inherent to models even at 0 temp. IRSA is meant to instruct in a way that reduces this uncertainty and increases accuracy.

Reviewer comment (Novelty):
The paper could better highlight its algorithmic and empirical contributions relative to a rapidly growing body of literature on how to improve a LLM's instruction-following abilities. At the moment, I'm not sure if the experiments conclusively demonstrate that IRSA improves the instruction-following (via algorithm execution) abilities of a LLM.

Author response:
Re. the focus on prompting strategy vs algorithmic contributions, we agree that both are important and we will discuss them as such in the final version. Briefly on the prompt engineering front, there are issues like discussed above and in Secs 2 and A.3.2. The algorithmic contributions are skip-attention which reduces the dependency on token limit and reduces the sensitivity to a problem illustrated in Fig. A.2; and fragmented prompting, which allows for exponentially reduced the prompt length (Prompt 3 discussed in Sec. 3)

Reviewer comment (Experiments):
How does the performance of the approach change if the LLM **hasn't** seen traces of the algorithms in its training data or not? For example, it seems plausible that traces of sorting and classical algorithmis might appear in training data. Does IRSA work if the trace is "new" to the LLM? Might synthetic tasks be needed here to eliminate this threat to validity?

Author response:
Re. the execution traces in the training data:
For classical algorithms, the traces were in the training data (try 0-shot Prompt A.10 with higher temp a few times), yet without IRSA, 0-shot code execution doesn't work anywhere near as well (GPT-4 gets only 69% compared to 93% for GPT-3 w/ IRSA, see Sec A.3.3).
Table 3 shows results on new random programs we synthesized, for which traces couldn't have been in the data. Thus, having examples of traces in training likely helps GPT interpret programs, but this is not limited to programs it saw in training. Our experiments show that generalization to new programs gets much better with IRSA, and extension to longer programs benefits greatly from skip attention and fragmented prompting

Reviewer comment (Reproducibility):
I was able to verify that GPT-4 correctly outputs a trace for the example shown in Prompt 1. However, the same prompt on a slightly larger input produces a correct answer but starts to include comments like one might see in code. Is this expected?  

```
<original Prompt 1>
Problem: 0, 3, 3, 1, 2, 10, 98, 2000, 1232, 454422, 001, -222, 4533, 24, 99
EXECUTION
```

starts to produce comments like these  

```
    // The iterations continue until no swaps are made in an entire iteration.

    Iteration:
       set swap_flag=false. 
       // Comparisons occur here for all pairs.
       // ...

       // After several iterations, the list is eventually sorted.
       // No swaps occurred so, swap_flag=false
```

Author response:
Prompt 1 you tried out is less structured, performing worse than prompt A.4 (Table 1). On your problem, with Prompt A.4 GPT-4 follows its style at 0 temperature (did you try at 0?) without inserting comments. However, GPT-4 does shorten the answer after a few dozen steps and jumps to the conclusion, likely due to constraints that OpenAI has imposed (through tuning or brute force) to limit the number of tokens generated by the API/Playground at high comp cost (GPT-3.5-turbo-16k happily goes forever on this prompt).