Paper: FILI: Syntax Repair By Learning From Own Mistakes (ICLR-2024)

Reviewer comment (Experiments):
Fili uses a beam width of 30, while Bifi uses a beam width of 10. Bifi is a somewhat more involved model though. Are the FLOPs used to train equivalent between these models? Appendix A.3 has Fili with a beam size of 10: why was this not chosen as the model evaluated in the paper (for fairness with Bifi)?

Author response:
**comparison in #parameters and #FLOPs of the BiFi and Fili models and beam widths**

To perform a fair comparison with BIFI, we adopt the exact same settings in terms of synthetic data generation and the model architecture, as employed by BIFI. Consequently, the number of parameters for the fixer (encoder-decoder) is identical for both BIFI and FILI. However, as we skip training the breaker model, we reduce the total number of parameters by half compared to BIFI.

Since FILI relies on the beam for generating additional data, we use a slightly higher beam-width than BIFI during inference. This adjustment is specific to the inference process, and the impact of beam-width on FLOPs is relatively small compared to the full breaker model used by BIFI for generating additional data. Overall, FILI exhibits significantly lower FLOPs than BIFI, as it avoids training a complete 13 million parameter model.

Reviewer comment (Experiments):
The LLM experiments are zero-shot and do not include GPT-4, but still surpass the proposed approach in the accuracy (without edit distance) metric. As for accuracy with edit distance, the LLM's prompt ("Fix all the syntax errors to make the program parsable") does not include the statement that the program should remain otherwise unchanged or that the edit distance should be minimized.

Author response:
**GPT-4 Experiments:**

Due to budget constraints, we could not run inference on the full test-set (15055 examples) using GPT-4. For comparison, we run a small scale experiment by randomly sampling 2000 programs from the test-set and prompting the model in zero-shot setting using the following prompt:  
The following python program has syntax errors. Fix all the syntax errors to make the program parsable. Do not make any other changes to the program. Just fix the syntax errors.

'''

<Incorrect Program>:  
```
{{incorrect}}
```
<Correct Program>:

'''

Results:  
| Model | Edit = 4 | Edit = 6 | Edit = 8 | Edit = 10 | Edit = inf |  
|-------|----------|----------|----------|-----------|------------|  
| GPT-4 | 57.3%    | 73.25%   | 80.95%   | 86.95%    | 99.75%     |

The observations from this experiment align with those using other LLMs. The model shows good performance in fixing syntax errors but introduces several changes to other parts of the program, which might be undesirable in real-world scenarios. Furthermore, both the training and inference processes of these models are orders of magnitude more computationally expensive (and even costly) compared to our model.

Reviewer comment (Evaluation):
What is the comparison in #parameters and #FLOPs of the BiFi and Fili models in the evaluation?

Author response:
**comparison in #parameters and #FLOPs of the BiFi and Fili models and beam widths**

To perform a fair comparison with BIFI, we adopt the exact same settings in terms of synthetic data generation and the model architecture, as employed by BIFI. Consequently, the number of parameters for the fixer (encoder-decoder) is identical for both BIFI and FILI. However, as we skip training the breaker model, we reduce the total number of parameters by half compared to BIFI.

Since FILI relies on the beam for generating additional data, we use a slightly higher beam-width than BIFI during inference. This adjustment is specific to the inference process, and the impact of beam-width on FLOPs is relatively small compared to the full breaker model used by BIFI for generating additional data. Overall, FILI exhibits significantly lower FLOPs than BIFI, as it avoids training a complete 13 million parameter model.

Reviewer comment (Evaluation):
The bottom-line improvement over BIFI is not significant. I do appreciate that it is hard to improve every basis point beyond 90.5% obtained by BIFI. I also understand that this is obtained without a Breaker model.

Author response:
**Hypothesis for no improvement in test-set performance after 2 rounds:**

We conducted a manual inspection of examples that both BIFI and FILI cannot fix. We randomly sampled 15 incorrect programs from the test set and then checked whether these programs could be corrected or not. Through this manual inspection, we discovered that only 6 out of these 15 could be repaired by fixing the syntax errors. In the remaining 9 instances, we observed that they are not solely syntax repair problems. For instance, some of these programs are incomplete and require generating additional program statements for correction. Similarly, some of these programs involve deleting specific program statements, which neither FILI nor BIFI is trained to perform. Finally, we noted that some programs were written in other languages such as C++ and Java. This observation indicates that the upper limit for success on this test set is not 100%, and a careful analysis is needed to determine how many programs are actually fixable. Consequently, a 1% improvement over BIFI might indeed be reasonable within this context.

Reviewer comment (Novelty):
You write "A key contribution of our work is to significantly simplify the process of training a syntax fixer of (slightly) higher quality than prior work (viz., BIFI)." - is this process a bottleneck for applying the technique? What is the cost/barrier for applying BIFI that is significantly improved by FILI?

Author response:
**cost/barrier for applying BIFI that is significantly improved by FILI?**

With our ablation study, we demonstrate that curriculum learning is helpful in generating more parsable programs.  From our experiments, we observe that curriculum learning helps improve the performance of BIFI also. Training the full BIFI model with curriculum would require more careful analysis, owing to the added complexity introduced by the breaker model. Additionally, it’s important to note that there exists several options to consider when deciding whether to apply curriculum learning to the breaker or the fixer, as well as how to effectively combine the data obtained from the beam and the breaker.

Reviewer comment (Evaluation):
Do you have any hypothesis on why you did not see further improvement beyond two rounds?

Author response:
**Hypothesis for no improvement in test-set performance after 2 rounds:**

We conducted a manual inspection of examples that both BIFI and FILI cannot fix. We randomly sampled 15 incorrect programs from the test set and then checked whether these programs could be corrected or not. Through this manual inspection, we discovered that only 6 out of these 15 could be repaired by fixing the syntax errors. In the remaining 9 instances, we observed that they are not solely syntax repair problems. For instance, some of these programs are incomplete and require generating additional program statements for correction. Similarly, some of these programs involve deleting specific program statements, which neither FILI nor BIFI is trained to perform. Finally, we noted that some programs were written in other languages such as C++ and Java. This observation indicates that the upper limit for success on this test set is not 100%, and a careful analysis is needed to determine how many programs are actually fixable. Consequently, a 1% improvement over BIFI might indeed be reasonable within this context.

Reviewer comment (Evaluation):
In Table 2, two accuracy scores (the last two columns) are reported. Can you elaborate on the key difference? Why is there a sharp drop for all approaches, especially GPT-3.5-turbo?

Author response:
**In Table 2, two accuracy scores (the last two columns) are reported. Can you elaborate on the key difference? Why is there a sharp drop for all approaches, especially GPT-3.5-turbo?**

The first column corresponds to the parse rate without considering the edit distance between the incorrect source program and the predicted program from the models. The models can introduce an arbitrary number of changes to the programs, which is undesirable. An ideal syntax fixer should solely address syntax errors in the programs, leaving the other parts unchanged. The second column corresponds to the parse rate with the edit distance, wherein the predicted program is considered correct only if it is both parsable and below a certain edit-distance threshold. Specifically, we set the edit-distance threshold to 4 to align with BIFI's evaluation criteria.

The significant difference in language model performance between the two columns is because it is challenging to constrain the model's output using natural language prompts to only rectify syntax errors. In addition to addressing syntax errors, these models end up making numerous changes to other parts of the code. Notably, they introduce stylistic changes and alter variable declarations by combining them, among other things. A detailed explanation, along with examples, is provided in Appendix A.6 regarding the outputs from these pre-trained language models.