Paper: Guess & Sketch: Language Model Guided Transpilation (ICLR-2024)

Reviewer comment (Experiments):
This work can be significantly beefed up with a synthetic test set. Evaluation on mere 100s of programs is likely not sufficient. Since it is possible to compile C into both architectures, and since test generation / fuzzing is a well established approach, this work can benefit from an artificial/synthetic test set consisting of about ~1k programs, to evaluate the correctness of the transpiler more thoroughly.

Author response:
**Re: evaluation set is small.**

Thanks for the feedback.  

To further rigorously evaluate the correctness of the GUESS & SKETCH transpiler against the most competitive transpiler baseline, we collected an additional test set of 82 real-world programs, and conducted experiments in both translation directions. The results show that our method correctly transpiles 41.5% of programs from RISC to ARM, compared to 34.1% by the Encoder-Decoder, and 51.2% from ARM to RISC, compared to 37.8% by the Encoder-Decoder. Details are further shared in the Appendix, section B. 

We note that we were limited by the goal of collecting high quality, evaluable programs. The hundreds of programs used in the existing test sets are collected from real mathematic programming challenge problems, computer program performance benchmarks, and realistic interactive programs.

Reviewer comment (Evaluation):
At least we should see confidence intervals of the results, or some kind of t-test to make sure that the proposed method is better than the baseline not due to noise. Kindly ask your colleagues in statistics to look over your tables and give recommendations on how it could be made bullet proof. I would love to see this update in the rebuttal period.

Author response:
**Re: lack of statistical tests**

Thanks for the suggestion.

We ran statistical tests on our results. The top two competitive baselines are the engineered Transpiler and the Encoder-Decoder. The z-score against the Transpiler is 6.89 and against the Encoder-Decoder is 4.57, both of which allow us to conclude with a significance level of 0.01 that GUESS & SKETCH outperforms the baselines. While we agree that thousands of synthetic programs could provide additional support for comparing the transpilation methods, the existing results do show strong statistical significance.

Reviewer comment (Novelty):
While I think this is a good paper, it might be a better fit at PLDI. As I am unsure what is the AI/ML lessons gained from this work, other than it is possible to build such a system, and some relatively detailed findings on how well language models are at learning over a highly stylized text (assembly code) when compared to English sentences. However, as other application papers of the compiler flavour have a precedence of appearing at ICLR, this is not a major concern.

Author response:
**Re: fit of venue**

We agree that cross-disciplinary nature of this work makes it a suitable paper for multiple different venues. We note previous ICLR submissions that explored code generation/translation (Synchromesh, Poesy et al. 2022; Leveraging Automated Unit Tests for Unsupervised Code Translation, Rosier et al. 2022; InCoder, Fried et al. 2023; Binding Language Models in Symbolic Languages, Cheng et al. 2023; DocPrompting, Zhou et al. 2023; etc.) and compiler innovation (DLVM, Wei et al. 2018; Code Translation with Compiler Representations, Szafraniec et al. 2023; etc.) have also been well-received.  Given the machine learning and program synthesis backbone of GUESS & SKETCH, as well as the real-world application of the task proposed, we think that many members of the ICLR audience would be interested in reading and building on this work.

Reviewer comment (Theory):
As I understand program equivalence is an undecidable problem. If I recall correctly, synthesis systems like SKETCH do not have a way to fully verify the SKETCH and the SPEC are identical over all input-outputs, but only over a bounded domain. Is this an issue for your work? Or is it because everything is bounded to begin with, as we're working over assembly and we only need to account for, for instance, 16-bit integers or similar? Or is it some Z3 theory magic that allows for a DSL where programs can be reasoned for full equivalence? At any rate, this should probably be clarified in the paper.

Author response:
**Re: verification of subprogram equivalence in Sketch**

Yes you are correct that this domain needs to be specified for Sketch. We fix the domain of inputs over which Sketch verifies equality. This input domain can be restricted for this domain, specifically by the registers of the assembly architectures used, which in this work is size-64 bitvectors.

Reviewer comment (Evaluation):
If I read correctly, success is measured over a set of input-output test cases to see if the input code runs the same as the compiled output code. Is this related to the program equivalence problem above somehow? Is this comprehensive enough to make sure the transpiling is not mistaken?

Author response:
**Re: measuring success and “program equivalence”**

We use test case input-output equivalence to measure success in the transpilation task. This is not a true measure for program equivalence, but this evaluation setup has been used as a proxy for success in related works in inductive program synthesis (Flashfill, Gulwani 2011; DreamCoder, Ellis et al. 2020) and today's major code generation benchmarks (HumanEval, Chen et al. 2021; MBPP, Austin et al. 2021; APPS, Hendricks et al. 2021).

Reviewer comment (Novelty):
Novelty within this approach is quite limited the translation is a standard approach the confidence is simple (see below), and they use an existing neuro-symbolic solver therefore, it is more on the sole idea of putting these together. This is the main criticism. However, they outperform prior work, and the idea is interesting and technically sound.

Author response:
**Re: novelty**

We agree that the final version of GUESS & SKETCH is quite simple. We experimented with more complex methods, but were able to simplify things down to a few core ideas (model confidence → sketch solver). We believe that this simplicity is an advantage for using this approach in practice.

Reviewer comment (Reproducibility):
Confidence is very trivially explained. In general, deep models are very confident even when they are wrong. It isn't clear how this was implemented and is critical to the method. As the author's rely on this to identify potential errors for solving.

Author response:
**Re: model prediction confidence. is this an imperfect measure of mistakes**

Our goal of identifying potential mistakes is to filter which output subsequences to correct, since correcting every output subsequences is too computationally expensive. Indeed, as previous works have studied (R-U-Sure, Johnson et al. 2023, 'Generation Probabilities Are Not Enough', Vasconcelos et al. 2023), the underlying language model can be very confident about incorrect predictions, so confidence alone may not be sufficient to identify all errors. However, empirically in this domain we found that there was significant signal in this property, as shown by the experimental results. While neither confidence nor attention provide hard guarentees, they both provide signal in this domain. Future versions of GUESS & SKETCH can be further improved by devising better domain-general heuristics for identifying potential errors.

Reviewer comment (Novelty):
Explain more how the confidence is applied and used is it based on prior work as there is significant literature in this area.

Author response:
**Re: model prediction confidence. is this an imperfect measure of mistakes**

Our goal of identifying potential mistakes is to filter which output subsequences to correct, since correcting every output subsequences is too computationally expensive. Indeed, as previous works have studied (R-U-Sure, Johnson et al. 2023, 'Generation Probabilities Are Not Enough', Vasconcelos et al. 2023), the underlying language model can be very confident about incorrect predictions, so confidence alone may not be sufficient to identify all errors. However, empirically in this domain we found that there was significant signal in this property, as shown by the experimental results. While neither confidence nor attention provide hard guarentees, they both provide signal in this domain. Future versions of GUESS & SKETCH can be further improved by devising better domain-general heuristics for identifying potential errors.

Reviewer comment (Experiments):
Does the Lamda hyper-parameter effect the output, was this ablated on, but not included?

Author response:
**Re: ablation on gamma hyper-parameter**

Thanks for the suggestion. 

We ran an ablation study on the gamma hyperparameter with the Project Euler test set. We found that a range of gamma hyper parameters still produces the same results. We sweep across gamma values: 0.8, 0.9, and 0.95 to study this trade-off between computation time and correctness. We find that across these gamma values, the number of corrected programs is the same, but the inference runtime increases with gamma. From 0.8 to 0.95, the inference time increases by 2.2x. This suggests that the model is extremely confident on most tokens, and that even a low-gamma is able to find plausible errors.

Reviewer comment (Experiments):
Why do you choose only the region of error to be solved? Did you consider using a buffer before and after as well to increase the consistency across the section and provide context?

Author response:
**Re: how do we choose the region of error to be solved?**

The region of error to be solved must be a contiguous subsequence whose source semantics can be extracted for the SKETCH step. In the assembly domain, we exploit the domain specific idea of a “pure basic blocks”. This is based on standard basic blocks (Computer Architecture, Patterson & Hennessy) and described further in Appendix A.1. For this problem and SKETCH setup, these block are functionally independent, so we would not improve by larger regions. For other domains we agree that we might need to expand the subsequence could provide more context.