Paper: An interpretable error correction method for enhancing code-to-code translation (ICLR-2024)

Reviewer comment (Evaluation):
Evaluation is based on translated unit tests generated by the same model that the authors are trying to correct translation errors for. Therefore, the unit tests that are generated could be wrong, and so the evaluation is unreliable. Evaluation should be performed based on a separate, high-quality set of unit tests. Possibly datasets like HumanEval-X would be better alternatives here.

Author response:
For this question, we add detailed explanations and experiments in the revised paper (Please refer to APPENDIX A.2 'GENERATION OF PYTHON UNIT TEST DATASET' and Table 6).

Reviewer comment (Experiments):
The experiments and results are fairly limited. First, the authors focus on only Java $\rightarrow$ Python and fail to consider other languages or even the reverse direction of Python $\rightarrow$ Java. Next, Table 1 seems to be missing many baselines and other models to which their approach should be compared. Namely, the only baseline is the pure TransCoder-ST model, which is only the starting point of their approach. The authors discuss that the main advantage of their approach is that no retraining is required, so it would be important to see how their approach performs relative to a retraining-based one. For this, they could have simply fine-tuned TransCoder-ST on the error correction pairs they collected for building their datastore. Next, latency is not measured, even though the authors discuss latency in related work. It seems that retrieving from a large datastore or retrieving multiple times from smaller datastores could take a long time, so it would be important to understand how the overall latency compares to other approaches. Finally, the authors do not report results on state-of-the-art code models, so it is difficult to assess the true value of their approach.

Author response:
(1) The current paper is a completely new approach in the field of program translation. Although the scope of our research is limited, we conducted detailed experiments to demonstrate the feasibility of this research. We appreciate your question about expanding the model to a multilingual environment, and this is precisely what we are currently working on. 

(2) we provide other more comprehensive baseline comparisons in the revised paper (Please refer to Table 1). 

(3) Thank you very much for your suggestion, we will try to fine-tune TransCoder-ST using error correction datasets in future work. 

(4) For the improvement and traditional approaches on $k$NN-MT, the generated results require both retrieving the datastore and running the Transformer model. However, in our work, the results of the error correction model are generated only by retrieving $(\text{key}, \text{value})$ pairs from the datastore, without incorporating the Transformer model. 

For $k\text{NN-ECD}$ (with the shortest processing time), we processed 2017 test samples with a total runtime of 520.28 seconds (average processing time of 0.26 seconds per sample). The translation accuracy achieved under this method was 84.5\%.

For $k\text{NN-ECS}_{18}$ (with the longest processing time), we executed 2017 test samples with a total runtime of 10207.46 seconds (average processing time of 5.06 seconds per sample). The corresponding translation accuracy for this method was 89.9\%.

Reviewer comment (Experiments):
Based on what is written in the paper, 10 Python functions with unit test cases are generated for each Java function. So, you have $(func_1, tests_1), (func_2, tests_2), (func_3, tests_3)... (func_{10}, tests_{10})$. Success is measured by executing the tests in some Python environment, where $func_i$ is considered a success if it passes all tests in $tests_i$. By this definition, suppose $func_1$ fails $tests_1$ and $func_2$ passes $tests_2$. The paper states "we combined the first failed Python function with the first successful Python function to form an error correction language pair." Based on this, it seems that $(func_1, func_2)$ would be considered an error correction pair. However, there is no guarantee that $tests_1 = tests_2$, meaning that the two functions could be executed against different test suites. Therefore, $func_2$ may not actually correspond to a correction of $func_1$. Could this please be clarified?

Author response:
For multiple Python functions under the same Java function, we use the same unit test dataset, and almost unit test dataset typically contains 3$\sim$6 unit tests. That is to say, $func_1$, $func_2$,..., $func_{10}$ use the same unit test dataset, where this unit test dataset usually contains 3$\sim$6 unit tests. (If a Python function can pass all tests under the unit test dataset, it succeeds; otherwise, it fails.)

Reviewer comment (Novelty):
The "interpretable decision-making" idea is not clear to me. It seems that you are suggesting that the reasoning for predicting a specific token at a timestep $t$ can be attributed to the source and partial target function predicted so far. This is also the case for transformer-based decoders, so it is not clear to me how your approach can be considered more interpretable than a transformer as they claim.

Author response:
In the newly submitted paper, we add detailed experiment and explanation in Section 4.2 'Interpretability of error correction model' and Table 7.

Reviewer comment (Writing):
In 3.2, you state that the hidden representations from the last layer of the decoder are used to build the (key,value) and query. My understanding is that the (key ,value) and query correspond to (failed Python function, partial Python function). It is not clear to me how there would be a decoder state corresponding to the failed Python function since that is passed into the encoder (Figure 1). Or is (failed Python function, partial Python function) meant to actually only represent the representation of the partial Python function generated so far, as labeled as "Key" in Figure 1?

Author response:
(1) Is (failed Python function, partial successful Python function) meant to actually only represent the representation of the partial Python function generated so far, as labeled as "Key"? Yes. The entire process is similar to the Transformer training process. The key is the output of the cross-attention layer in the decoder part. Reference [1] explains the technical details. 

(2) More specifically, the last hidden layer (i.e. cross-attention layer) is the coding (i.e. key) of <src\_func, tgt\_func\_pref >, where src\_func represents the failed Python function, and tgt\_func\_pref comes from the successful Python function. In the process of generating the key, the wrong Python function of the error correction language pair is the input of the encoder side, and the correct Python function of the error correction language pair is the input of the decoder side.

Reviewer comment (Experiments):
You claim that the improvement of ECS over ECD is "primarily attributed to its distributed structure, which includes diverse datastore variants." However, you do not seem to have multi-round experiments with ECD in which you repeatedly perform retrieval/correction on the same large datastore up to $m$ times. Therefore, isn't it possible that the advantage is actually from doing iterative code correction rather than the distributed nature of it?

Author response:
We provide detailed descriptions and experimental data in the revised paper (Please refer to  Appendix A.3 'ITERATIVE CODE CORRECTION').

Reviewer comment (Experiments):
On dataset construction, EvoSuite will overfit to the current implementation, which means there is an underlying assumption that the Java version is bug free. Further, translation of the PyTests from Java Unit tests can also be erroneous.

Author response:
For this issue, we provide a detailed explanation and experimental data in the revised paper  (Please refer to APPENDIX A.2 `GENERATION OF PYTHON UNIT TEST DATASET’ and Table 6).

Reviewer comment (Reproducibility):
In the multi-round error correction (kNN-ECD$_m$), does each subsequent ECD get the mis-corrected version from the previous module or does each ECD get the original wrong translation? If the former, does kNN-ECD$_m$ perform partial corrections that build on top of each other?

Author response:
In $k\text{NN-ECS}_{m}$, the input of each sub\_ECD is the original wrong translation. At the beginning of the experiment, we also considered the method you mentioned, using the wrong output from the previous sub\_ECD as the input for the next sub\_ECD. However, the experimental performance was very unsatisfactory.

Reviewer comment (Experiments):
On the dataset construction, have PyTest translations been manually or otherwise been validated to be correct/faithful to the EvoSuite output?

Author response:
Yes, this is one of the important prerequisites to ensure the accuracy of the experimental results.  For Python error correction language pairs, each wrong function in the training dataset and test dataset has a corresponding correct function (ground truth) that passes all unit tests. This guarantees that if the wrong function can be corrected, it can also pass all unit tests. Furthermore, for each Python function, its unit test dataset usually contains 3$\sim$6 unit tests. 
For this question, we add a detailed explanation and experiment in the revision of the paper (Please refer to APPENDIX A.2 'GENERATION OF PYTHON UNIT TEST DATASET' and Table 6).

Reviewer comment (Theory):
The motivation of applying $kNN-MT$ is not very clear. Although $kNN-MT$ is useful for natural language translation, is there some particular reasons that it will be more effective for programming languages.

Author response:
Response to Weaknesses_1&2:

(1) As you pointed out, our research goal is to significantly improve translation performance without retraining the large Transformer model. In our preliminary work, we found that the use of $k$NN-MT technology indeed provides substantial technical support. During this process, we were surprised to find that the $k$NN-MT method can provide interpretable results, but this point has not been thoroughly explored in previous research. 

(2) Regarding the interpretability, we add a detailed explanation and experiment in the revised paper (Please refer to  Section 4.2 `Interpretability of error correction model’ and Table 7). 

Response to Weaknesses_1&2:

Thanks for your response. The work of $k\text{NN-ECD}$
 is indeed based on $k$NN-MT, but the two are slightly different.

(1) The results generated by $k$NN-MT need to be combined with datastore and Transformer at the same time. In this case, it is impossible to return an explainable decision-making basis for each output token.

(2) Secondly, the results generated by $k\text{NN-ECD}$
 only use datastore, which can directly return an interpretable decision-making basis for each output token. (see Table 7)

(3) Regarding the use of decision-making basis: We can see the decision-making basis used by the uncorrected error type. This type of error can be corrected by replacing or deleting related  $(\text{key}, \text{value})$ pairs in datastore, where decision-making basis is the coding of key.

Reviewer comment (Presentation):
The presentation of experiment results is hard to read, especially for Table 3 and Table 4. I would suggest the authors to use Figures to present these results and put the detailed numbers in the Appendix.

Author response:
Response to Weaknesses_3: 

Thank you very much for your suggestions.  In the revision of the paper, we restate the experimental results more clearly and address the grammatical and expression problems. (please refer to  APPENDIX Figures 3-5)

Reviewer comment (Experiments):
The paper does not show the proposed method can perform error correction for OOD errors. The paper uses model $A$ to build the pair of incorrect programs and correct programs. Therefore, the error is specifically related to model $A$ itself. For a new model $B$, it may make different kinds of error, does the proposed method with learning datastore for model $A$ can fix the error of model $B$. If not, the method requires building datastore for every new method, which largely limits the application of the proposed method.

Author response:
Response to Weaknesses_4:

(1) Your suggestion is also one of the key issues we focused on in the early stages of our work. Firstly, our model is generalizable (please refer to Section 4.2 `Generalizability analysis’). 

(2) Secondly, since we were curious about which  errors could be corrected, we roughly classified translation error types in the initial experiments and found that errors could be categorized into fixed major types. In addition, kNN-ECD and kNN-ECSm contain 937362 $(\text{key}, \text{value})$ pairs, which include enough error correction information. 

(3) Finally, the datastore is scalable, and new error correction knowledge can be learned by adding new error correction information directly to the datastore, and the implementation method is extremely simple and efficient. This aspect is part of our future work and has been validated.

Response to Weaknesses_4:

(1) We understand your concerns very much. The results in Table 4 can provide some explanation for the generalization analysis. Under $k\text{NN-ECS}_{m}$,  each sub_ECD is generated from a completely different error correction dataset (with the same number of error correction pairs), but they all show similar performance on the same testset.The decision-making results (in Table 2) can also also provide the explanation for generalization.

(2) We will release the code, models, test dataset, training dataset and unit test dataset in the future. This is also the data support for the dataset comparison experiment mentioned in our generalization analysis.

 If the above experiments cannot solve your problem, we will supplement relevant experiments in the future.

Reviewer comment (Reproducibility):
For unified name rule, how to identify the variable name or function name in a program and replace them? Is it replaced by some automatic tools?

Author response:
Response to Questions_1:

Yes. First, we parse the code into an Abstract Syntax Tree (AST) using the Python standard library module ‘ast’. We then traverse the AST and replace corresponding function names, variable names, and parameter names with new names. In this process, we define is_builtin_name/ is_module_name/ is_parameter_name in the code to check if the name is builtin_name/ module_name/ given_parameter. If it is, we do not replace it.

Reviewer comment (Evaluation):
How do the method judge the wrong translations of TransCoder-ST? Is the error correction only applied to the wrong programs?

Author response:
Response to Questions_2: 

Indeed, we use unit testing to check whether the translation of TransCoder-ST [1] is correct. This is what the author of TransCoder-ST uses to verify whether the output is correct. Then, we correct the wrong translation of TransCoder-ST.

Response to Questions_2:

(1) In real life, users often put the output of the translation model directly into their compiler for testing. If it is wrong, the user inputs the wrong code into our error correction model.

(2) Thanks for the reminder, the following is another attempt we made in the early stages of the experiment. For the error correction system, another method that does not require unit testing: the user directly inputs the wrong translations of the translation model into all sub_ECDs under $k\text{NN-ECS}_{m}$  at the same time, and then the user verifies whether there are corrected translations in the $m$ outputs .

It's just a matter of which way to run the individual sub_ECDs under the same error correction system is better. It doesn't affect the internals of the individual sub_ECDs, and the final error correction results.
That is to say, for this method, the error correction performance is the same as the method mentioned in the paper. The detailed data of this method (no unit testing) is shown in Table 3 and Table 4, where the intersection of all sub_ECDs under the same $k\text{NN-ECS}_{m}$ is the error correction result of the system. This process is a bit like a translation model (beam_size = $m$), which outputs $m$ results for the user to verify.

If you also think this method is more effective, we will make modifications later.

Reviewer comment (Evaluation):
What does the method have better interpretability? The key value pairs in the datastore are still based on neural networks.

Author response:
Response to Questions_3:

(1) In the newly submitted paper, we add explanation and experimental details in Section 4.2 `Interpretability of error correction model’ and Table 7.

(2) The generation of $(\text{key}, \text{value})$ pairs is indeed based on the Transformer model. For each token generated by the Transformer-based model, the traditional Transformer model cannot track and identify which snippets in the training dataset contributed to the output token. However, $k\text{NN-ECD}/k\text{NN-ECS}_{m}$ can address this issue, providing an intuitive and readable decision-making basis for each output token. We consider this an advantage compared to the Transformer model.

Reviewer comment (Reproducibility):
Is there any fine-tuning stage of the TransCoder model?

Author response:
Response to Questions_4:

There is no fine-tuning process during the construction of the datastore.