Paper: Alignment-Enhancing Parallel Code Generation for Semi-Supervised Code Translation (ICLR-2024)

Reviewer comment (Novelty):
The novelty of this paper is limited. The synthetic generation and alignment-ascending curriculum learning seems simple and straightforward.

Author response:
Thank you for acknowledging that our method is simple and straightforward. We believe that when combined with effectiveness in improving the performance, simplicity and straightforwardness can be strengths instead of weaknesses.

Reviewer comment (Novelty):
The discussion of the baseline is not enough. I mean, I cannot get what the contribution this paper achieved.

Author response:
Thank you for your feedback. We discussed some of the baselines in the related work section, hence we curated the detailed introduction to each of them in the experiment section. We are ready to answer any specific questions about the baselines if you can kindly share them.

Reviewer comment (Experiments):
During the generation of synthetic code, can you generate the snippet-level alignment?

Author response:
The snippet-level aligned data is from an existing dataset, CoST (Zhu et al.). We did not generate them during the parallel code generation.

Reviewer comment (Experiments):
During selecting the synthetic code, why not run the code and compare the returned results directly?

Author response:
The synthetic code is function-level code, and to run them requires necessary dependencies and testcases. It means that we need customized cross-lingual testcase generation for every single code in every language, which is challenging and expensive to acquire. For example, to test 260110 codes generated by BT in table 7 (shown below), at least 260110 different test cases need to be generated in 4 different languages. On the other hand, compilation filtering only takes the compiler of the target language, and neither compilation nor static analysis require customization for each code. Therefore, our methods is significantly cheaper and can be extended to more languages with little overhead.

**Table 7: Datasets used for training. Volume means number of parallel codes.**

| Data     | Type      |  Volume | Source                        |
|----------|-----------|---------|-------------------------------|
| Function | Annotated |    3326 | ECoST                         |
| Snippet  | Annotated |   31818 | ECoST                         |
| AND      | Synthetic |   69358 | Static Analysis & Compilation |
| COMP     | Synthetic |   79550 | Compilation                   |
| STAT     | Synthetic |  176482 | Static Analysis               |
| BT       | Synthetic |  260110 | Back Translation              |

Reviewer comment (Writing):
Section 2 - “the paper focuses on function-level code translation”. This confused me because the final evaluation is on computational accuracy, which cannot be evaluated at the function level. I think you mean to say that the parallel aligned data is generated at a function level, but the technique is evaluated on full source files?

Author response:
Your understanding is correct. The parallel code generation is at function level, and the computational accuracy evaluation is done by wrapping the functions into executable programs, which contains the necessary dependencies and test cases.

Reviewer comment (Writing):
The word “snippet” is vague and is used to mean different things in different places. For example, in Section 2.1 - “...takes as input a code snippet $x$” - here, snippet means *function*, presumably. But then in Section 2.1.1, there is a contrast between “snippet level” and “function level”, suggesting that snippets are smaller than functions. Could you please define what a snippet is, somewhere early on?

Author response:
The word “snippet” in this paper means small chunks of code that are separated by code comments in a program. It is not necessarily a complete function. We will add the definition of both early in the draft.

Reviewer comment (Writing):
What is “SPACoder-function” (Table 2 and Section 4.1)? Is it BT, STAT, COMP or AND? Or is it *all* of them, but in a curriculum learning setup?

Author response:
SPACoder-function is a dataset that contains all of BT, STAT, COMP and AND in a curriculum learning setup. Compared to the final dataset we used to train the SPACoder models, the only difference is that it does not include the annotated data ECoST-*.

Reviewer comment (Writing):
When you use the terminology “SPACoder-PLBART” or “SPACoder-T5”, there are actually two models involved here - the original *generator* model used to produce the synthetic parallel data, and the *base model* that you’re fine-tuning. I assume “SPACoder-PLBart” refers to a model where the generator *as well as* the base model are PLBART? Would be nice if this was clarified.

Author response:
Your understanding is correct. SPACoder-PLBART means a PLBART model trained on PLBART generated data, same as SPACoder-CodeT5. This setup is to demonstrate that our proposed method can be generalized to different models, without the assistance of more powerful models.

Reviewer comment (Reproducibility):
When doing static analysis on function signatures, how do you match types between different languages? Like int[] in Java and vector< int > in C.

Author response:
We build the cross-lingual type matching rules by extracting function information from large code corpora like CodeNet through AST parsing. After AST parsing, the information about a function will be stored in a dictionary, from which we can access each function’s signature. All the types will first be converted to a small set of base types, and then checked for more fine-grained alignment. For example, ArrayList in Java and <vector> in C++ will both be mapped to the base type list. The Integer type in Java will be converted to base type int, which can be mapped to the int type in C++. In this way, ArrayList<Integer> and vector< int > will be labeled as a cross-lingual pair of matching types.

Reviewer comment (Writing):
If you are operating at the function level, how do you apply a compilation filter? In C/C++, you can compile individual functions without linking, but not in Java. And Python code is not compiled, just interpreted. It would be nice if this was clarified.

Author response:
The compilation is done by creating a temporary file with the given code and calling the corresponding compiler from the command line. For Java, the function will be wrapped in a class with an empty main() function, and run with the javac command. For Python, the file will be run with the “python2/python3” command, which is the same as executing the code. After each compilation, the temporary file is deleted and the stdout/stderr will be saved for filtering and evaluation later. This whole process is parallelized, with memory limit and timeout for each process.

Reviewer comment (Experiments):
Let us say that the generator takes code from Language A and converts it to Language B. Then while fine-tuning, do you train on [B, A] samples, or [A, B] samples or both? In other words, while fine-tuning, which is the source language and which is the target language? This is also related to my next comment below.

Author response:
We use both. We have conducted experiments to compare the performance of using A-B, B-A and both as training data, and found that using both consistently gives better performance. Therefore we have been using the combination of both A-B and B-A for all the experiments.

Reviewer comment (Theory):
Section 2.2 - “Without the selector, the generation is reduced to plain back-translation”. I’m having difficulty understanding why this is true. According to me, this would only be true if 1) the generator takes code from Language A and converts it to Language B, but you fine-tune another model on [B, A] samples. 2) both the generator and the fine-tuning model are *trained together* (back translation relies on this kind of joint improvement of the forward and the backward model). Could you please clarify this terminology?

Author response:
Your understanding is correct. Since we use data generated in both directions, it is indeed different from commonly known back-translation. What we refer to is “parallel data generation with trained translator models”, and for simplicity, we call it back-translation since it’s conceptually similar. We will add the clarification in the draft.

Reviewer comment (Writing):
Typically, curriculum learning starts with **easy** examples and moves to **difficult** examples. Here, it seems like it starts with **low-quality** examples and moves to **high-quality** examples, which is non-intuitive ([low-quality ~ easy] and [high-quality ~ difficult]?). After a little thought, I think I understand why this is set up like this, but it is an important subtlety and should be clarified.

Author response:
Thank you for your feedback. You are right that the alignment-ascending curriculum in this paper appears slightly different from the typical easy-to-difficult curriculum. However, just like you kindly already pointed out, in terms of programming languages, high-quality data can actually be more challenging than low-quality data. Because low-quality parallel data might share more similarities between the source and target language due to the “shallow translation” problem, where the model translates token by token or directly copies from the source language. This makes the low-quality data potentially easier to learn than the high-quality data. We agree that it is important to clarify the subtlety and we will add it in the draft.

Reviewer comment (Writing):
Section 2.2 - “we denote the synthetic parallel data from cross-lingual static analysis as STAT and COMP, respectively” - this line needs fixing.

Author response:
Thank you for your feedback. We will rephrase it to “we denote the synthetic parallel data from cross-lingual static analysis and compilation filtering as STAT and COMP, respectively”.

Reviewer comment (Experiments):
The ablation study doesn’t support the effectiveness of curriculum learning. Specifically:  
- The result for BT + STAT + COMP + AND is missing.  
- To show the effectiveness of curriculum learning, results for rearranging the training stages should be shown.  
- It would be better to show the result of training only on the AND dataset for the same amount of total computation as curriculum learning and compare the two results.

Author response:
**Effect of rearranging the curriculum:**

In Table 8 (shown above) , we presented some new experimental results (with PLBART). Here is an overview of each experiment:
- Function: trained on ECoST-function (annotated data).
- Snippet+Function: trained on ECoST-function and ECoST-snippet (annotated data with fine-grained alignment).
- AND+Snippet+Function: trained with AND (high quality synthetic data filtered by both static analysis and compilation).
- BT+Snippet+Function: trained with BT (noisy synthetic data with no filtering).
- AND+COMP+STAT+BT+Snippet+Function: SPACoder with reversed curriculum order on synthetic data.
- BT+STAT+COMP+AND+Snippet+Function: SPACoder

We can see that when using the high quality synthetic data, AND, the performance is not as good as SPACoder, which uses larger amounts of noisier synthetic data. Similarly, when using the unfiltered noisy data, BT, the performance is also subpar. This shows that both the quality and the quantity of the synthetic data hold significant importance to the performance. When we reverse the order of the curriculum to AND+COMP+STAT+BT+Snippet+Function, the performance drops significantly compared to SPACoder, which indicates the importance of the order of the curriculum to the performance. It is worth noting that the reversed curriculum has very similar performance with BT+Snippet+Function, likely due to the larger volume of the BT dataset overpowering the effect of the previous datasets.

Reviewer comment (Experiments):
It’s not clear how much more total computation SPACoder takes when compared to previous models. Considering the curriculum learning during training, SPACoder might have been trained on the fine-tuning dataset for more epochs than previous models.

Author response:
Thank you very much for re-evaluating our work and increasing the score. We greatly appreciate your recognition of our efforts in addressing your concerns in the rebuttal.

We acknowledge the importance of transparency about the computational costs involved and will further revise the paper to include this information. While SPACoder leverages large amounts of unlabelled data for semi-supervised learning and consequently requires more compute compared to supervised learning on small annotated datasets, it is important to consider its relative cost-efficiency compared to other semi-supervised/unsupervised learning methods. For example, static analysis and compilation used in our method are inherently less computationally demanding than test case execution used in TransCoder-ST. 

As for the varying performance trends between CodeT5 and PLBART, we acknowledge that this aspect of our results could benefit from further investigation. Due to CodeT5's size being significantly larger than PLBART's, one hypothesis could be that model size is related to how much the model benefits from large amounts of noisy data compared to small amounts of annotated data. We plan to delve deeper into this in our future work. 

Once again, we are thankful for your constructive feedback and for the increased score. Your comments have been instrumental in refining our paper and guiding our future research directions.

Reviewer comment (Evaluation):
It would be great to see a comparison of the total computation in the main results.

Author response:
Thank you very much for re-evaluating our work and increasing the score. We greatly appreciate your recognition of our efforts in addressing your concerns in the rebuttal.

We acknowledge the importance of transparency about the computational costs involved and will further revise the paper to include this information. While SPACoder leverages large amounts of unlabelled data for semi-supervised learning and consequently requires more compute compared to supervised learning on small annotated datasets, it is important to consider its relative cost-efficiency compared to other semi-supervised/unsupervised learning methods. For example, static analysis and compilation used in our method are inherently less computationally demanding than test case execution used in TransCoder-ST. 

As for the varying performance trends between CodeT5 and PLBART, we acknowledge that this aspect of our results could benefit from further investigation. Due to CodeT5's size being significantly larger than PLBART's, one hypothesis could be that model size is related to how much the model benefits from large amounts of noisy data compared to small amounts of annotated data. We plan to delve deeper into this in our future work. 

Once again, we are thankful for your constructive feedback and for the increased score. Your comments have been instrumental in refining our paper and guiding our future research directions.

Reviewer comment (Experiments):
It would be great to see a thorough ablation study of the effectiveness of curriculum learning.

Author response:
**Effect of rearranging the curriculum:**

In Table 8 (shown above) , we presented some new experimental results (with PLBART). Here is an overview of each experiment:
- Function: trained on ECoST-function (annotated data).
- Snippet+Function: trained on ECoST-function and ECoST-snippet (annotated data with fine-grained alignment).
- AND+Snippet+Function: trained with AND (high quality synthetic data filtered by both static analysis and compilation).
- BT+Snippet+Function: trained with BT (noisy synthetic data with no filtering).
- AND+COMP+STAT+BT+Snippet+Function: SPACoder with reversed curriculum order on synthetic data.
- BT+STAT+COMP+AND+Snippet+Function: SPACoder

We can see that when using the high quality synthetic data, AND, the performance is not as good as SPACoder, which uses larger amounts of noisier synthetic data. Similarly, when using the unfiltered noisy data, BT, the performance is also subpar. This shows that both the quality and the quantity of the synthetic data hold significant importance to the performance. When we reverse the order of the curriculum to AND+COMP+STAT+BT+Snippet+Function, the performance drops significantly compared to SPACoder, which indicates the importance of the order of the curriculum to the performance. It is worth noting that the reversed curriculum has very similar performance with BT+Snippet+Function, likely due to the larger volume of the BT dataset overpowering the effect of the previous datasets.

Reviewer comment (Experiments):
It would be great to see experiments demonstrating the efficiency and scalability of Static Analysis filtering compared to Test Case filtering.

Author response:
**Comparison between testcase filtering and compilation/static analysis filtering:**
1. Difference in cost. Testcase filtering requires customized cross-lingual testcase generation for every single code in every language, which is challenging and expensive to acquire. For example, to test 260110 codes generated by BT in table 7, at least 260110 different test cases need to be generated in 4 different languages. On the other hand, compilation filtering only takes the compiler of the target language, and neither compilation nor static analysis require customization for each code. Therefore, our methods is significantly cheaper and can be extended to more languages with little overhead.
2. Difference in hypothesis. Testcase filtering is essentially an alternative way to acquire annotated parallel code data, because target code that passes expected testcases is functionally parallel to the source code. Therefore, testcase filtering follows the straightforward hypothesis of more annotated data leads to better code translation performance. On the other hand, our hypothesis is that large amounts of noisy parallel data arranged in a specific curriculum benefits code translation more than small amounts of annotated data. Our experimental results (with more presented in Table 8) support this hypothesis.

Reviewer comment (Reproducibility):
What is the version of CodeT5 used? Small? Base? This should be specified.

Author response:
**Table 6: Model size and number of parameters.**

| Model         | Size | Parameters |
|---------------|------|------------|
| CodeBERT      | base | 125M       |
| PLBART        | base | 139M       |
| CodeT5        | base | 220M       |
| TransCoder    | -    | 110M       |
| TransCoder-ST | -    | 110M       |

Reviewer comment (Experiments):
The paper lacks some comparison with newer public models (StarCoder, CodeLLama etc), or maybe closed source commercial model like GPT-3.5. While such comparisons may seem like "comparing apple to pear" due to their differences in model size and corpus, they are necessary if the authors want to show that SPACoder is truly advancing the problem of code translation. For larger language models, they often make much less compilation or runtime errors, and many of the problems appearing in smaller models like CodeT5 would disappear. If that's the case, the improvement using AST augmentation would be smaller, given that their main goal is to reduce syntax and simple run-time errors.

Author response:
**Comparison with code LLMs:**

In Table 6 (shown above), we presented the size and number of parameters of each model. As shown in the table, all the models are in ther hundred million parameters range. On the other hand, StarCoder has 15.5 billion parameters, and CodeLLama models range from 7 billion to 34 billion parameters. As you have kindly pointed out, comparing our models with code LLMs would be like "comparing apple to pear" due to the significant difference in model size and computation scales.

Reviewer comment (Experiments):
The authors argue the effectiveness of the technique on "low-resource" language C. While this is true for the given dataset that parallel C data is much smaller, the community won't agree C is a true low-resource language given that C has the largest size in many public pretraining datasets (e.g., the Stack). If the author truly wants to demonstrate the performance of SPACoder on a resource language, some DSL would be a good target.

Author response:
**Low-resource language:**

We agree that C is not necessarily a low-resource language in the software engineering community. In this paper, we refer to it as a low-resource language because of its significantly smaller training set size (~150 training samples) compared to other languages. We aim to evaluate how our method performs with a weak generator resulted from limited annotated data, and this setting can be extended to more languages with limited training samples. As shown in Table 4 (in the paper), our method improves computational accuracy of C by up to 43%, demonstrating strong performance under the low-resource setting.

Reviewer comment (Experiments):
The paper missed an experiment to compare no-curriculum vs curriculum training in terms of BT -> STAT -> COMP -> AND. Specifically, what would the model performance be like if you directly finetune PLBart or CodeT5 on AND data without other steps? This would explain whether curriculum or the dataset matter more.

Author response:
**Effect of rearranging the curriculum:**

In Table 8 (shown above) , we presented some new experimental results (with PLBART). Here is an overview of each experiment:
- Function: trained on ECoST-function (annotated data).
- Snippet+Function: trained on ECoST-function and ECoST-snippet (annotated data with fine-grained alignment).
- AND+Snippet+Function: trained with AND (high quality synthetic data filtered by both static analysis and compilation).
- BT+Snippet+Function: trained with BT (noisy synthetic data with no filtering).
- AND+COMP+STAT+BT+Snippet+Function: SPACoder with reversed curriculum order on synthetic data.
- BT+STAT+COMP+AND+Snippet+Function: SPACoder

We can see that when using the high quality synthetic data, AND, the performance is not as good as SPACoder, which uses larger amounts of noisier synthetic data. Similarly, when using the unfiltered noisy data, BT, the performance is also subpar. This shows that both the quality and the quantity of the synthetic data hold significant importance to the performance. When we reverse the order of the curriculum to AND+COMP+STAT+BT+Snippet+Function, the performance drops significantly compared to SPACoder, which indicates the importance of the order of the curriculum to the performance. It is worth noting that the reversed curriculum has very similar performance with BT+Snippet+Function, likely due to the larger volume of the BT dataset overpowering the effect of the previous datasets.

Reviewer comment (Experiments):
The reviewer would like the authors to answer questions related to comparison with LLMs using zero-shot or few-shot experiments, and to explain how non-curriculum training would affect the result.

Author response:
**Comparison with code LLMs:**

In Table 6 (shown above), we presented the size and number of parameters of each model. As shown in the table, all the models are in ther hundred million parameters range. On the other hand, StarCoder has 15.5 billion parameters, and CodeLLama models range from 7 billion to 34 billion parameters. As you have kindly pointed out, comparing our models with code LLMs would be like "comparing apple to pear" due to the significant difference in model size and computation scales. 

**Effect of rearranging the curriculum:**

In Table 8 (shown above) , we presented some new experimental results (with PLBART). Here is an overview of each experiment:
- Function: trained on ECoST-function (annotated data).
- Snippet+Function: trained on ECoST-function and ECoST-snippet (annotated data with fine-grained alignment).
- AND+Snippet+Function: trained with AND (high quality synthetic data filtered by both static analysis and compilation).
- BT+Snippet+Function: trained with BT (noisy synthetic data with no filtering).
- AND+COMP+STAT+BT+Snippet+Function: SPACoder with reversed curriculum order on synthetic data.
- BT+STAT+COMP+AND+Snippet+Function: SPACoder

We can see that when using the high quality synthetic data, AND, the performance is not as good as SPACoder, which uses larger amounts of noisier synthetic data. Similarly, when using the unfiltered noisy data, BT, the performance is also subpar. This shows that both the quality and the quantity of the synthetic data hold significant importance to the performance. When we reverse the order of the curriculum to AND+COMP+STAT+BT+Snippet+Function, the performance drops significantly compared to SPACoder, which indicates the importance of the order of the curriculum to the performance. It is worth noting that the reversed curriculum has very similar performance with BT+Snippet+Function, likely due to the larger volume of the BT dataset overpowering the effect of the previous datasets.