Paper: ADELT: Transpilation Between Deep Learning Frameworks (ICLR-2024)

Reviewer comment (Reproducibility):
One suggestion is that in the code skeleton process, all the API keywords are replaced by placeholders (namely PLACEHOLDER_1,2 ...). I wonder if it will be beneficial to add more context information into the placeholders, giving them more concrete meanings.

Author response:
We appreciate your insightful recommendations. Rather than using generic placeholders like `[PLACEHOLDER_1]`, `[PLACEHOLDER_2]`, we agree that using more specific tags such as `[FUNC_1]`, `[ARG_1]` could provide valuable context. However, given that our current approach has achieved 100% accuracy for code skeleton transpilation in our benchmark evaluations, this modification might be not necessary for this given task. That said, your suggestion will be useful for more challenging tasks, and will be considered in future work. Thank you again for your valuable input.

Reviewer comment (Experiments):
As the transpilation is decoupled into two sub-parts: the code skeleton and the API keywords mapping, is it possible to measure the precision of each process and how do they affect the final performance?

Author response:
We appreciate your insightful question regarding the precision of our transpilation process's sub-parts and their subsequent effect on ADELT's overall performance.

For skeletal code transpilation, Section 3.4 describes its accuracy of 100% on the benchmark, because skeletal code transpilation is designed to be simple and straightforward.

For dictionary lookup, the performance is reported in Table 3 (ablation studies), where we show metrics such as Precision@1, Precision@5, and MRR. The dictionary of ADELT has Precision@1 of 87.1% and 90.0% for both transpilation directions respectively.

Reviewer comment (Evaluation):
Very small evaluation dataset size of 50 samples. This makes any evaluation or comparisons made on this data less robust, with 95% deviations of upto ± 8.8 points.

Author response:
**W1. Very small evaluation dataset size of 50 samples. 95% confidence interval of upto +- 8.8 points.**

Thank you for your insightful feedback regarding the evaluation dataset's size. Recognizing the importance of a robust and reliable assessment of ADELT's performance, we have expanded the evaluation dataset to 100 examples. The additional 50 examples were curated by identifying PyTorch modules sampled from GitHub repositories with over 1,000 stars. Human experts subsequently translated these into the target frameworks, and you can find the selection process detailed in Appendix A.4 of the revised manuscript.

Upon re-evaluating the PyTorch-Keras transpilation with the augmented benchmark, ADELT maintains a substantial performance lead, achieving an improved exact match score increase of 17.4 percentage points over the baseline models. Updated results have been incorporated into the relevant sections of our manuscript. We believe that this improvement in our evaluation substantiates the reliability of our results.

The size of the revised dataset is comparable to the standard in domain-specific benchmarks: the general-purpose code generation dataset OpenAI HumanEval contains 164 examples, and most program synthesis benchmarks comprise less than 100 examples, like Casper (37 examples) [1], C2TACO (71 examples) [2], TF-Coder (70 examples) [3]. While we were able to only double the size of our evaluation dataset due to the limited time over the rebuttal period, we will expand the dataset further post-rebuttal.

Regarding statistical significance, 95% deviation of 8.88 points was observed from F1 score of GPT-3 (35.62 ± 8.88), which is statistically significantly inferior to ADELT originally reported (82.01 ± 3.08).

We have introduced a full results table with error bars for GPT-4 and other models on our new 100-example benchmark (also shown in Table 10 of the revised manuscript):

|   |    **Keyword**   | **(Precision@1)** | **Source-to-Source** | **(F1 Score)**   |
|-|:-|:-|:-|:-|
| | PyTorch-to-Keras | Keras-to-PyTorch  |   PyTorch-to-Keras   | Keras-to-PyTorch |
| GPT-3 |     35.4±6.1     | 39.1±4.2          | 26.6±5.1             | 32.1±6.7         |
| Codex  |     67.5±8.3     | 79.1±7.8          | 59.9±2.7             | 67.1±2.2         |
| GPT-4 |     74.5±5.8     | 83.2±3.5          | 67.7±2.6             | 74.9±1.7         |
| ADELT (Small) |     82.9±1.2     | **90.0±1.6**      | 79.0±2.2             | 76.7±1.5         |
| ADELT (Base) |   **87.1±1.2**   | **90.0±2.5**      | **83.4±0.8**         | **82.0±2.2**     |

These results illustrate that ADELT (Base) surpasses GPT-4 in all benchmarks with a statistically significant difference. We hope that this larger dataset and comprehensive re-evaluation address your concerns and bring further rigor to our work.

Reviewer comment (Evaluation):
The evaluation dataset construction process introduces bias by selecting only pairs with high BLEU score, which makes the dataset somewhat simpler than what a more uniform sampling might have yielded (for example, with a manually created dataset, with humans transpiling the code). This "simpler" dataset obscures how the proposed method may compare to others (such as prompting LLMs) in more complicated instances.

Author response:
**W2. The evaluation dataset construction process itself introduces biases. By selecting only pairs with high BLEU score**

Thank you for your astute observations concerning potential biases caused by our initial data collection method. You are correct that our initial reliance on BLEU score heuristics may have inadvertently skewed our dataset towards simpler examples.

However, it's important to clarify that the BLEU score was used as an initial guide to locate repositories potentially containing parallel examples of code across deep learning frameworks, rather than as a strict selection criterion.

In response to your suggestion, we have doubled our evaluation dataset with more challenging, realistic examples sourced from PyTorch repositories, which were then manually transpiled by human experts, as we discussed in our response to weakness 1.

Reviewer comment (Experiments):
The BLEU score filtering also biases the dataset towards shorter snippets with approximately 10 lines of code. This restricts the scope of this method to real-world scenarios with very small code snippets to translate, and again makes realistic comparisons difficult.

Author response:
**W3. The pre-training dataset perhaps will not be feasible to release due to copyright concerns.**

Thank you for your comment regarding the dataset used in our study. We acknowledge your concern about potential copyright issues related to parts of the data such as StackOverflow. To address this, while we have added these data in the supplementary material, we will release the script used to gather and process such data upon publication of the paper.

[1] Leveraging Parallel Data Processing Frameworks with Verified Lifting, https://arxiv.org/abs/1611.07623

[2] C2TACO: Lifting Tensor Code to TACO, https://dl.acm.org/doi/abs/10.1145/3624007.3624053

[3] TF-Coder: Program Synthesis for Tensor Manipulations, https://arxiv.org/abs/2003.09040

Reviewer comment (Reproducibility):
The pre-training dataset used may itself have been of interest, but perhaps will not be feasible to release due to copyright concerns.

Author response:
**W3. The pre-training dataset perhaps will not be feasible to release due to copyright concerns.**

Thank you for your comment regarding the dataset used in our study. We acknowledge your concern about potential copyright issues related to parts of the data such as StackOverflow. To address this, while we have added these data in the supplementary material, we will release the script used to gather and process such data upon publication of the paper.

[1] Leveraging Parallel Data Processing Frameworks with Verified Lifting, https://arxiv.org/abs/1611.07623

[2] C2TACO: Lifting Tensor Code to TACO, https://dl.acm.org/doi/abs/10.1145/3624007.3624053

[3] TF-Coder: Program Synthesis for Tensor Manipulations, https://arxiv.org/abs/2003.09040

Reviewer comment (Evaluation):
For Table 9, the reviewer asks if the authors could provide GPT-4's 95% intervals, specifically for F1 (corresponding to scores in Table 1).

Author response:
**Q1. For table 9, could the authors provide GPT4's 95% intervals, specifically for F1?**

We apologize for the oversight in our original submission. The results with 95% confidence intervals for GPT-4, have now been added to Table 10 of the revised manuscript. We also discussed those results in our response to weakness 1. Thank you for bringing this to our attention.

Reviewer comment (Presentation):
In Table 1 and similar tables, showing 2 decimal digits makes it hard to read and quickly compare values. Reducing to 1 decimal digit would make it easier/faster to understand. However, given that the 95% confidence intervals are so large, any decimal digit is arguably meaningless — perhaps no decimal digits should be shown.

Author response:
**PS1. 2 decimal digits (in Table 1, etc) make it hard to read the table to quickly compare values. 1 decimal digit will be easier/faster to understand.**

Thank you for your valuable suggestion. We appreciate your attention to clarity and have updated our tables accordingly, presenting all numbers rounded to 1 decimal digit.

Reviewer comment (Experiments):
Datasets in Appendix A.1 are reported in gigabytes, but a significant fraction of ipynb notebook files is binary data in output cells. A better metric to report would be either tokens, or dataset size without the binary parts of ipynb files.

Author response:
**PS2. Datasets in A.1 are reported in GBs, but often a significant fraction of ipynb notebook files is binary data in output cells**

We appreciate your insight. In response, we have revised Appendix A.1 to reflect the size of clean Python code in all datasets, measured in gigabytes, thereby providing a more accurate sizing metric.

Reviewer comment (Evaluation):
Lack of Negative Results: The paper primarily focuses on the strengths and successes of ADELT. Including discussions on potential limitations, challenges, or cases where ADELT might not perform as well would make the paper more balanced and provide a more realistic perspective.

Author response:
**W1. Lack of Negative Results**

Although our paper centered on ADELT's strengths, we have indeed discussed its limitations in detail. 

In Table 2, we provided a detailed case study illustrating instances where ADELT demonstrated less optimal performance. Notably, a case where the parameter `output_dim` was inaccurately mapped to `embeddings_initializer` due to an error in generating vocabulary mapping. 

Also, ADELT is unable to transpile `layers.Dense(out_dim, activation='relu')` into corresponding PyTorch API calls, `nn.Linear` and `nn.ReLU`. This is because the training corpus does not include enough examples that reflect such use cases in Keras. In Appendix A.9, we describe the potential for ADELT's improved modeling of such mappings in a scenario where we add synthetic data into the corpus.

We value your perspective and will expand our discussion of both the pros and cons in our paper.

Reviewer comment (Reproducibility):
How does ADELT handle situations where different versions of deep learning frameworks require different expressions? Can you give more explanations? Such as, for the old version of Python:  
x = torch.autograd.Variable(torch.Tensor([1.0]), requires_grad=True)  
for a newer version:  
x = torch.tensor([1.0], requires_grad=True)

Author response:
**Q1. ​​How does ADELT handle situations where different versions of deep learning frameworks require different expressions?**

Thank you for your insightful question. ADELT addresses variability across framework versions by treating each version-specific expression as a unique entry. Both `torch.autograd.Variable` and `torch.tensor` in PyTorch are mapped to `tf.Tensor` in Keras. The reverse process, subsequently, interprets `tf.Tensor` from Keras as the more recent `torch.tensor` expression in PyTorch. This approach maintains consistency across different framework versions and ensures accurate transpilation.

Reviewer comment (Writing):
The description in the Method section is somewhat challenging to understand. For instance, the example in Figure 1 is too simple, and it's not clear how the complete actions of step 2 and step 3 are reflected. Additionally, based on the description, it seems like steps 3, 4, and 5 are executed sequentially. However, in Figure 1, it appears that step 4 is executed in parallel with the steps 3,5, which is somewhat perplexing.

Author response:
Thank you for your insightful comments regarding the Method section of our paper. We apologize for any confusion caused by our initial presentation in Figure 1. We have revised the figure in the updated manuscript to better depict our approach. The updated Figure 1 now provides a more detailed illustration, clarifying how keyword substitutions from step 4 integrate with the other steps.

Yes, steps 3 and 4 are executed concurrently, each building upon the output of step 2, while step 5 synthesizes the results from both steps 3 and 4.

Reviewer comment (Experiments):
The size of the evaluation dataset is somewhat small, with only 50 samples, making it challenging to ensure the reliability of the experiments and comparisons.

Author response:
Thank you for your insightful feedback regarding the evaluation dataset's size. Recognizing the importance of a robust and reliable assessment of ADELT's performance, we have expanded the evaluation dataset to 100 examples. The additional 50 examples were curated by identifying PyTorch modules sampled from GitHub repositories with over 1,000 stars. Human experts subsequently translated these into the target frameworks, and you can find the selection process detailed in Appendix A.4 of the revised manuscript.

Upon re-evaluating the PyTorch-Keras transpilation with the augmented benchmark, ADELT maintains a substantial performance lead, achieving an improved exact match score increase of 17.4 points over the baseline models. Updated results have been incorporated into the relevant sections of our manuscript. We believe that this improvement in our evaluation substantiates the reliability of our results.

The size of the revised dataset is comparable to the standard in domain-specific transpilation benchmarks: the general-purpose code generation dataset OpenAI HumanEval contains 164 examples, and most program synthesis benchmarks comprise of less than 100 examples, like Casper (37 examples) [1], C2TACO (71 examples) [2], TF-Coder (70 examples) [3]. While we were able to only double the size of our evaluation dataset due to the limited time over the rebuttal period, we will expand the dataset further post-rebuttal.

Reviewer comment (Reproducibility):
Some details of the comparisons were not clearly explained, such as what prompts were used for GPT-4, as the choice of prompts can significantly impact the model's performance.

Author response:
Thank you for your insightful comments. We apologize for missing the GPT-4 prompt details. To address this, we have included a comprehensive explanation of the prompts used for GPT-4 in Appendix A.6 and detailed the post-processing protocol in Table 8 of the revised manuscript. As we explain in the appendix, our experiments leverage Markdown-formatted prompts in GPT-4's chat mode, and then GPT-4 sometimes generates a mixture of natural language and code. After generation, we programmatically extract code from the first code block from GPT-4's responses as the finalized transpilation. Our prompts for GPT-4 are designed to mimic the straightforward and intuitive approach of a human user, as done in prior approaches in evaluating such models [4].

Reviewer comment (Reproducibility):
If I did not overlook, the article only provides an account of the construction process for the PyTorch-Keras benchmark in Appendix A.4, but, it does not cover the construction process for the PyTorch-MXNet benchmark.

Author response:
Thank you for pointing out this oversight. We have now detailed the construction method for the PyTorch-MXNet benchmark in the revised Appendix A.4. Briefly, human experts manually transpiled the PyTorch snippets from our PyTorch-Keras dataset into MXNet, ensuring consistency across frameworks. Your feedback is greatly appreciated for the improvement of our manuscript.

Reviewer comment (Evaluation):
The execution efficiency also needs some comparisons, such as runtime, as the speed of a pipeline is typically expected to be slower than a single end-to-end model. However, the paper does not quantify the magnitude of this difference.

Author response:
We appreciate your concern on ADELT's computational efficiency. Efficiency is actually one of the strengths of ADELT.

ADELT is composed of two main procedures: large language model (LLM) prompting for code transpilation and dictionary lookup for API keyword mapping. Using Codex for transpiling the code skeleton is as efficient as using Codex to do transpilation end-to-end. The API dictionary lookup process is highly efficient and only takes nanoseconds since it does not involve neural network inference for each example. 

To illustrate, ADELT outperforms GPT-4 in transpilation results, and its skeletal code transpilation component uses Codex, which is of a similar scale as GPT-3. Because GPT-3/Codex is significantly faster, ADELT is faster than GPT-4.

[1] Leveraging Parallel Data Processing Frameworks with Verified Lifting, https://arxiv.org/abs/1611.07623

[2] C2TACO: Lifting Tensor Code to TACO, https://dl.acm.org/doi/abs/10.1145/3624007.3624053

[3] TF-Coder: Program Synthesis for Tensor Manipulations, https://arxiv.org/abs/2003.09040

[4] GPT-4 Technical Report, https://arxiv.org/abs/2303.08774