Paper: AUTOPARLLM: GNN-Guided Automatic Code Parallelization using Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Instead, more interesting would be to include all OpenMP pragmas, loop level scheduling, architecture aware compute efficiencies with hardware aware scheduling strategies, etc (for reference see [1]). Combining all of these non-trivial loop level parallelizations would be a right task for such large capacity LLMs.

Author response:
**Challenges of extending support for all OpenMP clauses:** Also, there are not many open source code available for all the OpenMP pragmas for example pragmas like “simd” that offer vectorization and “target” that offer massive data parallelism through GPU-offloading are difficult to find. Hence extending support for all pragmas is challenging due to data scarcity. 

**Observations from (Chennupati et al., 2015) paper:** The mentioned approach needs program execution to generate performant parallel codes, which is challenging for HPC codes as most HPC codes are complex and executing them using different configurations has significant overheads in terms of computational resources. However, AUTOPARLLM does not need to execute programs to generate the parallel version of codes. Also, the training time of AUTOPARLLM is very low (around 17 minutes. See Global Response 4.) AUTOPARLLM is also evaluated on a wider range of applications that solve problems in different domains, unlike the one mentioned, which is evaluated on the same type of applications (only sorting).

Reviewer comment (Evaluation):
The difference in the execution times of the vanilla LLM generated parallel code versus AUTOPARLLM-CodeGen/GPT generated parallel code is so small (<3.0% in the best case) that it is difficult to state that the difference is significant. There is no mention of the reported execution times being an average/median of multiple runs of the generated program from each method.

Author response:
**Execution time:** Please see Global Response 1. We clarified the confusion. All reported times are average of 5 runs.

**Improvements:** Please see Global Response 2. AUTOPARLLM approach achieved as high as **14.75%** speedup when the speedup of individual applications are considered.

Reviewer comment (Experiments):
There are approaches in LLM literature to transform one language code to another [2]; why not try something similar to this directly to address the problem. Of course, it requires some minimal refactoring in the form of pre-training/fine-tuning these open CodeTransformer models.

Author response:
**CodeTransformer:** The referred paper (Zügner et al., 2021) uses source code and AST representation of programs. However, for parallelism detection, not only the structure (AST) but also the control, data, and call flows are extremely important. There was no mention of the above flows being incorporated in CodeTransformer. That is why we use a more recent and fine-grained program representation PERFOGRAPH, which works at the IR level and incorporates all those control, data, and call flow-related dependencies in the representation. Also, the transformer-based approaches require a large amount of data to learn. For example, CodeTransformer is trained on 691,974 samples. However, such a large dataset is not available for the problem of parallelism detection and OpenMP-based parallel code generation. Also, as mentioned above the training time for AUTOPARLLM is very low (around 17 minutes. Please see Global Response 4). But it is not mentioned what is the training overhead of CodeTransformer. We also like to mention again that on a smaller dataset, Graph-based learning approaches like Graph2Par, PERFOGRAPH showed better performance than transformer-based sequential approach like PragFormer on parallelism detection.

Reviewer comment (Reproducibility):
In the methods section, the details of loss functions at different components of the proposed method are missing.

Author response:
**Loss functions:** In all cases, the Cross Entropy-based loss function of PyTorch is used.

Reviewer comment (Novelty):
The paper uses large language models for code generation. As an auto-parallelizing framework, it is unclear whether we really need such LLMs. It appears that LLMs are primarily utilized for inserting pragmas into the outermost loop of a given loop nest, a task that could potentially be accomplished through simpler means such as loop analysis or pattern recognition. The paper could benefit from a more detailed explanation of the advantages of LLMs. For instance, could LLMs be employed to rewrite the code in a way that offers additional parallelization opportunities?

Author response:
**Challenges of parallelization, advantages of LLMs:** We can parallelize using OpenMP by inserting pragmas in loops. The insertion of pragmas may seem trivial however, before considering the parallelization of loops, let alone inserting pragmas, the control, data, and call flow-related characteristics need to be analyzed. Traditional loop analysis and pattern recognition-based approaches like AutoPar and DiscoPoP can do parallelization to some extent, but they miss a lot of parallelization opportunities due to being overly conservative. The advantage of LLMs is that, given enough background information regarding the intrinsic parallel characteristics of the loops, they can find more parallelism opportunities. That is why we proposed GNNs to learn the intrinsic control, data, and call flow-related features from programs and develop OMP Prompts that help the LLMs to have the necessary background information regarding the underlying parallel characteristics of source programs. We also showed in Appendix 8.1 that AUTOPARLLM finds more parallelism opportunities than those traditional loop-analysis or pattern recognition-based approaches.

Reviewer comment (Experiments):
The paper mentions that source-to-source compilers miss a lot of parallelism opportunities due to being overly conservative. However, the paper does not compare the optimization results against these compilers or manually-parallelized programs. Consequently, the true effectiveness of this approach remains uncertain.

Author response:
**Comparing with source-to-source (S2S) compiler, and traditional tools:** We compared AUTOPARLLM with both DIscoPoP and AutoPar (S2S compiler)  on the DiscoPoP and AutoPar subset of OMP_Serial dataset in Appendix 8.1 of the main paper for the task of Parallelism Discovery. AUTOPARLLM achieved 36% higher accuracy than DiscoPoP and 54% higher accuracy than AutoPar on detecting parallel loops. This clearly indicates the advantage of adopting LLMs for automatic parallelization. However, LLMs struggle to generate proper parallel codes even for simple programs, as shown in Motivation Examples in our paper. The control, data, and flow-related characteristics of loops learned by our GNN module help the LLMs to have the required background information regarding loops to generate the proper parallel version of the loops.

Reviewer comment (Evaluation):
It seems that the OMPScore can only be applied to auto-parallelization with OpenMP directives. How does it compare to other metrics with parallel semantics (e.g., ParaBLEU[1])? Is there any common design philosophy of metrics targeting different code generation approaches?

Author response:
**Comparing with ParaBLEU:** We compared the ParaBLEU score along with other metrics and updated Table 1 and Table 3 in the paper. Our findings remain consistent with ParaBLEU also along with other metrics as it can be observed from Table 1 and Table 3 that AUTOPARLLM improved the ParaBLEU score of LLMs. 

Also, we showed a comparison with ParaBLEU using Chain-Of-Thoughts prompting too (Please see Appendix 8.5 Table 8). 

The resulting correlation score of ParaBLEU with human evaluation was as low as 22.55 (Please see updated Table 5), whereas OMPScore shows a 99.99 correlation score with human judgment. This outcome underscores the substantial superiority of OMPScore in evaluating OpenMP directives.

We encountered two challenges that hindered our ability to assess the effectiveness of ParaBLEU in evaluating OpenMP directives. Initially, we sought access to the source code of ParaBLEU by reaching out to the authors, but unfortunately, it had not been made publicly available at that time. Consequently, we endeavored to replicate the concept of ParaBLEU scoring for our evaluation. In this replicated version, we assigned additional weights to highlight OpenMP keywords such as 'for,' 'private,' and 'reduction' in addition to the existing weights assigned to CUDA-related keywords. For more details regarding the formula please see Appendix 8.7. The implementation of ParaBLEU is given in the repository:
https://anonymous.4open.science/r/Project-A-AE4A/parableu_replicated/readme.md


**Design Philosophy:** We adopted a design philosophy similar to that of CodeBLEU and CodeBERTScore when creating OMPScore. Both CodeBLEU and CodeBERTScore carefully analyze the characteristics of target programming languages and provide scores that effectively capture these traits. CodeBLEU incorporates information about essential keywords and the syntactic graph of the code, while CodeBERTScore harnesses the capabilities of BERT-based code representations for scoring. OMPScore is designed to adapt to the specific characteristics of pragma directives, including order-sensitive and order-insensitive clauses. This adaptability allows OMPScore to be compatible with various code generation techniques, not limited to AutoParLLM. Furthermore, we have observed similar characteristics in other programming languages, such as Java. For instance, reordering variables in a variable declaration maintains the same code snippet's semantics, while reordering arguments in a method declaration can fundamentally alter the program. Therefore, we plan to enhance OMPScore to accommodate additional programming languages beyond Domain-Specific Languages in future work.

Reviewer comment (Reproducibility):
Graph information is not clear: The authors used GNN to determine the parallel regions and predict the OMP clauses from the data-flow, control-flow, and call graphs. However, it remains unclear how the heterogeneous graphs are constructed and how the features of nodes are determined. Also, it would be better if the authors could provide the graph properties, such as the number of nodes, the number of node types, the number of edges, as well as the training details, such as training time and learning curves.

Author response:
**Heterogeneous Graph Construction, node-types, edge-types, and features:**  Please see Appendix 8.3.

**Training Details, learning curves:** Please see Global Response 4.

**Num of Nodes, edges:** The average number of nodes in the PerfoGraph representation of OMP_serial dataset is 67.39, and the average number of edges in the dataset is 116.05.

Reviewer comment (Evaluation):
Correctness of the parallel version: The code generation heavily relies on GNN predictions; however, the training/test accuracy of the GNN predictions is not reported. What if the GNN gives a wrong prediction for the parallel regions or the OMP clauses? Furthermore, not all the parallel code can run correctly. The authors only reported the speedup but didn't mention if the program output matches that of the sequential version.

Author response:
**Correctness:** We ensured that all parallel codes that are generated by LLMs are correct before program execution. Please see Appendix 8.4 where we describe how we handle cases where LLMs wrongly parallelize a loop or generate wrong OMP clauses.

**GNN accuracy:** We reported the GNN prediction accuracy of Parallelism Discovery for NAS benchmark. Also, we reported the number of correctly classified loops of each type for NAS benchmark. Please see Section 5.2 (Page 7). For Rodinia benchmark, we also mentioned the number of correctly classified loops of each type. Please see Section 5.3 (Page 8). However, we also include the summary below.

|              Task                             | NAS Benchmark | Rodinia Benchmark|
|------------------------------------------|-------------------------|---------------------------|
|Parallelism Discovery Accuracy |          94.44%       |          100%            | 
|Private Detection Accuracy        |          92.86%       |          100%            |
|Reduction Detection Accuracy   |          100%          |          100%            |

Reviewer comment (Evaluation):
Improvement is not significant: Although the authors claim that their methods can improve the execution time, a 2-3% speedup is not significant. It would be better to conduct multiple runs and report the mean and variance of the execution time to determine if it can indeed bring improvements. Alternatively, the author can demonstrate in which scenarios the framework can achieve more significant improvements.

Author response:
**Execution times:** We reported the average execution time of 5 runs for each application in the paper. Please see Global Response 1. 

**Improvements:** When speedup of individual applications are considered then AUTOPARLLM achieves as high as **14.75%** speedup.  Please see Global Response 2 for details.

Reviewer comment (Experiments):
Comparison with Chain of thoughts: The idea of the proposed method is to use another model (GNN) to generate better prompts to guide the LLM in generating better results. The root cause is that LLMs cannot generate good results at once without hints. Therefore, it would be reasonable to compare it with the Chain of Thoughts method [1], which aims to complete a task using multiple intermediate reasoning steps. The GNN prediction part can be replaced with another prompt for the LLMs, and the output can be part of the next prompt. This could potentially yield better results compared to using only the basic prompt and could reduce the time required to construct the graphs and train the GNN model.

Author response:
**COT Prompting:** Please see Table 8 in Appendix 8.5. Due to time constraints, we performed Chain-of-Thought (COT) experiments with three of the LLMs: GPT-3.5, GPT-4, and CodeLlama on the NAS benchmark test set of 90 loops. However, AUTOPARLLM outperformed the COT approach in terms of all code generation metrics. Detailed prompts and results are given in Appendix 8.5.

Reviewer comment (Evaluation):
New metric OMPScore: The OMPScore is an extension of the Rough-L score, which is measured by the longest common subsequence. However, almost all the OpenMP pragmas share the same subsequence, ``#pragma omp parallel for``. Therefore, it doesn't appear to be a suitable metric. Additionally, the authors claim that the OMPScore metric has a high correlation with human evaluation, with a 99.99% Spearman correlation. Considering the potential human bias, it is not entirely convincing to achieve such a high correlation. It is recommended that the authors provide the details or raw data of the evaluation for both the OMPScore and the human evaluation to support this claim.

Author response:
**Incorporating ROGUE-L in OMPScore:** Before incorporating ROUGE-L as a component of OMPScore, we conducted an analysis to assess the correlation between several established translation evaluation metrics and human evaluations. The results, presented in Table 5 of our submission, indicated that ROUGE-L exhibited the highest correlation with human scores, as determined by Spearman ranking. We attribute the superior performance of ROUGE-L to a trade-off between the complexities of NLP-based metrics and programming language-based metrics. While NLP-based metrics like BLEU and Meteor primarily emphasize similarity at the word or n-gram levels without considering the alignment between the expected and translated results, metrics like CodeBLEU and CodeBERTScore are designed for object-oriented and scripting languages, which are considerably more intricate compared to domain-specific languages like OpenMP.

**Calculating correlation removing common terms:** Please see Appendix 8.6 for details. We calculated the correlation of OMPScore with human judgment by removing the common terms “#pragma omp parallel for”. OMPScore still achieved a very high and the best Spearman score (99.99) among all other metrics. 

**Raw data of Human Evaluation:** All data regarding the Human Evaluation scores are given in the repo.
https://anonymous.4open.science/r/Project-A-AE4A/Human-Evaluation/AutoParLLM_HumanScore.xlsx

Reviewer comment (Novelty):
The idea is simple and clear, but more details should be provided in the methods. Improving LLM generation by prompt engineering is not novel, as far as this reviewer is concerned. Using GNN to predict the parallel region is novel, but it is not quite convincing. It's hard to determine if it can be reproduced, especially regarding the GNN training, execution time improvement, and human evaluation for the metrics. Can the authors comment on the novelty of the proposed approach, i.e. how does the proposed approach improve over the state-of-the-art?

Author response:
**Reproducibility:** We have already provided with the codes necessary for training the GNNs (base.py) and doing predictions using the GNNs (main.py) in the anonymous repository (https://anonymous.4open.science/r/Project-A-AE4A/base.py), and we mentioned the repository in the Appendix (Page 12). Also, the codes generated by AUTOPARLLM using all four LLMs are also provided in the repository. The virtual environment dependencies are also given in the requirements.txt file. We also included the file-name, loop-number etc. so that one can easily find from which benchmark and from which application a certain loop is extracted. Both of the benchmarks are open-source, and they have very clear instructions regarding the execution of the applications. 

**Novelty:** Please see Global Response 5.

Reviewer comment (Novelty):
The authors are urged to include further literature references in related work. As the presented approach relies on the compilation through IR, two works that stand out and should be included are:  
- Transcoder-IR --> Code translation with Compiler Representations  
- Automap/PartIR --> Automap: Towards Ergonomic Automated Parallelism for ML Models

Author response:
**Reference of Transcoder-IR and automap/part:** Thanks for the suggestions. Yes, both of these works are IR based where one uses IR representation for code translation and the other uses IR based representation for parallelizing ML models. We referred the Automap/PartIR paper in Section 2 (Related Works: Data-driven approaches) and the Transcoder-IR paper in Section 4 (Approach: Program Representation) in the paper.