A uto P ar LLM : GNN -guided Context Generation for Zero-Shot Code Parallelization using LLM s (2025)

Abstract: In-Context Learning (ICL) has been shown to be a powerful technique to augment the capabilities of LLMs for a diverse range of tasks. This work proposes AutoParLLM, a novel way to generate context using guidance from graph neural networks (GNNs) to generate efficient parallel codes. We evaluate AutoParLLM on 12 applications from two well-known benchmark suites of parallel codes: NAS Parallel Benchmark and Rodinia Benchmark. Our results show that AutoParLLM improves the state-of-the-art LLMs (e.g., GPT-4) by 19.9% in NAS and 6.48% in Rodinia benchmark in terms of CodeBERTScore for the task of parallel code generation. Moreover, AutoParLLM improves the ability of the most powerful LLM to date, GPT-4, by achieving 17% (on NAS benchmark) and 16% (on Rodinia benchmark) better speedup. In addition, we propose OMPScore for evaluating the quality of the parallel code and show its effectiveness in evaluating parallel codes.

## introduction the rise in the number of on-chip cores has led to more frequent development of parallel code @xcite . nevertheless, to unleash the capabilities of multi-core systems, the need for developing parallel programs will continue to grow. however, developing parallel programs is not a trivial task. the communication among cores, effective data sharing among the threads, synchronization, and many other factors need to be considered while crafting parallel programs, which makes the process of developing parallel programs far more complex than serial ones. hpc communities have published different tools and programming models to ease the process of moving from serial to parallel code. one of the well-established parallel programming models is openmp @xcite , which is a directive-based programming model that allows users to parallelize sequential code with minimal changes. most modern compilers recognize and support parallelization through openmp constructs. however, even with openmp, developers must carefully decide which clauses or directives they need to use. inappropriate usage of clauses can cause concurrency bugs such as data race or decrease performance. recently, large language models have also been used to generate parallel codes. one of the recent works @xcite showed that llms struggle to generate correct parallel codes using basic prompts. icl can help llms to generate better results. icl has been applied to both the training @xcite and inference stages @xcite @xcite @xcite . in all these works, context means providing the model with some sample input and the expected response before providing the test inputs. that involves constructing samples that are close to the test inputs or constructing enough samples such that all cases are covered. in practice, it may be difficult to construct such a set of sample inputs along with the expected outputs. this is especially true for code parallelization, as the length of the codes that are given as context can easily exceed the context length. to overcome this, we propose autopar-llm, a framework that works at the specific inputs provided to llms and generates context that is specific to that input only. autoparllm has two main components. firstly, a gnn-based context generator uses gnn to model flow-aware dependencies, i.e., data, control, and call flow and generates relevant context regarding whether the code is parallelizable and what parallel configurations are suitable. secondly, the llm-based code generator uses the context generated by the gnns to create an enhanced prompt and then generates the parallel code based on the prompts that contain the related context. in this work, we focus on openmpbased parallelization. due to the specific nature of openmp directives, where the order might be important in some cases, traditional code synthesis metrics may not be suitable to evaluate the quality of the generated openmp code. as such, we propose a new metric called ompscore that is more suitable to measure the quality of generated openmp constructs. in summary, our paper provides the following key contributions: ## background openmp offers different types of parallelization configurations. this work focuses on loop-level parallelism. not every configuration is applicable to every loop. for example, a loop having no inter-iteration dependencies can be parallelized by simply adding the '#pragma omp parallel for' directive if there is no variable inside the loop context that needs to be private to each thread when parallel execution occurs. this type of loops are considered as do-all. however, there may be cases when a variable needs to be private to each thread such that any other thread running parallel can not modify the content of that variable, as it may result in inconsistency. such cases are usually handled by privatization of the variable using the openmp clause private. also, there may be cases where the result of multiple loop iterations is combined into a final output. this is usually known as a reduction operation. such loops can also be parallelized by using reduction clause in openmp. it makes a separate copy of the reduction variable for each thread. when all threads are done with their calculations, it combines all the outputs of each separate copies of the variable and generates the final result. we provide examples of each in appendix a.1. ## approach in this section, we present autoparllm. a framework that leverages graph neural network to learn the flow-aware characteristics of the programs, such as control flow, data flow, and call flow, to add additional context and guide llms to generate parallel code by constructing a gnn-guided parallelism discovery do all private reduction github hpc benchmark code dataset (omp_serial) preprocessing flow-aware program representation (perfograph) heterogeneous gnn (gnn encoder) parallelism discovery & pattern prediction trained heterogeneous gnn (a) training flow-aware program representation (perfograph) querying trained gnn privatization needed? output sequential code sequential code llm parallel code yes parallelism found? initial prompt add do-all context parallelize given code using openmp with following rules: 1. code contains do-all pattern yes code needs privatization stage 1 prompt reduction needed? yes code needs reduction combiner no keep stage 1 prompt parallelize given code using openmp with following rules: 1. code contains do-all pattern {{add context generated from combiner}} gnn-guided context generation chain gnn-guided prompt with relevant parallelization context parallelize given code using openmp with following rules: omp prompt. figure 2 shows the overall workflow of autoparllm. in figure 2 (a), we show the training process where we train gnn models to predict parallelism opportunity and the parallel pattern. then, in figure 2 (b), we show how at inference time, gnn is used to create context for the gnn-guided omp prompt to guide the llm to generate better parallel code. ## experimental results we evaluate the effectiveness of autoparllm on several applications. in this section, we describe the details of those experiments. also, we describe the components of autoparllm in detail. all deep learning models are run on computing nodes with the same configuration. each computing node has an intel xeon gold 6244 cpu with 32 cores and 366 gb of ram. ## related works we describe the related works in detail in appendix a.14. here, we discuss works that are more closely related to our study. there are dlbased works that focuses on parallelization @xcite @xcite @xcite . these works mostly predict parallelism opportunities, but they do not generate complete parallel code. finetuning of llms has also been explored for openmp-based parallelization @xcite . however, the results indicate that the accuracy of the generated parallelization clauses is quite low. in contrast to these prior works, our approach uniquely combines the strengths of gnns with llms to generate prompts enriched with relevant context. this hybrid method leverages the advantages of both gnns and llms, providing a versatile solution compatible with any llm. also as discussed earlier a large body of works use icl to enhance the performance of llms. icl has been utilized in both the training @xcite @xcite and inference stages @xcite @xcite @xcite @xcite . in these studies, context involves providing the model with sample inputs and corresponding expected responses before the test inputs. whereas, we proposed a novel approach for generating context of input codes using gnn-based guidance. ## conclusion autoparllm, sits on top of gnns and llms to enable automatic parallelization of programs. we developed autoparllm as an intelligent parallelism assistant for developers but not as a replacement. based on our results, we believe it is fair to say that autoparllm generates parallelized versions of loops with very little human effort and greatly increases developers' efficiency.