Introducing Compiler Semantics into Large Language Models as Programming Language Translators: A Case Study of C to x86 Assembly (2024)

Abstract: Compilers are complex software containing millions of lines of code, taking years to develop. This paper investigates to what extent Large Language Models (LLMs) can replace hand-crafted compilers in translating high-level programming languages to machine instructions, using C to x86 assembly as a case study. We identify two challenges of using LLMs for code translation and introduce two novel data pre-processing techniques to address the challenges: numerical value conversion and training data resampling. While only using a 13B model, our approach achieves a behavioral accuracy of over 91%, outperforming the much larger GPT-4 Turbo model by over 50%. Our results are encouraging, showing that LLMs have the potential to transform how compilation tools are constructed.

## introduction there is growing interest in using large language models (llms) for software engineering tasks @xcite ) like code retrieval @xcite , completion @xcite and translation @xcite . the training data of many llms, including @xcite , @xcite , and gpt4 @xcite contains code examples. however, these models are not explicitly trained for code translation. consequently, they are prone to errors during code translation @xcite . on the other hand, llms trained in natural language corpus have demonstrated impressive results in natural language understanding @xcite . as such, it is interesting to know if llms can learn to compile code. this paper investigates the feasibility of using large language models to translate a high-level * corresponding author programming language to machine instructions, a problem known as neural compilation @xcite . traditionally, this is performed by a manually crafted compiler that usually takes many person-years of compiler engineers' time to build. recent developments in llms have shown promising results in leveraging pre-trained transformer models for tasks like decompilation (e.g., translating assembly code to c programs) @xcite and program synthesis @xcite . however, few works use llms as a compilation tool to translate a high-level programming language into low-level assembly instructions. we hypothesize that llms may be able to learn from examples generated from other means, e.g., code synthesis, but the learnt model can directly produce the translated or even optimized code. our work seeks to bridge this gap by taking c to x86 assembly as a case study. a key challenge we face is managing the semantic gap between high-level languages optimized for human usability and low-level languages designed for hardware executions. this gap often manifests in a lack of direct correspondence between elements of the source and target languages. for instance, some commonly used data structures and programming constructs in c, such as struct and complex for-loop, do not have single equivalent x86 instructions. similarly, c uses identifiers for variables, while assembly instructions use stack and memory addresses or registers. as a single line of c code can be translated into a varying number of assembly instructions, learning the translation from c to assembly would require different amounts of training samples depending on the complexity of the mapping, making it difficult to construct a balanced training corpus. to overcome the aforementioned challenges, we leverage low-rank adaptation (lora) @xcite to fine-tune a pre-trained 13b codellama model @xcite . however, using the standard natural language training pipeline, our initial attempt yields a model with poor performance for c-to-assembly translations. after a close examination of the failure cases, we propose to introduce compiler semantics as two key data pre-processing techniques to enhance the trained model: symbolic interpretation for numerical value conversion and switch-case normalization for switch-case inconsistency. furthermore, we propose an automatic compiler semantics guided refinement learning framework to improve the fine-tuned model iteratively. our framework automatically resamples the distribution of semantic mapping samples and synthesizes the failure test cases in the validation set to improve the quality of the model training data. we perform a large-scale evaluation on over 57k executable c programs and compare them against the state-of-the-art large language model gpt-4 turbo. we verify the correctness of the generated x86 assembly code by executing them against unit test cases. experimental results show that our neural compiler generates code that is more accurate than all competing baselines. compared to gpt-4 turbo, our approach improves the translation accuracy by over 50%, from 40.85% to 91.88%. our main contributions are: ## problem statement we target the problem of machine translating highlevel programs(specifically, in the c language) into semantically equivalent low-level programs(in x86 assembly) with limited bilingual parallel corpora. one way to generate the training data is to use an existing compiler, such as gcc, as an oracle to generate semantically aligned assembly code from c language corpora. however, there are other options like search-based code synthesis techniques @xcite . since training data generation is performed offline, the overhead of generating the corpora does not affect the end user of the llm. our approach is also useful in porting a pre-trained llm to other hardware architecture, e.g., fine-tuning an llm trained on c-x86 samples to generate arm instructions from c programs. in this case, the pre-trained model can be fine-tuned on a small set of c-to-arm-assembly samples generated through code synthesis, reducing the cost of targeting compilers for a new hardware architecture. besides, aligning different programming language other than c to x86 assembly is also possible, which we discuss in appendix a.3. definition 1 there is a high-level programming language l high and a low-level programming language l low , each is an infinite set of valid program strings. there exists a unary relation ⇀ from l high to l low . given two monolingual corpora l high ⊂ l high and l low ⊂ l low , the problem is to learn a translator f such that ∀@xmath0, (∃@xmath1, x ⇀ u) → (x ⇀ f (x)). our main challenge in this work is the larger semantic gap between c and x86 assembly compared to translation between high-level codes like c-to-cuda @xcite and java-to-python @xcite . for example, like for-loop and if-else semantics, the translation must learn a posteriori to generate jump instructions and corresponding labels to express the original control flows. according to @xcite , there is no set of rules that can accurately model the relation ⇀, because it is undecidable whether two programs are semantically-equivalent. instead, we will use behavioral-equivalent to approximate. ## methodology our approach for translating high-level c code to low-level x86 assembly code targets generating semantically equivalent code in best efforts. in this work, we target non-optimized code generation, using the result given by the gcc compiler as a reference to train our model. numeric conversion llm @xmath2.15625f = x86 numeric c numeric float func(float x) { float temp; @xmath3.15625f; return temp; } .globl func .type func, @function func: .lfb0: .cfi_startproc addss .lc0(%rip), %xmm0 ret .cfi_endproc .lc0: ## case study we conduct case studies to demonstrate how to overcome the challenges using data augmentation methods to learn c-to-x86 translation. the first case study demonstrate a function that need float/double numerical value conversion. in x86 language, float/double immediate numbers can not be encoded in instructions directly, and modern compilers like gcc save them in binary format following the ieee-754 standard. so as long as the program exists numerical initialization, there are numerical conversions during the translation process, where llms perform poorly. as depicted in figure 5 , direct value conversion using implicit ieee-754 rule makes llms hard to predict, where the baseline models are very likely to generate wrong numbers. by delegating the numerical conversions from llms to rule-based interpreters, where we augment the model to generate symbolic expressions instead of direct guessing, llms delegate the numerical conversion to rule-based interpreters, which can handle their conversions well, so that the numerical handling drawback of llms is efficiently mitigated. the second case study depicted in figure 6 shows the challenge of switch-case generation, float func() { float @xmath4.0; float @xmath5.125; float @xmath6.0; float @xmath7.0; float numb; float temp; @xmath8; @xmath9; @xmath10; return numb; } func: the last case study shows how our refinement framework improving the long-tails performance. as depicted in figure 7 , avx instructions are the simd extension in x86 assembly language, and is encapsulated as avx intrinsics to be used in c language. recalling figure 3 , we introduce the refinement framework to augment the incorrect generations, which is inspired by @xcite . initially, there are no avx-related samples in the training corpora at all, where the model without any surprise translate incorrectly without apriori. then the incorrect avx sample is captured by the evaluator together with other incorrect samples. we then use llm to analyze the c code, and synthesize more based on several rules as prompts to generate more c samples closely related to the incorrect cases. we use mistral-7b @xcite as the synthesizer llm in our implementation. finally, the sythesized augmented c corpora of incorrect samples is added back to the training dataset, where retraining/finetuning can be performed depending on the need. back to the case itself, a 10x synthesizing is sufficient enough to learn a new feature with simple semantic pattern, like the _mm256_add_ps intrinsic in the case, which simply generates a vaddps instruction. such llm's learning ability of aligning c and x86 semantics is very impressive, which shows the few-shot learning potential in the language translation task. although more complex patterns need more cases to learn well, luckily, the refinement framework can be executed iteratively, which can resample the corpora based on the generation accuracy, so that more complex cases can get more samples to be learned. ## related work code translation aims to translate a piece of code (usually a function or method) into another programming language. early studies like @xcite uses traditional statistical machine translation method. neural-based method like @xcite starts to be dominant, and capture the tree structure of programming languages. the emergence of pre-trained language models of code, such as codebert @xcite and codet5 @xcite , has further improved the state of code translation. large language models(llms) @xcite have continued this trend, showing promise in code translation task. however, the above approaches usually require fine-tuning on parallel corpora, which is often scarce. data augmentation techniques have been extensively used and found effective in machine translation tasks, which served as a solution to the scarcity of parallel corpora. transcoder @xcite first propose back translation approach to learn unsupervised code translation, where the back-translation process also gener-ates an automatic parallel corpora augmentation method. transcoder-st @xcite , codexglue @xcite , babeltower @xcite and cmtrans @xcite also follow this approach, to obtain parallel corpora during the learning process. besides direct generation, @xcite explores an ir-in-themiddle approach, while @xcite both introduce an intermediate code summary stage, to improve the code translation accuracy. to construct a balanced corpora in limited size in monolingual language is also challenging, it is naturally in a long-tailed distribution for different aspects of code semantics. where neural models tend to perform low accuracy on the tails due to lack of samples. @xcite reveals that llms can perform between 30% to 254% worse in long-tailed cases, where the model under-fits them. inspired by the survey of long-tailed learning @xcite , we establish a refinement augmentation method, where long-tailed c samples are recognized in the evaluation process via metrics, then analyzed, synthesized by another powerful llm, compiled by gcc to obtain parallel samples, finally augmented the corpora with more long-tailed knowledge. cross level code translation. on highlevel code to low-level code translation researches, @xcite first gives a try of using neural machine translation on this scenario. (guo and moses, 2022) further studies on c-to-llvm ir translation. however, they only perform limited investigations on the methods, and their results are still on the preliminary stage. there are more related works on the reverse process, to recover high-level code from low-level code @xcite @xcite .unlike the difficulty on semantic mapping to low level code in our challenges, their challenges mainly are on optimization recovery and type inference, while the semantic recovery is relatively simpler. ## conclusion machine translation from high-level language to low-level machine instructions is difficult. even using advanced llms can not reach high accuracy directly. by implementing symbolic interpretation and switch-case normalization, two novel data preprocessing methods, we overcome numerical value conversion and switch-case semantic inconsistency, two significant challenges in c-to-x86 language translation. to improve the accuracy on long-tailed samples where the model under-fits to learn, we propose an automatic refinement augmentation framework to obtain improved accuracy on the long-tails by using synthesizing method on incorrect cases. finally we achieve state-of-the-art io accuracy, over 91%, when translating c-to-x86 on a largescale evaluation dataset. comparing to llm-only method(gpt-4-turbo, 40.85%), and finetuningonly baseline method(59.87%), the methods show great efficiency. more importantly, we show llms can perform well in c-to-x86 neural compilation task, and potentially other language pairs. in conclusion, these advancements demonstrate the potential of combining compiler-semanticguided data pre-processing and augmentation techniques with llms to significantly enhance machine translation accuracy, paving the way for future innovations in tasks like neural compilation.