IRC oder: Intermediate Representations Make Language Models Robust Multilingual Code Generators (2024)

Abstract: Code generation has fast become one of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs, such as cross-lingual transfer between different programming languages, language-specific data augmentation, and post-hoc LM adaptation, alongside the exploitation of data sources other than the original textual content, has been much sparser than for their natural language counterparts. In particular, most mainstream Code-LMs have been pre-trained on source code files alone. In this work, we investigate the prospect of leveraging readily available compiler intermediate representations (IR)—shared across programming languages—to improve the multilingual capabilities of Code-LMs and facilitate cross-lingual transfer. To this end, we first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files coupled with their respective intermediate representations. Next, starting from various base Code-LMs (ranging from 1.1B to 7.3B parameters), we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to (1) learn the IR language and (2) align the IR constructs with respective constructs of various programming languages. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across various code generation tasks and metrics, including prompt robustness, multilingual code completion, code understanding, and instruction following.

## introduction language models for code generation @xcite are some of the most promising tools for enhancing the productivity of software developers. they have proliferated into automating several parts of the traditional software development lifecycle, including code infilling, comment generation, refactoring, and build error prediction @xcite , inter alia. despite a strong demand for such capabilities across all programming languages, the benchmarking of code-lms has largely been dominated by the most resourced languages. for instance, popular benchmarks such as humaneval @xcite , mbpp @xcite and apps @xcite all test code-lms' competence only in python: this can result in misleading conclusions on the global utility of code-lms. more recent transpilationoriented benchmarks like multipl-e @xcite and babelcode @xcite -that test competence on several languages-have laid bare the gaps in code-lms' performance across different programming languages. for instance, the state-of-the-art deepseekcoder's @xcite code completion performance in bash, the most popular shell scripting language, lags its python pass@1 performance by 30%+ points. the problem is further exacerbated by the fact that the distribution of programming languages in code corpora is far more skewed than that of natural languages in standard multilingual text corpora. as an example, ukrainian, considered to be a moderate-to-low-resource natural language @xcite , comprises a higher proportion of the massively multilingual mc4 corpus @xcite than rust (the 13th most popular program-ming language) does of github 3 . this relative scarcity in code corpora, however, belies rust's criticality to digital systems: rust is one of only two languages approved for use in linux kernel development. 4 figure 1 illustrates this problem by comparing relative distributions of natural and programming languages in respective multilingual corpora. moreover, unlike the spread of digital content over natural languages, global code distribution over programming languages changes rapidly, reflecting sudden gains or drops in the popularity of individual programming languages. such changes mean that consequential programming languages at some point in time, may not have been represented in the pre-training corpora of the code-lms. one prominent example is hcl, the fastest growing programming language according to github, 5 used for configuring production infrastructure deployments, which did not make it into common pre-training corpora for code-lms @xcite . ## related work we provide a concise overview of the three most relevant lines of work: (1) high-quality pre-training data curation, (2) grounding with toolchain metadata, and (3) alignment across and cross-lingual transfer between languages. ## sltrans: a source code to llvm ir translation pairs dataset in order to test the hypotheses we posit in section 1, we seek to acquire parallel source-ir data for a mixture of low-, medium-, and high-resource programming languages. ## conclusion in this work, we investigated the effects of grounding heterogeneous source-code to a shared intermediate representation (ir) on code understanding and generation abilities of code-lms. to this end, we first create sltrans, a 26.2b token source code-ir parallel dataset containing nearly 4m training examples. we then perform continued pretraining on the corpus that includes parallel source code-ir data from sltrans for 6 established code-lms, demonstrating that ir grounding brings substantial performance gains in prompt robustness, multilingual code completion, code understanding, and instruction following, all while training on data orders of magnitude smaller than code-lm pretraining corpora. we hope that our encouraging results catalyze broader research efforts on the inclusion of intermediate code representations in pretraining and in the post-hoc adaptation of code language models.