U ni C oder: Scaling Code Large Language Model via Universal Code (2024)

Abstract: Intermediate reasoning or acting steps have successfully improved large language models (LLMs) for handling various downstream natural language processing (NLP) tasks.When applying LLMs for code generation, recent works mainly focus on directing the models to articulate intermediate natural-language reasoning steps, as in chain-of-thought (CoT) prompting, and then output code with the natural language or other structured intermediate steps. However, such output is not suitable for code translation or generation tasks since the standard CoT has different logical structures and forms of expression with the code. In this work, we introduce the universal code (UniCode) as the intermediate representation. It is a description of algorithm steps using a mix of conventions of programming languages, such as assignment operator, conditional operator, and loop. Hence, we collect an instruction dataset UniCoder-Instruct to train our model UniCoder on multi-task learning objectives. UniCoder-Instruct comprises natural-language questions, code solutions, and the corresponding universal code. The alignment between the intermediate universal code representation and the final code solution significantly improves the quality of the generated code. The experimental results demonstrate that UniCoder with the universal code significantly outperforms the previous prompting methods by a large margin, showcasing the effectiveness of the structural clues in pseudo-code.

## introduction the field of code translation and generation has advanced significantly @xcite with the advent of code-specific large language models (llms). code llms, such as starcoder @xcite and code-// function to split a string into several equal parts and return them as a list of strings def split_string(string, part_count) // loop through the string and extract parts for i from 0 to str_len step part_len do // if it's the last part and the string can't be divided equally, // take all remaining characters if i + @xmath0, i) // take the substring from i to the end else @xmath1, i, i + part_len) // take the substring of part length endif append(parts, part) // append the part to the list endfor return parts // return the list of parts enddef problem description: design a function to split a string into several equal parts and to return the parts as a list of strings. considering the low accuracy of cot in coder generation, structure cot (scot) @xcite is proposed to minimize the gap between the intermediate steps and the generated code. more intuitively, using a universal code as the intermediate representation to handle multiple programming languages (pl) is promising. here, universal code is a blueprint for implementing an algorithm, which helps to make the design of algorithms logically clear and readily comprehensible. moreover, it is universal across different programming languages (pl-agnostic) since it typically does not follow spe-cific syntax and omits execution details. yet, how the universal code is used for code translation and generation in multilingual scenarios remains underexplored. in this work, we scale up the code llms to support multiple programming languages via the universal code (unicode), which is used as an efficient and language-independent intermediate representation of the key algorithm principles. specifically, we first define unicode by specifying grammar rules and providing paradigms, followed by prompting @xcite to create an instruction dataset unicoder-instruct comprising natural-language questions, code solutions, and the corresponding universal code, as shown in figure 1 . then, the unicoder model is built by performing instruction tuning @xcite on multi-task learning objectives, including zero-shot question-answer generation (@xmath2, question-universal-code generation (@xmath3, universalcode-solution translation (@xmath4, and universal-code-of-thought (uot) objectives. in uot, the model is required to generate the universal code before the executable code. unicoder is evaluated on the python benchmark (humaneval @xcite and mbpp @xcite ) and the extended multilingual benchmark multipl-e. the results demonstrate that unicoder consistently achieves state-of-the-art performance across all languages, notably surpassing the previous baselines. furthermore, the ablation study verifies the efficacy of the proposed method, and extra discussions provide insights into the effect of our method. the contributions are summarized as follows: definition of universal code: 1. comments: use `//` for single-line comments and `/* ... */` for multiline comments. 2. variables: choose clear, type-free names for variables. 3. input/output: keep input/output straightforward. 4. conditionals: employ `if`, `elsif`, and `else` with proper indentation. 5. loops: use `for`, `while`, or `do...while` loops, specifying conditions and indenting code. 6. functions/procedures: name them descriptively and consider parameters. 7. formatting: maintain consistent 2-4 space indentation for clarity. example: ```pseudocode //this is the quicksort algorithm which sorts an array by recursively partitioning it around a pivot. quicksort(arr[], low, high) { if (@xmath5, low, high); quicksort(arr, low, pivot -1); quicksort(arr, pivot + 1, high); } } ``f igure 2: definition of the universal code. unicoder consistently outperforms the previous baselines on different benchmarks, including humaneval, mbpp, and multipl-e. to further verify the effectiveness of the universal code, we propose unicoder-bench to test the capabilities of code llms. ## unicoder-instruct definition of universal code. universal code is designed for expressing algorithms in a form that is easily understood by humans, blending programming language syntax with natural language descriptions and mathematical notation to outline the steps of an algorithm without the complexity of full coding details. it omits machine-specific implementations to focus on the core logic, making it a popular choice for documentation in educational materials and the preliminary design phases of software development. by abstracting away from the intricacies of actual code, pseudocode facilitates clear communication of algorithmic concepts across various programming environments. the definition of the universal code, as shown in figure 2 , is based on the following principles: construction from instruction dataset. for a programming language l, given the existing code instruction pair (q α , a α ) ∈ d l s , where q α and a α are question and answer from d l s , we create the universal code instruction dataset d l uα by prompting llms to generate the universal code p α and then add (q α , a α , p α ) into d l uα . figure 2 shows the definition of the code universal and figure 3 is the prompt for llms to generate unicode. {definition of universal code}, {question}, and {answer} denote the slots for definition of the universal code p α , the question of the instruction data q α , and the answer of the instruction a α , respectively. given k different programming languages l @xmath6, the multilingual programming instruction dataset with the universal code d @xmath7. in this work, we adopt the opensource instruction dataset. construction from code snippets. for the unsupervised data (code snippets) widely existing on many websites (e.g., github), we also construct the instruction dataset with the universal code from raw code snippets. specifically, we ask the llm to generate the question q β and the corresponding code answer a β pair based on the original code snippet c using the prompt "please generate the self-contained question and answer based on the given code snippet". then, we generate unicode p β and construct (q β , a β , p β ) triplets the same way as in paragraph 2. in addition, an llm scorer is applied to filter out the low-quality (q β , a β , p β ) triplets. therefore, given raw code snippets of different programming languages l @xmath8, we can construct instruction dataset with the universal code d u β = {d l k u β } k @xmath9. finally, we combine these two instruction datasets to obtain d @xmath10, where evaluation task for universal code. to test the capability of the llms in generating unicode from questions and translating unicode into answers, we design a code reconstruction task for evaluation. given the code snippet c, we require the llm to generate unicode p and then translate it into the code c ′ . the evaluation metric is not the similarity between c and c ′ but whether the restored code c ′ can pass the test cases. we expand the humaneval and mbpp datasets to create our benchmark unicoder-bench comprising 164 humaneval samples and 500 mbpp test samples.