C3PO : A Lightweight Copying Mechanism for Translating Pseudocode to Code (2022)

Abstract: Writing computer programs is a skill that remains inaccessible to most due to the barrier of programming language (PL) syntax. While large language models (LLMs) have been proposed to translate natural language pseudocode to PL code, they are costly in terms of data and compute. We propose a lightweight alternative to LLMs that exploits the property of code wherein most tokens can be simply copied from the pseudocode. We divide the problem into three phases: Copy, Generate, and Combine. In the Copy Phase, a binary classifier is employed to determine and mask the pseudocode tokens that can be directly copied into the code. In the Generate Phase, a Sequence-to-Sequence model is used to generate the masked PL code equivalent. In the Combine Phase, the generated sequence is combined with the tokens that the Copy Phase had masked. We show that our C3PO models achieve similar performance to non-C3PO models while reducing the computational cost of training as well as the vocabulary sizes.

## introduction in recent years, computer programs have found applications in almost every field, from scientific to artistic fields. the demand and cost for programmers have gone up because writing code is a specialised skill. although people may be able to describe the functionality of the required code, the syntax of a programming language serves as a barrier to writing code @xcite . recently there has been an increase in low-code applications that only require the functionality of code to be specified as pseudocode in natural language (nl), which is then translated to source code in a programming language (pl). pseudocode enables people unfamiliar with a pl's syntax to write the functionality of the required code in nl, and allows programmers to write pl-independent code, which emphasizes functionality over syntax. translating pseudocode to code is cumbersome due to the complex structures of programs that result from their syntax, semantics and logic. existing state-of-the-art pseudocode-to-code translators, like codex @xcite , which is used to power github copilot @xcite , and codet5 @xcite , are being powered by complex transformer llms like gpt-3 @xcite and t5 @xcite ) respectively. they are pre-trained on very large code datasets like codesearchnet @xcite and @xcite . while these transformer architectures are good at generalizing to many downstream pl-related tasks, they require high-performance computational resources, large amounts of data, and significant time to train. our contributions are as follows: we propose c3po (computationally efficient copying mechanism for conversion from pseudocode to code), a lightweight alternative to the current translators. we exploit the property of code wherein a large number of tokens (like identifiers and variable names) are present in both pseudocode and its corresponding pl code. these tokens can therefore be simply copied into the resultant pl code translation. the remaining tokens can then be generated based on pl syntax. we divide the task of pseudocode to code translation into three phases: the copy phase, generate phase and the combine phase. in the copy phase, we use a decision tree classifier to decide whether each token in the pseudocode needs to be copied or translated. in the generate phase, a sequence-to-sequence (seq2seq) model takes in a pseudocode sequence, in which the tokens that can be copied are masked, and generates a masked pl sequence. in the combine phase, the generated masked pl code is unmasked with the appropriate true pseudocode tokens. c3po has been trained on the spoc dataset @xcite , which consists of human-written pseudocode lines in english with their corresponding c++ code lines. the binary tag sequence (t ) is used to mask the tokens (p masked ) and generate the masked code sequence (c masked ). this is finally combined with the copied tokens from the input p , to result in the translation combined. ## problem definition as we handle the task on a line-by-line basis, the task can be formulated as follows: the input sentence consists of a sequence of m pseudocode tokens @xmath0, p 2 ...p m , and the objective is to translate this into its corresponding sequence of n code tokens @xmath1, c 2 , ...c n . we additionally split the pseudocode tokens p into 2 sets of tokens -p cpy and p gen -where @xmath2. the main idea for the copying mechanism stems from the fact that a good number of the code tokens (identifier names, constants and keywords) in the code translation are also present in the pseudocode input. since such tokens can be simply copied into the translation, generating them from a seq2seq model is not necessary. such tokens are referred to as the set of copied tokens denoted by p cpy . in the spoc dataset, we observed that the mean ratio of copied tokens to sequence length in pseudocode was around 60%. ## methodology we devise a three-stage solution, which we call c3po, to the above-defined problem. the three phases -copy, generate and combine -are defined in an attempt to reduce the complexity of the model. the stages of c3po are illustrated with an example in fig. 1 . ## results bleu-4 score @xcite is chosen as the evaluation metric and reported in the bleu score for the non-c3po version was 0.44 and for the c3po was 0.51. therefore we can say that using the c3po mechanism, the model performed relatively better than the non-c3po version. as for the seq2seq with attention model, the bleu score for the non-c3po version is 0.75 and for the c3po version is 0.69. in this case, the non-c3po version performs better, but it comes at the cost of high training time and model size. further, we point out the significant difference in the number of parameters, and hence the training time, for both the versions in the bleu scores of the non-c3po and c3po transformers are 0.01 and 0.19, respectively. since transformers are data and compute-heavy architectures and we trained them on limited resources, they perform much worse than the rnn models. in fig 2 , we can notice that in both vanilla and attention models, the c3po version converges much faster than the non-c3po version. this would lead us to believe that the c3po versions would perform similarly, even if they were trained for lesser epochs than the non-c3po version. it version pseudocode code non-c3po version 6495 5647 c3po version (ours) 1984 1080 ## conclusion and future work we have introduced c3po, a copying mechanism that emphasises computational and data efficiency. the methodology exploited the property of code where most tokens remain consistent across input and output. by masking such tokens, the vocabulary sizes are reduced significantly, which also reduced the training times. in future works, the method for filling masked tokens in output can be improved to fill the tokens while handling cases where the assumption that the order of masked tokens would remain consistent fails.