On the Embeddings of Variables in Recurrent Neural Networks for Source Code (2021)

Abstract: Source code processing heavily relies on the methods widely used in natural language processing (NLP), but involves specifics that need to be taken into account to achieve higher quality. An example of this specificity is that the semantics of a variable is defined not only by its name but also by the contexts in which the variable occurs. In this work, we develop dynamic embeddings, a recurrent mechanism that adjusts the learned semantics of the variable when it obtains more information about the variable’s role in the program. We show that using the proposed dynamic embeddings significantly improves the performance of the recurrent neural network, in code completion and bug fixing tasks.

## introduction deep learning is now actively being deployed in source code processing (scp) for solving such tasks as code completion @xcite , generating code comments @xcite , and fixing errors in code @xcite . source code visually looks like a text, motivating the wide use of nlp architectures in scp. a lot of modern scp approaches are based on recurrent neural networks @xcite , other popular architectures are transformers, and convolutional and graph neural networks. utilizing the specifics of source code as a data domain may potentially improve the quality of neural networks for scp. these specifics include three main aspects. firstly, the source code is strictly structured, i. e. the source code follows the syntactic rules of the programming language. secondly, the vocabularies may be large or even potentially unlimited, i. e. a programmer is allowed to define the identifiers of the arbitrary complexity. thirdly, the identifiers are invariant to renaming, i. e. renaming all the user-defined identifiers does not change the algorithm that the code snippet implements. the first two mentioned specifics have been extensively investigated in the literature. for example, the tree-based architectures, such as treelstm @xcite or treetransformer @xcite allow for the utilization of the code's structure. on the other hand, using byte-pair encoding @xcite or the anonymization of outof-vocabulary identifiers @xcite deals with the unlimited vocabulary problem. however, the property of source code being invariant to renaming user-defined identifiers has not been paid much attention to. in this work, we aim to close this gap for the recurrent neural networks (rnns). let us take a closer look at the invariance property. in fig. 1 (a ) and (b), a code snippet implementing a simple mathematical calculation is presented with two different variable naming schemes. both code snippets implement the same algorithm, i. e. are equivalent in the "program semantics" space, but have different text representations. classic nlp approach implies using the embedding layer as the first layer in the network where learnable embeddings encode the global semantics of the input tokens. in example (a), the embeddings of variables x and y make sense, as these variables are usually used in mathematical calculations, but in example (b), the embeddings of variables foo and foo2 do not reflect any semantics. moreover, even in case (a), the semantics of identifier y reflected by its embedding are too broad, i. e. this identifier could be used in a lot of different calculations, while variable y has a much more specific role in the program, i. e. storing the result of the particular function. the key idea of this work is that the embedding of a variable in the program should reflect the variable's particular role in this program and not only its name. the name of the variable may act as the secondary source of information about the variable's role, but the main source of this information is the program itself, i. e. the contexts the variable is used in. we develop the recurrent mechanism called dynamic embeddings that captures the representations of the variables in the program based on the contexts in which these variables are used. being initialized before a program processing, the dynamic embedding of a variable is updated each time the variable has been used in the program, see the scheme in fig. 1(g ). we test the dynamic embedding approach in two settings: the standard setting with the full data, and the anonymized setting, when all variable names are replaced with unique placeholders var1, var2, var3 etc. in the full data setting, we initialize the dynamic embeddings with standard embeddings, see fig. 1(e ), to implement the idea of the variable name being a secondary source of information about the variable's semantics. in the anonymized setting, we initialize the dynamic embeddings using a constant initial embedding, the same for all identifiers, see fig. 1(f ). in this setting, the variable names are not used at all, and the model detects the role of the variable purely based on the contexts in which the variable is used in the program. although being less practically oriented, the anonymized setting is a conceptually interesting benchmark, as it highlights the capabilities of deep learning architectures to understand the pure program structure, that is actually the main goal of scp, without relying on the unstructured textual information contained in variable names. in addition, the anonymized setting could be the case in practice, e. g. when processing the decompiled or obfuscated code @xcite . in the experiments, we show that using the proposed dynamic embeddings significantly outperforms the model that uses the standard embeddings, called static embeddings in our work, in both described settings in two scp tasks, namely code completion and bug fixing. to sum up, our contributions are as follows: our source code is available at https: //github.com/nadiinchi/dynamic_ embeddings. the possibility of improving deep learning models of source code by taking into account the invariance property of variable names has been superficially discussed in the literature. @xcite replace variables with their types, while @xcite and @xcite use the static embeddings for anonymized variables. however, the existing scp work did not consider developing a special architecture that dynamically updates the embeddings of the variables during processing a code snippet. our research is also related to the field of processing out-of-vocabulary (oov) variable names. the commonly used approaches for dealing with oov variables are using the pointer mechanism @xcite or replacing oov variables with their types @xcite . as we show in our work, both methods may be successfully combined with the proposed dynamic embeddings. in the context of nlp, @xcite use a similar model with dynamic embeddings to process oov and anonymized named entities in natural text. in contrast to their approach, we apply dynamic embeddings to the whole vocabulary of variable names, and incorporate dynamic embeddings into the model that relies on the syntactic structure of code. this results in more meaningful dynamic embeddings. ## proposed method we firstly describe what format of the model input we use, i. e. the procedure of code preprocessing, and then describe our model. at the end of this section, we discuss how we use the proposed model in two code processing tasks. de y de y de x de x de x de x de x de x @xmath0: (g) code snippet processing: assign namestore y binopsub binoppow nameload x nameload x 1 2 3 4 5 6 7 nameload x 1 @xmath1. 2. 3. 4. 5. 6. 7. ## experimental setup data and preprocessing. we conduct experiments on python150k @xcite and javascript150k @xcite datasets. both datasets are commonly used in scp and were obtained by downloading repositories from github. however, the train / test split released by the authors of the dataset does not follow the best practices of splitting data in scp @xcite , so we use another train / test split released by @xcite . this split is based on the repositories, i. e. all files from one repository go either to train or test, and was deduplicated using the tools provided by @xcite , i. e. code files in the test set that are duplicated in the train set were removed; this is a common case in source code downloaded from github. in addition, the python dataset includes only redistributable code @xcite . splitting by repository and deduplication are highly important in scp to avoid a percentage of testing accuracy being provided by the examples the model saw during training. with the described new split, the results in our tables are not directly comparable to the results reported in other works. to validate our implementation, we compared the quality of baseline models trained in our implementation with the quality reported in the papers describing these baselines, and observed that the numbers are close to each other (see details in section 5.4). for the code completion task, we use the entire code files as training objects, filtering out exceptionally long files, i. e. files longer than 3 • 10 4 characters. the resulting training / testing set consists of 76k / 39k files for python and of 69k / 41k for javascript. the mean length of the code files in 567 / 669 ast nodes for python / javascript. for the variable misuse task, we select all toplevel functions, including functions inside classes from all files, and filter out functions longer than 250 ast nodes, and functions with fewer than three positions containing user-defined variables or less than three distinct user-defined variables. the resulting training / testing set consists of 417k / 231k functions for python and 202k / 108k functions for javascript. one function may occur in the dataset up to 6 times: 3 times with a synthetically generated bug and 3 times without bug. the buggy examples are generated synthetically by choosing random bug and positions from positions containing user-defined variables. the described strategy for injecting synthetic bugs is the same as in @xcite . in both tasks, the size of the node type vocabulary is 330 / 44 for python / javascript, the vocabulary of node values is limited to 50k of the most frequent values. metrics. following @xcite , we use accuracy to measure model quality in the code completion task, counting all predictions @xmath2. following hellendoorn et al. (2020), to measure the quality in the variable misuse task, we use the joint localization and repair accuracy (what portion of buggy values is correctly located and fixed). details. in all our models, node type embeddings have 300 units, node value embeddings have 1200 units (for static embeddings), and the one-layer lstm's hidden state has 1500 units. the described model size matches the configuration of the model of @xcite . the proposed dynamic embeddings of values have 500 units in all experiments to show that they outperform the static embeddings with much less dimension. in the code completion task, we split the input ast traversals into the chunks, each chunk has the length of 50 ast nodes, and apply attention and pointer only over the last 50 positions. in the variable misuse task, we pass the entire function's ast traversal to the model. in code completion / variable misuse tasks, we train all models for 10 epochs with adamw @xcite @xcite with an initial learning rate of 0.001 / 0.0001, a learning rate decay of 0.6 after each epoch, a batch size of 128 / 32, and using weight decay of 0.01 / 0. we also use early stopping for the variable misuse task. for code completion, all hyperparameters are the same as in @xcite . we tuned hyperparameters to achieve convergence on the training set, for variable misuse. we use the same hyperparameters for static and dynamic embedding models. both used datasets are large, which helps to avoid overfitting, thus the regularization is not much needed. ## conclusion in this work, we presented dynamic embeddings, a new approach for capturing the semantics of the variables in code processing tasks. the proposed approach could be used in any recurrent architecture. we incorporated dynamic embeddings in the rnn-based models in two tasks, namely code completion and variable misuse detection, and showed that using the proposed dynamic embeddings improves quality in both full data setting and the anonymized setting, when all user-defined identifiers are removed from the data.