A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code (2021)

Abstract: There is an emerging interest in the application of natural language processing models to source code processing tasks. One of the major problems in applying deep learning to software engineering is that source code often contains a lot of rare identifiers, resulting in huge vocabularies. We propose a simple, yet effective method, based on identifier anonymization, to handle out-of-vocabulary (OOV) identifiers. Our method can be treated as a preprocessing step and, therefore, allows for easy implementation. We show that the proposed OOV anonymization method significantly improves the performance of the Transformer in two code processing tasks: code completion and bug fixing.

## introduction natural language processing (nlp) is widely used for source code processing (scp), e. g. for learning the meaningful vector representations of code @xcite @xcite , that can be used in various downstream tasks, e. g. code summarization @xcite , code completion @xcite , or bug fixing @xcite ). an important question, one should answer before building an scp model, is how to create a vocabulary? @xcite underline that modern source code datasets may incorporate millions of unique identifiers, of which less than 1% occur in the dataset frequently, e. g. more than 5 times. the common practice is to crop the vocabulary based on top-n identifiers and replace all occurrences of out-of-vocabulary (oov) identifiers with an unk identifier to avoid huge embedding matrices and the meaningless embeddings of rare tokens. but can one process rare identifiers in a better way? vocabulary: {np, sin } input: @xmath0.sin(my_x) + my_x standard oov processing procedure: @xmath1.sin(unk) + unk proposed oov anonymization procedure: @xmath2.sin(var2) + var2 python150k dataset (custom train-test split) 0 0.02 0.1 0.3 1 10 25 50 622 vocabulary size (thousand) 80 82 84 86 88 joint accuracy oov anonymization (ordered) oov anonymization (random) standard (no anonymization) javascript150k dataset (custom train-test split) 0 0.02 0.1 0.3 1 10 25 50 266 vocabulary size (thousand) 74 76 78 80 82 84 joint accuracy oov anonymization (ordered) oov anonymization (random) standard (no anonymization) figure 2 : results for transformer in the variable misuse task: joint bug localization and repair accuracy, mean @xmath3. models with the proposed oov anonymization significantly outperform the standard model (all oov identifiers are replaced with an unk token). the numerical data for the plots is given in there are two main directions in the nlp literature to tackle rare tokens: open vocabulary and copy-based approaches. an open vocabulary solution implies splitting rare tokens into subtokens @xcite . the copy-based approaches are used in generation tasks and imply using the pointer mechanism @xcite to copy tokens from the input sequence. we propose a new, simple, yet effective approach for processing oov identifiers in source code, namely oov anonymization. anonymization implies replacing rare identifiers with unique placeholders, i. e. var1, var2, var3 etc., while preserving the names of frequent identifiers. an example of oov anonymization is shown in figure 1. the intuition behind using anonymization is that it preserves the semantics of the algorithm that the code snippet implements, i. e. renaming userdefined identifiers does not change the underlying algorithm. by contrast, replacing all rare identifiers with an unk identifier changes the algorithm. we underline that we propose anonymizing only rare identifiers, because frequently used identifier names may serve as an additional source of information, and neural networks are indeed capable of capturing this information. the proposed oov anonymization strategy allows for easy implementation as a preprocessing step, thus no modification of model code is required. another advantage of the oov anonymization is that it enhances both the encoder and the decoder. the proposed approach significantly outperforms the model with all rare identifiers being replaced with an unk, in code completion and bug fixing tasks, with the transformer @xcite architecture being used (see example comparison in fig. 2 ). our code and data split are available at https://github.com/ bayesgroup/code_transformers . ## related work handling oov identifiers in source code. code processing often borrows ideas from nlp. source code can be represented as a sequence of identifiers. in this case, identifiers can be further split into subtokens using byte-pair encoding (bpe) @xcite resulting in an open vocabulary model. this approach has several drawbacks. firstly, splitting identifiers into subtokens increases the length of the sequence several times. this substantially slows down inference, e. g. vanilla transformer's forward pass has a complexity quadratic w. r. t. the input length. secondly, splitting breaks one-to-one alignment between identifiers and nodes in the parsing tree, e. g. abstract syntax tree (ast), in other words, several subtokens correspond to one node in the ast, which makes it harder to apply structureaware models such as @xcite or @xcite . to the best of our knowledge, all scp works, proposing structure-aware models, either use entire tokens without subtokenization / bpe, or average the embeddings over subtokens (this strategy provides only a slight quality improvement compared to the first one), and the question of how to incorporate bpe in structureaware models needs further investigation. taking into account the described disadvantages of bpe, we do not consider bpe in this work and do not split tokens into subtokens. an orthogonal direction for handling oov identifiers in source code is the modification of the computational graph. for the task of code generation, the pointer mechanism is widely adapted @xcite . @xcite also propose a graph-structured cache for inferring the representations of the rare identifiers in source code. the major drawback of the mentioned approaches is that they are quite hard to implement. ## proposed method consider a vocabulary of all identifiers in the training data. it could be a vocabulary of all tokens if we treat input code snippets as text sequences, or a vocabulary of all user-defined variables if we parse the asts of code snippets. let us now select the vocabulary v f ull of frequent identifiers and call all others oov identifiers. we propose an elegant way of tackling oov identifiers based on anonymization. particularly, we propose replacing all oov identifiers with placeholders var1, var2, var3 etc. all occurrences of one identifier in one input sequence are replaced with the same placeholder (anonymized identifier), but different identifiers are replaced with different placeholders. one identifier may be replaced with different placeholders in different input sequences. an example of oov anonymization is presented in figure 1 . we consider two strategies for the oov anonymization, namely ordered anonymization and randomized anonymization. the ordered anonymization implies assigning an anonymized identifier var1 to the first seen rare identifier, var2 to the next seen rare identifier, etc. for example, the snippet from the proposed oov anonymization can be seen as a preprocessing step, thus no model parts change. in the encoder, the embedding matrix contains embeddings for both anonymized and in-vocabulary identifiers: {e v } @xmath4. in the decoder, when generating the next identifier, the softmax is computed over all anonymized and invocabulary identifiers. we note that the ordered oov anonymization may need a more careful implementation, e. g. of metric computation, see details in section 4. ## conclusion in this work, we propose the effective anonymization-based encoding of out-ofvocabulary identifiers, with two options, namely ordered and randomized oov anonymization. our preprocessing technique allows for easy implementation, could be easily plugged into various transformer models and outperforms the widely used standard approach by a significant margin. the ordered anonymization performs slightly better than the randomized anonymization but requires a more careful implementation.