NS 3 : Neuro-Symbolic Semantic Code Search
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Semantic code search is the task of retrieving a code snippet given a textual description of its functionality.
Recent work has been focused on using similarity metricsbetween neural embeddings of text and code.
However, current language modelsare known to struggle with longer, compositional text, and multi-step reasoning.
To overcome this limitation, we propose supplementing the query sentence witha layout of its semantic structure.
The semantic layout is used to break down theﬁnal reasoning decision into a series of lower-level decisions.
We use a Neural
Module Network architecture to implement this idea.
We compare our modelNS 3 (Neuro-Symbolic Semantic Search) - to a number of baselines, includingstate-of-the-art semantic code retrieval methods, and evaluate on two datasetsCodeSearchNet and Code Search and Question Answering.
We demonstrate thatour approach results in more precise code retrieval, and we study the effectivenessof our modular design when handling compositional queries 1 .
Introduction
The increasing scale of software repositories makesretrieving relevant code snippets more challenging.
Traditionally, source code retrieval has been limitedto keyword [33, 30] or regex [7] search.
Both rely onthe user providing the exact keywords appearing inor around the sought code.
However, neural modelsenabled new approaches for retrieving code from atextual description of its functionality, a task knownas semantic code search (SCS) .
A model like Transformer [36] can map a database of code snippets andnatural language queries to a shared high-dimensionalspace.
Relevant code snippets are then retrieved by searching over this embedding space using apredeﬁned similarity metric, or a learned distance function [26, 13, 12].
Some of the recent workscapitalize on the rich structure of the code, and employ graph neural networks for the task [17, 28].
Despite impressive results on SCS, current neural approaches are far from satisfactory in dealingwith a wide range of natural-language queries, especially on ones with compositional languagestructure.
Encoding text into a dense vector for retrieval purposes can be problematic because werisk loosing faithfulness of the representation, and missing important details of the query.
Not onlydoes this a) affect the performance, but it can b) drastically reduce a model’s value for the users,because compositional queries such as “ Check that directory does not exist before creating it ” requireperforming multi-step reasoning on code.
We suggest overcoming these challenges by introducing a modular workﬂow based on the semanticstructure of the query.
Our approach is based on the intuition of how an engineer would approach a
SCS task.
For example, in performing search for code that navigates folders in Python they would ﬁrst only pay attention to code that has cues about operating with paths, directories or folders.
Afterwards,they would seek indications of iterating through some of the found objects or other entities in thecode related to them.
In other words, they would perform multiple steps of different nature - i.e.ﬁnding indications of speciﬁc types of data entities, or speciﬁc operations.
Figure 1 illustrates whichparts of the code would be important to indicate that they have found the desired code snippet ateach step.
We attempt to imitate this process in this work.
To formalize the decomposition of thequery into such steps, we take inspiration from the idea that code is comprised of data, or entities,and transformations, or actions, over data.
Thus, a SCS query is also likely to describe the code interms of data entities and actions.
We break down the task of matching the query into smaller tasks of matching individual dataentities and actions.
In particular, we aim to identify parts of the code that indicate the presenceof the corresponding data or action.
We tackle each part with a distinct type of network – a neuralmodule.
Using the semantic parse of the query, we construct the layout of how modules’ outputsshould be linked according to the relationships between data entities and actions, where each dataentity represents a noun, or a noun phrase, and each action represents a verb, or a verbal phrase.
Correspondingly, this layout speciﬁes how the modules should be combined into a single neuralmodule network (NMN) [4].
Evaluating the NMN on the candidate code approximates detecting thecorresponding entities and actions in the code by testing whether the neural network can deduce onemissing entity from the code and the rest of the query.
This approach has the following advantages.
First, semantic parse captures the compositionality of aquery.
Second, it mitigates the challenges of faithful encoding of text by focusind only on a smallportion of the query at a time.
Finally, applying the neural modules in a succession can potentiallymimic staged reasoning necessary for SCS.
We evaluate our proposed NS 3 model on two SCS datasets - CodeSearchNet (CSN) [24] and
CoSQA/WebQueryTest [23].
Additionally, we experiment with a limited training set size of CSN of10K and 5K examples.
We ﬁnd that NS 3 provides large improvements upon baselines in all cases.
Our experiments demonstrate that the resulting model is more sensitive to small, but semanticallysigniﬁcant changes in the query, and is more likely to correctly recognize that a modiﬁed query nolonger matches its code pair.
Our main contributions are: (i)
We propose looking at SCS as a compositional task that requiresmulti-step reasoning.
(ii) We present an implementation of the aforementioned paradigm based on
NMNs.
(iii)
We demonstrate that our proposed model provides a large improvement on a number ofwell-established baseline models.
(iv) We perform additional studies to evaluate the capacity of ourmodel to handle compositional queries.
Background2.1 Semantic Code Search
Semantic code search (SCS) is the process of retrieving a relevant code snippet based on a textualdescription of its functionality, also referred to as query .
Let C be a database of code snippets
c i .
For each c i 2 C , there is a textual description of its functionality q i .
In the example in Figure 2, thequery q i is “ Load all tables from dataset ”.
Let r be an indicator function such that r ( q i , c j ) = 1 ifi =
j ; and 0 otherwise.
Given some query q the goal of SCS is to ﬁnd c ⇤ such that r ( q , c ⇤ )
= 1 .
We assume that for each q ⇤ there is exactly one such c ⇤ .
Here we look to construct a model which takesas input a pair of query and a candidate code snippet: ( q i , c j ) and assign the pair a probability ˆ r ijfor being a correct match.
Following the common practice in information retrieval, we evaluate theperformance of the model based on how high the correct answer c
⇤ is ranked among a number ofincorrect, or distractor instances { c } .
This set of distractor instances can be the entire codebase C , ora subset of the codebase obtained through heuristic ﬁltering, or another ranking method.
2.2 Neural Models for Semantic Code Search
Past works handling programs and code have focused on enriching their models with incorporatingmore semantic and syntactic information from code [1, 10, 34, 47].
Some prior works have cast the
SCS as a sequence classiﬁcation task, where the code is represented as a textual sequence and inputpair ( q i , c j ) is concatenated with a special separator symbol into a single sequence, and the outputis the score ˆ r ij : ˆ r ij = f
( q i , c j ) .
The function f performing the classiﬁcation can be any sequenceclassiﬁcation model, e.g. BERT [11].
Alternatively, one can deﬁne separate networks for independently representing the query ( f ), the code( g ) and measuring the similarity between them: ˆ r ij = sim ( f ( q i ) , g ( c j )) .
This allows one to designthe code encoding network g with additional program-speciﬁc information, such as abstract syntaxtrees [3, 44] or control ﬂow graphs [15, 45].
Separating two modalities of natural language and codealso allows further enrichment of code representation by adding contrastive learning objectives [25, 6].
In these approaches, the original code snippet c is automatically modiﬁed with semantic-preservingtransformations, such as variable renaming, to introduce versions of the code snippet - c 0 with theexact same functionality.
Code encoder g is then trained with an appropriate contrastive loss, such as
Noise Contrastive Estimation (NCE) [19], or InfoNCE [35].
Limitations
However, there is also merit in reviewing how we represent and use the textual query tohelp guide the SCS process.
Firstly, existing work derives a single embedding for the entire query.
This means that speciﬁc details or nested subqueries of the query may be omitted or not representedfaithfully - getting lost in the embedding.
Secondly, prior approaches make the decision after a singlepass over the code snippet.
This ignores cases where reasoning about a query requires multiple stepsand thus - multiple look-ups over the code, as is for example in cases with nested subqueries.
Ourproposed approach - NS 3 - attempts to address these issues by breaking down the query into smallerphrases based on its semantic parse and locating each of them in the code snippet.
This should allowus to match compositional and longer queries to code more precisely.
Neural Modular Code Search
We propose to supplement the query with a loose structure resembling its semantic parse, as illustratedin Figure 2.
We follow the parse structure to break down the query into smaller, semantically coherentparts, so that each corresponds to an individual execution step.
The steps are taken in succession by aneural module network composed from a layout that is determined from the semantic parse of thequery (Sec. 3.1).
The neural module network is composed by stacking “modules”, or jointly trainednetworks, of distinct types, each carrying out a different functionality.
Method Overview In this work, we deﬁne two types of neural modules - entity discovery module(denoted by E ; Sec. 3.2) and action module (denoted by A ; Sec 3.3).
The entity discovery moduleestimates semantic relatedness of each code token c ji in the code snippet c j =
[ c j 1 , . . .
, c jN ] to an entitymentioned in the query – e.g. “ all tables ” or “ dataset ” as in Figure 2.
The action module estimatesthe likelihood of each code token to be related to an (unseen) entity affected by the action in the querye.g.
“ dataset ” and “ load from ” correspondingly, conditioned on the rest of the input (seen), e.g. “ alltables ”.
The similarity of the predictions of the entity discovery and action modules measures howwell the code matches that part of the query.
The modules are nested - the action modules are takingas input part of the output of another module - and the order of nesting is decided by the semanticparse layout.
In the rest of the paper we refer to the inputs of a module as its arguments .
Every input instance fed to the model is a 3-tuple ( q i , s q
i , c j ) consisting of a natural language queryq i , the query’s semantic parse s q i , a candidate code (sequence) c j .
The goal is producing a binarylabel ˆ r ij = 1 if the code is a match for the query, and 0 otherwise.
The layout of the neural module network, denoted by L ( s q i ) , is created from the semantic structure of the query s q i .
During inference,given ( q i , s q i , c j ) as input the model instantiates a network based on the layout, passes q i , c
j ands q i as inputs, and obtains the model prediction ˆ r ij .
This pipeline is illustrated in Figure 2, and detailsabout creating the layout of the neural module network are presented in Section 3.1.
During training, we ﬁrst perform noisy supervision pretraining for both modules.
Next, we performend-to-end training, where in addition to the query, its parse, and a code snippet, the model is alsoprovided a gold output label r ( q i , c j )
= 1 if the code is a match for the query, and r ( q i , c j ) = 0otherwise.
These labels provide signal for joint ﬁne-tuning of both modules (Section 3.5).
3.1 Module Network Layout
Here we present our deﬁnition of the structural representation s q i for a query q i , and introduce howthis structural representation is used for dynamically constructing the neural module network, i.e.building its layout L ( s q i ) .
Query Parsing To infer the representation s q i , we pair the query ( e.g. , “ Load all tablesfrom dataset ”, as in Figure 2), with a simple semantic parse that looks similar to: DO WHAT [(to/from/in/...)
WHAT, WHEN, WHERE, HOW, etc] .
Following this semantic parse, we breakdown the query into shorter semantic phrases using the roles of different parts of speech.
Nounsand noun phrases correspond to data entities in code, and verbs describe actions or transformationsperformed on the data entities.
Thus, data and transformations are separated and handled by separateneural modules – an entity discovery module E and an action module A .
We use a Combinatory
Categorial Grammar-based (CCG) semantic parser [43, 5] to infer the semantic parse s q i for thenatural language query q i .
Parsing is described in further detail in Section 4.1 and Appendix A.2.
Specifying Network Layout
In the layout L ( s q i ) , every noun phrase (e.g., “ dataset " in Figure 2)will be passed through the entity discovery module E .
Module E then produces a probability scoree k for every token c jk in the code snippet c j to indicate its semantic relatedness to the noun phrase:
E ( “ dataset ” , c j ) =
[ e 1 , e 2 , . . .
, e N ] .
Each verb in s q i (e.g., “ load ” in Figure 2) will be passed throughan action module: A ( “ load ” , p i , c j ) =
[ a 1 , a 2 , . . . , a N ] .
Here, p i is the span of arguments to the verb(action) in query q i , consisting of children of the verb in the parse s q i (e.g. subject and objectarguments to the predicate “load” ); a 1 , . . .
, a N are estimates of the token scores e 1 , . . .
, e N foran entity from p i .
The top-level of the semantic parse is always an action module.
Figure 2 alsoillustrates preposition FROM used with “ dataset ”, handling which is described in Section 3.3.
3.2 Entity Discovery Module
The entity discovery module receives a string that references a data entity.
Its goal is to identify tokensin the code that have high relevance to that string.
The architecture of the module is shown in Figure 3.
Given an entity string, “ dataset ” in the example, and a sequence of code tokens [ c j 1 , . . .
, c jN ] , entitymodule ﬁrst obtains contextual code token representation using RoBERTa model that is initializedfrom CodeBERT-base checkpoint.
The resulting embedding is passed through a two-layer MLP toobtain a score for every individual code token c j k : 0  e k  1 .
Thus, the total output of the moduleis a vector of scores: [ e 1 , e 2 , . .
, e N ] .
To prime the entity discovery module for measuring relevancybetween code tokens and input, we ﬁne-tune it with noisy supervision, as detailed below.
Noisy Supervision We create noisy supervisionfor the entity discovery module by using keywordmatching and a Python static code analyzer.
Forthe keyword matching, if a code token is an exactmatch for one or more tokens in the input string, itssupervision label is set to 1, otherwise it is 0.
Same istrue if the code token is a substring or a superstring ofone or more input string tokens.
For some commonnouns we include their synonyms (e.g. “ map ” for“ dict ”), the full list of those and further details are presented in Appendix B.
We used the static code analyzer to extract information about statically known data types.
Wecross-matched this information with the query to discover whether the query references any datatypesfound in the code snippet.
If that is the case, the corresponding code tokens are assigned supervision label 1, and all the other tokens are assigned to 0 .
In the pretraining we learned on equal numbersof ( query, code ) pairs from the dataset, as well as randomly mismatched pairs of queries and codesnippets to avoid creating bias in the entity discovery module. [Figure]
3.3 Action Module
[Figure]
First, we discuss the case where the action module has only entity module inputs.
Figure 4 providesa high-level illustration of the action module.
In the example, for the query “ Load all tables fromdataset ”, the action module receives only part of the full query – “ Load all tables from ??? ”.
Action module then outputs token scores for the masked argument – “ dataset ”.
If the code snippetcorresponds to the query, then the action module should be able to deduce this missing part from thecode and the rest of the query.
For consistency, we always mask the last data entity argument.
Wepre-train the action module using the output scores of the entity discovery module as supervision.
Each data entity argument can be associated with 0 or 1 prepositions, but each action may havemultiple entities with prepositions.
For that reason, for each data entity argument we create onejoint embedding of the action verb and the preposition.
Joint embeddings are obtained with a2-layer MLP model, as illustrated in the left-most part of Figure 4.
If a data entity does nothave a preposition associated with it, the vector corresponding to the preposition is ﬁlled withzeros.
The joint verb-preposition embedding is stacked with the code token embedding c j k andentity discovery module output for that token, this is referenced in the middle part of Figure 4.
This vector is passed through a transformer encoder model, followed by a 2-layer MLP and asigmoid layer to output token score a k , illustrated in the right-most part of the Figure 4.
Thus,the dimensionality of the input depends on the number of entities. We use a distinct copy ofthe module with the corresponding dimensionality for different numbers of inputs, from 1 to 3. [Figure]
The ﬁnal score ˆ r ij = f ( q i , c j ) is computed basedon the similarity of action and entity discovery module output scores.
Formally, for an action modulewith verb x and parameters p x =
[ p x 1 , . .
. , p xk ] , theﬁnal model prediction is the dot product of respective outputs of action and entity discovery modules:ˆ r ij =
A ( x, p x 1 , . . . , p xk  1 ) · E ( p xk ) .
Since the action module estimates token scores for the entityaffected by the verb, if its prediction is far from the truth - then either the action is not found in thecode, or it is not fully corresponding to the query, for example, in the code snippet tables are loadedfrom web, instead of a dataset.
We normalize this score to make it a probability.
If this is the onlyaction in the query, this probability score will be the output of the entire model for ( q i , c j ) pair: ˆ r ij ,otherwise ˆ r ij will be the product of probability scores of all nested actions in the layout.
Compositional query with nested actions Consider a compositional query “ Load all tables fromdataset using Lib library ”.
Here action with verb “ Load from ” has an additional argument “ using ” –also an action – with an entity argument “Lib library ”.
In case of nested actions, we ﬂatten the layoutby taking the conjunction of individual action similarity scores.
Formally, for two verbs x and yand their corresponding arguments p x =
[ p x 1 , . .
. , p xk ] and p y =
[ p y 1 , . . . , p yl ] in a layout that lookslike: A ( x, p x , A ( y, p y )) , the output of the model is the conjunction of similarity scores computedfor individual action modules: sim
( A ( x, p x 1 , . . .
, p xk  1 ) , E ( p xk ))
· sim ( A ( y, p y 1 , . . .
, p yl  1 ) , E ( p yl )) .
This process is repeated until all remaining p x and p y are data entities.
This design ensures thatcode snippet is ranked highly if both actions are ranked highly, we leave explorations of alternativehandling approaches for nested actions to future work.
3.5 Module Pretraining and Joint Fine-tuning
We train our model through supervised pre-training, as is discussed in Sections 3.2 and 3.3, followedby end-to-end training.
End-to-end training objective is binary classiﬁcation - given a pair of queryq i and code c j , the model predicts probability ˆ r ij that they are related.
In the end-to-end training,we use positive examples taken directly from the dataset - ( q i , c i ), as well as negative examplescomposed through the combination of randomly mismatched queries and code snippets.
The goal ofend-to-end training is ﬁne-tuning parameters of entity discovery and action modules, including theweights of the RoBERTA models used for code token representation.
Batching is hard to achieve for our model, so for the interest of time efﬁciency we do not performinference on all distractor code snippets in the code dataset.
Instead, for a given query we re-ranktop-K highest ranked code snippets as outputted by some baseline model, in our evaluations weused CodeBERT.
Essentially, we use our model in a re-ranking setup, this is common in informationretrieval and is known as L2 ranking.
We interpret the probabilities outputted by the model as rankingscores.
More details about this procedure are provided in Section 4.1.
Experiments
4.1 Experiment Setting
Dataset We conduct experiments on two datasets:
Python portion of the CodeSearchNet (CSN)[24], and CoSQA [23].
We parse all queries with the CCG parser, as discussed later in this section,excluding unparsable examples from further experiments.
This leaves us with approximately 40% ofthe CSN dataset and 70% of the CoSQA dataset, the exact data statistics are available in Appendix Ain Table 3.
We believe, that the difference in success rate of the parser between the two datasetscan be attributed to the fact that CSN dataset, unlike CoSQA, does not contain real code searchqueries, but rather consists of docstrings, which are used as approximate queries.
More details andexamples can be found in Appendix A.3.
For our baselines, we use the parsed portion of the datasetfor ﬁne-tuning to make the comparison fair.
In addition, we also experiment with ﬁne-tuning allmodels on an even smaller subset of CodeSearchNet dataset, using only 5K and 10K examples forﬁne-tuning.
The goal is testing whether modular design makes NS3 more sample-efﬁcient.
All experiment and ablation results discussed in Sections 4.2,4.3 and 4.4 are obtained on the test setof CSN for models trained on CSN training data, or WebQueryTest [31] – a small natural languageweb query dataset of document-code pairs – for models trained on CoSQA dataset.
Evaluation and Metrics We follow CodeSearchNet’s original approach for evaluation for a testinstance ( q, c ) , comparing the output against outputs over a ﬁxed set of 999 distractor code snippets.
We use two evaluation metrics: Mean Reciprocal Rank (MRR) and Precision@K (P@K) for K=1, 3,and 5, see Appendix A.1 for deﬁnitions and further details.
Following a common approach in information retrieval, we perform two-step evaluation.
In the ﬁrststep, we obtain CodeBERT’s output against 999 distractors.
In the second step, we use NS 3 tore-rank the top 10 predictions of CodeBERT.
This way the evaluation is much faster, since unlike ourmodular approach, CodeBERT can be fed examples in batches.
And as we will see from the results,we see improvement in ﬁnal performance in all scenarios.
Compared Methods We compare NS 3 with various state-of-the-art methods, including sometraditional approaches for document retrieval and pretrained large NLP language models.
(1) BM25is a ranking method to estimate the relevance of documents to a given query.
(2) RoBERTa
(code) isa variant of RoBERTa [29] pretrained on the CodeSearchNet corpus.
(3) CuBERT [26] is a BERT
Large model pretrained on 7.4M Python ﬁles from GitHub.
(4) CodeBERT [13] is an encoder-only Transformer model trained on unlabeled source code via masked language modeling (MLM) andreplaced token detection objectives.
(5) GraphCodeBERT [17] is a pretrained Transformer modelusing MLM, data ﬂow edge prediction, and variable alignment between code and the data ﬂow.(6) GraphCodeBERT* is a re-ranking baseline.
We used the same setup as for NS3, but used
GraphCodeBERT to re-rank the top-10 predictions of the CodeBERT model.
Query Parser
We started by building a vocabulary of predicates for common action verbs and entitynouns, such as “ convert ”, “ ﬁnd ”, “ dict ”, “ map ”, etc.
For those we constructed the lexicon (rules) ofthe parser.
We have also included “catch-all” rules, for parsing sentences with less-common words.
To increase the ratio of the parsed data, we preprocessed the queries by removing preceding questionwords, punctuation marks, etc.
Full implementation of our parser including the entire lexicon andvocabulary can be found at https://anonymous.4open.science/r/ccg_parser-4BC6 .
Detailsare available in Appendix A.2.
Pretrained Models Action and entity discovery modules each embed code tokens with a RoBERTamodel, that has been initialized from a checkpoint of pretrained CodeBERT model 3 .
We ﬁne-tunethese models during the pretraining phases, as well as during ﬁnal end-to-end training phase.
[Table]
[Figure]
[Table]
Hyperparameters The MLPs in entity discovery andaction modules have 2 layers with input dimension of768.
We use dropout in these networks with rate 0 .
The learning rate for pretraining and end-to-end trainingphases was chosen from the range of 1e-6 to 6e-5.
Weuse early stopping with evaluation on unseen validationset for model selection during action module pretraining and end-to-end training.
For entity discovery modelselection we performed manual inspection of producedscores on unseen examples.
For ﬁne-tuning the CuBERT, CodeBERT and GraphCodeBERT baselines weuse the hyperparameters reported in their original papers.
For RoBERTa (code), we perform the search forlearning rate during ﬁne-tuning stage in the same interval as for our model.
For model selection onbaselines we also use early stopping.
4.2 Results
Performance Comparison Tables 2 and 1 present the performance evaluated on testing portion of CodeSearchNet dataset, and WebQueryTest dataset correspondingly.
As it can be seen, our proposedmodel outperforms the baselines.
Our evaluation strategy improves performance only if the correct code snippet was ranked amongthe top-10 results returned by the CodeBERT model, so rows labelled “Upper-bound” report bestpossible performance with this evaluation strategy.
Query Complexity vs. Performance Here we present the breakdown of the performance for ourmethod vs baselines, using two proxies for the complexity and compositionality of the query.
Theﬁrst one is the maximum depth of the query.
We deﬁne the maximum depth as the maximum numberof nested action modules in the query.
The results for this experiment are presented in Figure 5a.
As we can see, NS 3 improves over the baseline in all scenarios.
It is interesting to note, that while
CodeBERT achieves the best performance on queries with depth 3+, our model’s performance peaksat depth = 1.
We hypothesize that this can be related to the automated parsing procedure, as parsingerrors are more likely to be propagated in deeper queries.
Further studies with carefully curatedmanual parses are necessary to better understand this phenomenon.
Another proxy for the query complexity we consider, is the number of data arguments to a singleaction module.
While the previous scenario is breaking down the performance by the depth of the
[Figure]
query, here we consider its “width”. We measure the average number of entity arguments per actionmodule in the query.
In the parsed portion of our dataset we have queries that range from 1 to 3textual arguments per action verb.
The results for this evaluation are presented in Figure 5.
As it canbe seen, there is no signiﬁcant difference in performances between the two groups of queries in either
CodeBERT or our proposed method - NS 3 .
4.3 Ablation Studies
Effect of Pretraining In an attempt to better understand the individual effect of the two modules aswell as the roles of their pretraining and training procedures, we performed two additional ablationstudies.
In the ﬁrst one, we compare the ﬁnal performance of the original model with two versionswhere we skipped part of the pretraining.
The model noted as ( NS 3  AP ) was trained with pretrainedentity discovery module, but no pretraining was done for action module, instead we proceeded to theend-to-end training directly.
For the model called NS 3  ( AP & EP ) , we skipped both pretrainingsof the entity and action modules, and just performed end-to-end training.
Figure 6a demonstrates thatcombined pretraining is important for the ﬁnal performance.
Additionally, we wanted to measurehow effective the setup was without end-to-end training.
The results are reported in Figure
6a underthe name NS 3  E 2 E .
There is a huge performance dip in this scenario, and while the performanceis better than random, it is obvious that end-to-end training is crucial for NS 3 .
Score Normalization
We wanted to determine the importance of output normalization for themodules to a proper probability distribution.
In Figure 6b we demonstrate the performance achievedusing no normalization at all, normalizing either action or entity discovery module, or normalizingboth.
In all cases we used L1 normalization, since our output scores are non-negative.
The versionthat is not normalized at all performs the worst on both datasets.
The performances of the other threeversions are close on both datasets.
Similarity Metric Additionally, we experimented with replacing the dot product similarity with adifferent similarity metric.
In particular, in Figure 6c we compare the performance achieved usingdot product similarity, L2 distance, and weighted cosine similarity.
The difference in performanceamong different versions is marginal.
4.4 Analysis and Case Study
Appendix C contains additional studies on model generalization, such as handling completely unseenactions and entities, as well as the impact of the frequency of observing an action or entity duringtraining has on model performance.
Case Study Finally, we demonstrate some examples of the scores produced by our modules atdifferent stages of training.
Figure 8 shows module score outputs for two different queries and withtheir corresponding code snippets.
The ﬁrst column shows the output of the entity discovery moduleafter pretraining, while the second and third columns demonstrate the outputs of entity discovery andaction modules after the end-to-end training.
We can see that in the ﬁrst column the model identiﬁessyntactic matches, such as “folder” and a list comprehension, which “elements” could be related too.
After ﬁne-tuning we can see there is a wider range of both syntactic and some semantic matchespresent, e.g. “dirlist” and “ﬁlelist” are correctly identiﬁed as related to “folders”.
Perturbed Query Evaluation
In this section we study how sensitive the models are to smallchanges in the query
q i , so that it no longer correctly describes its corresponding code snippet
c i .
Our expectation is that evaluating a sensitive model on c i will rate the originalquery higher than the perturbed one.
Whereas a model that tends to over-generalize and ig- nore details of the query will likely rate the perturbed query similar to the original.
Westart from 100 different pairs ( q i , c i ) , that both our model and CodeBERT predict correctly.
We limited our study to queries with a single verb and a single data entity argument to that verb.
For eachpair we generated perturbations of two kinds, with 20perturbed versions for every query.
For the ﬁrst type ofperturbations, we replaced query’s data argument witha data argument sampled randomly from another query.
For the second type, we replaced the verb argumentwith another randomly sampled verb.
To account forcalibration of the models, we measure the change inperformance through ratio of the perturbed query scoreover original query score (lower is better).
The resultsare shown in Figure 7, labelled “ V ( arg 1 ) !
V ( arg 2 ) ”and “ V 1 ( arg ) !
V 2 ( arg ) ”.
Discussion One of the main requirements for the application of our proposed method is being ableto construct a semantic parse of the retrieval query.
In general, it is reasonable to expect the usersof the SCS to be able to come up with a formal representation of the query, e.g. by representing itin a form similar to SQL or CodeQL.
However, due to the lack of such data for training and testingpurposes, we implemented our own parser, which understandably does not have perfect performancesince we are dealing with open-ended sentences.
Related work
Different deep learning models have proved quite efﬁcient when applying to programming languagesand code.
Prior works have studied and reviewed the uses of deep learning for code analysis ingeneral and code search in particular [39, 31].
A number of approaches to deep code search is based on creating a relevance-predicting modelbetween text and code.
[16] propose using RNNs for embedding both code and text to the same latentspace.
On the other hand, [27] capitalizes the inherent graph-like structure of programs to formulatecode search as graph matching.
A few works propose enriching the models handling code embeddingby adding additional code analysis information, such as semantic and dependency parses [12,
2],variable renaming and statement permutation [14], as well as structures such as abstract syntax treeof the program [20, 37].
A few other approaches have dual formulations of code retrieval and codesummarization [9, 40, 41, 6]
In their work [21] propose considering the code search scenario whereshort annotative descriptions of code snippets are provided.
Appendix E discusses more related work.
Conclusion
We presented NS 3 a symbolic method for semantic code search based on neural module networks.
Our method represents the query and code in terms of actions and data entities.
In contrast to existingcode search methods, NS 3 better captures the compositional nature of queries.
In an extensiveevaluation, we show that this method works better than strong but unstructured baselines.
We furtherstudy model’s generalization capacities, robustness, and sensibility of outputs in a series of additionalexperiments.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the Discussion paragraph
under Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code and data
in supplemental material
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes ] See paragraph titled Hyperparameters in Section
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e [Table] g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[No]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]