C ODE E DITING FROM F EW E XEMPLARS BY A DAPTIVE M ULTI -E XTENT C OMPOSITION
A BSTRACT
This paper considers the computer source code editing with few exemplars.
The editing exemplar, containing the original and modified support code snippets, showcases a certain editorial pattern, and code editing adapts the common pattern derived from few support exemplars to a query code snippet.
In this work, we propose a novel deep learning approach to solve this code editing problem automatically.
Our learning approach combines edit representations extracted from support exemplars and compositionally generalizes them to the query code snippet editing via multi-extent similarities ensemble.
Specifically, we parse the support and query code snippets using language-specific grammar into abstract syntax trees.
We apply the similarities measurement in multiple extents from individual nodes to collective tree representations for query and support sample matching, and ensemble the matching results through a similarity-ranking error estimator.
We evaluate the proposed method on C# and Python datasets and show up to 8.6% absolute accuracy improvements compared to non-composition baselines
I NTRODUCTION
In recent years, a surge of interest has been witnessed in applying machine learning techniques to code editing (Zhao et al., 2019; Chen et al., 2019; Dinella et al., 2020; Chakraborty et al., 2020;
Yasunaga & Liang, 2020).
Code editing in software engineering intends to revise the design, structure, functionality, or implementation of existing programming codes into a desirable pattern.
To maintain high-quality code projects or merge several repositories, the programmers typically fix one or few code snippets, and expect the same revision automatically applied to other places in demand over the whole project.
Automating this process can facilitate a broad range of programming applications such as code migration, refactoring, version update, and bug repair.
The problem of code editing with exemplar(s) aims to adopt the common editorial pattern from given exemplar(s) to a query code snippet.
The editorial pattern describes the type of change between two code snippets, and implies the underlying intent of making a kind of edit for a specific purpose, e.g., updating some keywords or reformatting redundant codes.
Pink rectangles in Figure 1 provide an example of editorial patterns with comparisons between original and edited code snippets.
We also present a list of editorial patterns that appear in our dataset in Appendix A for readers’ understanding.
This problem is conceptually similar to programming by example in software engineering (Menon et al., 2013; Meng et al., 2013), and recently has been studied by Yin et al.
; Brody et al.
Yao et al.
(2021) with deep learning.
Technically, their methods parse the source code as an abstract syntax tree using language-specific grammar, apply one-time or sequential edit actions over the tree, and always keep the modified code snippet satisfying grammatical rules.
In particular, all these works focus on performing code editing with exact one editing exemplar, i.e., the one-shot setting.
While the above studies have demonstrated interesting results, simply adapting the editorial pattern from one single exemplar can lead to poor generalization or even incorrect editing.
Figure 1 presents two cases.
In Case (a), when a model observes only the Support #1 exemplar, it may intuitively interpret the code refactoring rule as replacing the conditional expression ( “ if (s==null)... ” ) by a coalesce one ( “ ...??...
; ” ) only for a newly-declared variable ( “ string s=... ” ).
Consequently, applying the model to a query input could mistakenly transform the newly-declared variable ( “ string title=... ” ) rather than its re-assigned correspondence ( “ title=ReformatTitle(...) ”; see Solution #1).
In contrast, Case (b) demonstrates a situation where two editing exemplars turn out to be sufficient to capture the logical transformation.
The foregoing articulates the necessity of leveraging few but not one exemplar to adapt an editorial pattern to a new code snippet.
The few exemplars help the editing escape from the inductive bias learned from one exemplar, and enhance generalized capacity for other snippets with various contexts.
In light of this, in this paper, we consider the computer source code editing with few exemplars.
The editing of a query code snippet is triggered by few editing exemplars that containing the original and modified support code snippets.
Different from the above one-shot code editing scenario, few exemplars, even within the same editorial pattern, perform divergent coding contexts and edit actions.
Therefore, it is not eligible to adapt the edit actions directly from an arbitrary exemplar.
For this scenario, the major challenge lies in how to identify and match helpful support exemplars for a query input and compose them to guide the editing .
To address this challenge, we propose a novel approach that combines edit representations extracted from exemplars and compositionally generalizes them to the query code snippet editing via multiextent similarities ensemble.
Specifically, we parse the support and query code snippets using language-specific grammar into abstract syntax trees, where editing actions are executed on nodes in the tree.
Based on this, we model the similarity between the representations of support and query abstract syntax trees.
We notice that among few exemplars, some editing actions only happen in a local region, while concurrently some editing actions rely on more coding context.
This inspires us to consider multi-extent similarities between the representations of support and query abstract syntax trees.
We design a λ -softmax function by scaling the importance of nodes in an abstract syntax tree, i.e., λ → 0 means the support/query code snippet is denoted by tree representation, while λ → ∞ means the support/query code snippet is denoted by one single node representation in the abstract syntax tree.
By controlling λ , the intermediate position between individual nodes and a collective tree, we provide multiple representational query-support matchings, then adaptively ensemble outcomes from these extents through a similarity-ranking error estimator for a robust composition.
We use the term multi-extent to describe the scope of query-support matching at various levels from nodes to a collective tree representation.
We evaluate the performance on two code editing datasets, one in C#
(Yin et al., 2019) and one in Python (collected by us).
We implement the proposed multi-extent exemplar composition mechanism on top of the state-of-the-art Graph2Edit model (Yao et al., 2021).
On both datasets, our model outperforms baseline methods by 8.0-10.9% in terms of absolute accuracy.
In addition, our experimental results show that (1) enabling few support exemplars can greatly improve code editing; (2) compared with equally treating all support exemplars, precisely capturing the code snippet similarity and compositing support exemplars are crucial for code editing; and (3) the multi-extent approach offers better performance compared to its single-extent counterpart and other existing methods.
P RELIMINARY
In what follows we shall use C − and C + to represent the abstract syntax trees of the input (previous) and the output (edited) code snippets.
We address the problem of code editing learning with few exemplars.
Giving a set of K support exemplars with an identical editorial pattern { ( C s k − , C s k + ) } Kk =1 , the goal is to adapt the editing from support set to a query snippet C q − and obtain its desirable C q + .
Here we introduce one of the state-of-the-arts edit learning models Graph2Edit (Yao et al., 2021), which builds a one-shot code editing framework with three major components.
(1) A tree encoder based on Gated Graph Neural Networks (Li et al., 2015) to embed the input abstract syntax tree C − into node representations Z ∈ R N × D , where N is the number of nodes in the tree and D is the latent feature dimension.
The whole tree embedding t can be obtained by a graph pooling operation over all node representations; (2) An editing encoder to embed input C − and corresponding output C + into a vector f ∆ ( C − ,
C + ) ∈
R D to represent the entire L sequential editing actions (four types of actions: add/delete a node, add a subtree, or stop editing) from C − to C
+ as well as the procedural editorial pattern; (3) A decoder for editing action prediction working conditionally on the tree embedding t and editing representation f ∆ ( C − , C + ) .
Predictions include the operational type, the executive location on abstract syntax tree, and associated action values, e.g., to be replaced by which node.
Denote t 1: l = ( t 1 , . . .
, t l ) as the tree embedding history along the sequential editing, and a 1: l = ( a 1 , . . .
, a l ) as the editing action history until step l .
Formally, the whole predictions of actions can be represented as a conditional likelihood Pr ( a 1: L | f ∆ ( C − , C + ) , t 1 ) =
(cid:81)
Ll =1 Pr ( a l | f ∆ ( C − , C + ) , t 1: l ) .
An earlier model Graph2Tree (Yin et al., 2019), where Graph2Edit derives from, shares the same paradigm but edits the input in one-pass, instead of sequentially predict and apply the edit actions.
The Graph2Edit framework is trained with self-reconstruction.
The training feeds the tree encoder and the editing encoder with a pair of C − and C + , and maximizes the likelihood of the whole predictions of actions towards the shortest sequential actions produced by dynamic programming.
The training details with the dynamic programming algorithm can be found in Yao et al.
The training only sends the vanilla editing encoder with the ground-truth from one exemplar, while { ( C s k − , C s k + ) } Kk = with an identical editorial pattern are hard to collect from a wild code dataset.
Based on the training, Graph2Edit is mainly designed for code editing with only one support exemplar.
It is struggled to directly tackle few supports since few exemplars, even within the same editorial pattern, perform divergent coding contexts and edit actions.
We argue the one-exemplar case lacks the capacity for compositional generalization on editorial patterns as we discussed in the introductory part.
We also empirically evidence this claim in our experimental section.
Our method composites f ∆ ( C s k − , C s k + ) from support set based on a multi-extent query-support matching to form the query edit representation f q ∆ for decoding purpose which done by the code editing frameworks.
In the next section, we elaborate on our proposed adaptive multi-extent similarity ensemble on exemplar composition to address the foregoing challenges.
A DAPTIVE M ULTI -E XTENT C OMPOSITION
We assume that if the input tree of a support exemplar enjoys more similarity with the query input tree, the query snippet is more likely to adopt the support exemplar’s edit representation towards correct code editing.
Based on this, we address compositional learning from a query-support matching perspective.
Deriving from the basis of code editing reference, some editing can be adapted depending on individual node representation and regardless of its neighbor nodes and context.
For instance, the removal of a redundant bracket should partially be independent of the inside context, or the change of some outdated function calling mode in a previous language version should be invariant to the host object.
On the other hand, some editing highly relies on the contextual information among several objects and connections therein, leading the editing learning to be determined by a collective representation.
Continuing with this analysis, we design a multi-extent measurement to perform multiple intermediate query-support matching between individual nodes and the collective tree.
Specifically, we measure the multi-extent similarities between query and support snippets in the tree embedding space, where we can define the coverage of nodes in a tree we want to involve for matching (from a single node to every node), then ensemble and learn a convex combination of the editing representations from the support set to maximize the adaptability for query snippet editing.
In the coming subsections, we introduce the query-support matching over abstract syntax tree, extend it with multi-extent representation in an ensemble mechanism, then illustrate the meta-learning paradigm and inference procedure.
3. Q UERY -S UPPORT M ATCHING : F ROM C OLLECTIVE T REE TO I NDIVIDUAL N ODE
Instead of treating each node equally, our model softly emphasizes some nodes in the query snippet when meet a similar node in a support exemplar, and vice versa.
Considering one query snippet and a set with K support exemplars, let z qn and z s k n denote the n -th node representation in the abstract syntax tree of the query and k -th support snippet, respectively.
Let φ θ ( · , · ) :
Z × Z → R be a query-support node matching similarity function (e.g., a neural network) with a learnable parameter θ that takes the node representation from both sides as input.
Note that φ θ ( · , · ) does not have to be non-negative or symmetrical since the measurement across query and support can be deemed as directional.
We compute the initial query-support node matching activation m qs k n and m s k n as follows:
[Equation]
where N q and N s are the numbers of nodes in the query and support abstract syntax trees.
The maximum operation over the set of one-to-all node emphasizes the matching between individual nodes, and returns a high activation when there is at least one good match in the counterpart.
The activation represents the likelihood that a query node appears in a support snippet and vice versa.
To capture the multi-extent matching, we design a λ -softmax function by scaling the importance of nodes in an abstract syntax tree, and seek the query-support matching at the intermediate position between individual nodes and a collective tree:
[Equation]
σ q,s k n,λ denotes the parameterized node activation from the query snippet to the k -th support snippet, and a similar expression can denote as σ s k ,q n,λ .
Based on the above normalized activation for individual node representation, the collective tree representation of the query and support snippets can be calculated using a weighted average pooling as follows:
[Equation]
The representation of the query snippet has the summation of matching values on nodes from all support exemplars.
Combining Equation (2) and Equation (3), an intuitive interpretation on variable λ raises: a larger λ denotes greater domination of matched individual nodes in the final tree representation, i.e., the sharpness of the outcomes after λ -softmax normalization.
λ → 0 preserves the final query/support representation as to their initial tree representation, meaning it calculates a more smoothly weighted average of node representations over the entire tree.
λ → ∞ represents the final query/support representation approximately with only one single node representation most of the time, where the node is selected upon the maximum activation from Equation (1).
Setting a large λ value implies a smaller coverage over the tree.
This coverage scaling property holds due to the monotonous increase of the first-order gradient of the exponential function.
We will tackle the λ value selection problem in the next subsection.
Currently, we reach the generation of edit representation for query sample editing, and the updated tree representation yields the expression:
[Equation]
where ϕ θ ( · , · ) is a similarity measure over query and support tree representations and should satisfy the convexity in the above combination.
Note that for a simplified expression, we do not distinguish the learnable parameter θ for each module but their parameters are isolated.
3. M ULTI -E XTENT C OMPOSITION AND E NSEMBLE
The λ -softmax in Equation (2) controls the matching extent from individual node to collective tree.
However, it is hard to access the optimal intermediate position, since code editing samples suffer from a huge variance.
Therefore, a single extent may not be robust enough to help the model generalize in diverse editing scenarios.
We hereby propose a multi-extent measurement to complement the above weakness by setting λ with different continuous values, and ensemble all the results ϕ θ ( t qλ i , t s k λ i ) for { λ i } N λ
i
=1 to enhance the generalization on complicated code snippets.
However, setting λ arbitrarily and ensemble all perspectives linearly without selection may corrupt the overall tree representation, and thus deteriorate the model.
For a robust ensemble and aggregation, we consider assessing the quality of ϕ θ ( t qλ i , t sλ i ) towards different λ i by using a marginal ranking error on edit representation.
Since we have the accessible ground-truth C q + during training, we employ the ranking error
e λ i to explicitly reflect how good ϕ θ ( t qλ , t s k λ ) is under a certain λ with respect to the similarity between query and K support exemplars.
In the inference phase, due to the missing of C q + , we employ a similarity-ranking error estimator ˆ e λ i =
R µ ( t qλ i , t s 1 λ i , t s 2 λ i , . . .
t s K λ
i ; λ i ) :
R ( K +1)
× D → R to predict the quality of each λ i , where µ is the learnable parameter in R µ , and ensemble the outcomes.
Training details of R µ will be elaborated on in the next subsection.
To achieve multi-extent composition and ensemble, we first measure the similarity between query and support edit representations via the editing encoder:
[Equation]
Then we access the error for each λ by:
[Equation]
where Γ( · ) is the index mapping for K support exemplars that satisfy w Γ(1) ≥ w Γ(2) ≥ . .
≥ w Γ( k ) , i.e., the mapping to sort s k in a descending order, and γ is the margin set as a hyperparameter.
The formulation indicates how well the nearest support exemplars can outperform other support exemplars, and the factor ( w Γ(1) − w Γ( k ) ) reveals the error confidence.
Note that the above error term depends on the edit model f ∆ , which is only an approximate estimation.
Then we extend Equation (4) by involving multiple extents and the reciprocal of its error for adaptive ensemble:
[Equation]
where ϵ is a constant in denominator for numerical stability.
We omit the normalization over ϕ θ ( t qλ i , t s k λ i ) /
( e λ i + ϵ ) in the equation.
In practice, we apply a softmax function on this term to keep the convexity of the combination.
Note that in inference e λ i will be replaced by ˆ e λ i .
3. L EARNING AND I NFERENCE
In the learning phase, we have a set of K support exemplars { ( C s k − , C s k + ) } Kk =1 and one complete query snippet { C q − , C q + } as inputs.
Following the Graph2Edit framework (Yao et al., 2021), we obtain the z q n and z s k n of the n -th node representation of the query and k -th support snippet and the edit representation f ∆ ( C s k − , C s k + ) for each support exemplar.
L q is the number of steps towards the shortest sequential actions from C q − to C q + , and a 1: L q denotes the corresponding action sequence.
Let e =
( e λ 1 , e λ 2 , . . .
, e λ N ) calculated by Equation (6) and ˆ e =
(ˆ e λ 1 , ˆ e λ 2 , . . .
, ˆ e λ N ) is its estimation from the estimator R µ .
With N λ extents, our objective function can be written as:
[Equation]
where || · || 1 is the vector l 1 -norm to sum up the absolute values of all elements, D KL ( ·∥· ) is the KLdivergence, f q ∆ is the composition of edit representations in Equation (7), t q = (cid:80) N q n =1 z qn /N q is the initial tree representation of C q − , θ and µ are the learnable parameters in the similarity measurement ϕ θ , φ θ and the estimator R µ .
We omit the balanced factor for each term for a simplified expression.
The above objective function consists of three parts.
The first term maximizes the likelihood between the learned convex combination of edit representations and optimal editing actions, while the rest two terms aim to minimize the ranking error between the query and support snippets at different extent levels and train an estimator to accurately estimate the error for inference usage.
The parameters for tree encoder, editing encoder, and decoder are obtained via a pre-training stage on another data split following the self-reconstruction paradigm from Graph2Edit, while their parameters are not tuned in the meta-learning stage for optimization, since we observe a serve overfitting if doing so.
We follow the episode training mechanism (Snell et al., 2017) which is widely used in few-shot learning to simulate the encounter of new editorial patterns in testing.
For each training forward, we sample a fixed number of editorial patterns with K support exemplars and one query snippet per class and backpropagate the loss.
In the inference phase, given a set of K support exemplars { ( C s k − , C s k + ) } Kk =1 and one input query snippet C q − , we replace e λ i with ˆ e λ i in Equation (7) and yield the formulation.
E XPERIMENT 4. E XPERIMENTAL S ETTING
Dataset We include two code editing datasets for performance evaluation.
(1) C#Fixer (Yin et al., 2019) is a dataset containing 2,878 editing pairs, generated by applying two C# fixing tools Roslyn (Microsoft, 2021) and Roslynator (Pihrt, 2020) on six projects.
A fixer is building on top of the C# compiler used to perform refactoring and modernization on codes.
One fixer is designed for one type of refactoring and 16 fixers are selected in this dataset.
A pre-trained model for C#Fixer is trained on GitHubEdits dataset using self-reconstruction.
The dataset collects 54 C# projects on GitHub which extract the source code before and after the commits from users.
We use the data of GitHubEdits only for pre-training and follow the training protocol in Yao et al.
(2021) but not for few-shot experiments.
(2) Benefiting from the flexibility and generalized capacity of abstract syntax description language, beyond C#, we collect another dataset PyFixer written in Python 2.
We use Python-Future (Ltd, 2019), a tool that edits the codes in 5,959 projects written in Python 2 to make them compatible for Python 3.
We obtain the code pairs by taking codes before and after applying the tool and inspecting the changes.
To simulate the pre-training and few-shot learning as in GitHubEdits and C#Fixer , we select 12 types of fixers and extract a fixed number of each fixer from the main data to form the dataset for few-shot learning, while the rest are preserved for pre-training.
After data cleaning, we have 14,941 code pairs for (pre) training, 1,664 for validation, and 2,319 for few-shot experiments.
The two datasets only contain source code but without any author’s name, contact information, or offensive content.
Baseline We employ modifications on two abstract syntax tree-based code editing frameworks,
Graph2Tree (Yin et al., 2019) and Graph2Edit (Yao et al., 2021), to fit them into the few exemplars editing setting and involve the following baselines.
Note that all baselines are only different in calculating the similarity measure ϕ θ in Equation (4).
RS (Random Selection) : we randomly select one support exemplar from the support set and use its edit representation for standard decoding on query snippet; NN (Nearest Neighbor) :
we identify the nearest neighbor in the support set based on the distance calculated by graph edit distance (Sanfeliu & Fu, 1983; Abu-Aisheh et al., 2015) over AST trees ( GED-NN ), and cosine similarity over input tree representation ( CS-NN ).
AE (Average
Edit Representation) : we infer the edit representation of K support exemplars using editing encoder respectively, and take the mean of these K representations for query decoding.
This is equivalent to setting ϕ θ = 1 /K in Equation (4).
Moreover, we include two other baselines within our proposed composition framework.
GED-Comp (Composition via graph edit distance) employs the graph edit distance metric as ϕ θ when comparing the query and the support exemplar among every querysupport snippet pair; we take the reciprocal proportion of this metric and form a convex combination over K edit representations.
CS-Comp (Composition via cosine similarity) means that we employ mean pooling over query and support trees, compute the similarity ϕ θ by cosine similarity on tree representations for every query-support snippet pair, normalize, and form a convex combination
[Table]
over the entire available edit representations from support set.
Finally, Ours is the proposed model which learns the similarity measure ϕ θ as described in Section 3, where we additionally consider an ensemble of multi-extent similarity scores, as shown in Equation (7).
Note that we find Graph2Tree contains numerous C# specific implementations and is currently not applicable for Python, so we do not include this baseline on PyFixer .
All the baselines are under MIT license for public usage.
Protocol We randomly split fixers for few-shot experiments into meta-train and meta-test sets.
For
C#Fixer , the meta-train set contains 12 types of fixers and the meta-test set has the rest 4 fixers.
The meta-train set of PyFixer contains 8 fixers and the meta-test set takes the rest 4.
To perform a comprehensive empirical evaluation, we repeat the random split procedure and obtain 5 different splits denoted from Split #1 to Split #5.
For meta-training, to simulate the low-resource scenario and mitigate the imbalance in samples per fixer, we take 10 samples for each fixer and follow a standard episode training strategy.
Every time we construct the support set to train on one query sample, we randomly select the exemplars who are sharing the same fixing type with the query sample in the sampled data.
For meta-testing, we iterate through all samples and each of them as a query snippet at a time, then randomly sample from the rest of its fixer peer to build its support set.
All numerical results are run with 5 different random seeds.
Mean and standard deviation are reported.
Note that the variance not only stems from the training per se, but also is affected by the random sampling of support set in testing.
Since we fix the random seed every time, all methods are guaranteed to encounter identical query-support combinations in the evaluation under the same seed.
All configurable hyperparameters are tuned on a validation split outside the five evaluative splits.
We defer the model architecture and configurable hyperparameters to supplementary material.
The accuracy per pair is a binary value obtained by whether the model edits C − to meet the exact ground-truth C + .
Following Yao et al.
(2021), we report both Macro and Micro accuracy since there is a significant fixer imbalance in C#Fixer .
K is set to 5 and λ is set to the ensemble over (1.0, 2.0, 4.0, 10.0) by default if no specification is noted.
We instantiate φ θ (in Equation (1)) as neural networks and ϕ θ (in Equation (4)) as cosine similarity.
Experiments are run on one NVIDIA RTX 3090.
We complement more technical details in Appendix B.
4. R ESULT AND A NALYSIS
Main numerical results of code editing with 5 support exemplars on C#Fixer and PyFixer are presented in Table 1 and Table 2.
Compared to baseline methods, we nearly achieve the best results on all splits of two datasets, across macro and micro accuracy.
We outperform the second-best baseline method by bringing an absolute improvement range from 8.0% (Macro accuracy on C#Fixer ) to 10.9% ( PyFixer ).
[Table] With the base architecture (exclude Graph2Tree), compared to and CS, our improvements show the significance of multi-extent composition learning of edit representations from support set.
Quantitative results demonstrate the effectiveness of treating nodes in abstract syntax tree differently and ensemble learning by the meta-learning strategy.
We present extra experiments in this subsection as well as in Appendix C to further investigate our approach.
We attach a failure case analysis and a visualization diagram in Appendix D and Appendix E, respectively.
More Support Exemplars Figure 2 shows the performance trends when enlarging the size of support set, i.e., offering more resources for code editing.
An estimated regression with uncertainty is plotted from K = 3 to K = 7 .
It is relatively straightforward that the performance on random selection is not affected by the scale of support set.
The average edit representation performs differently on two datasets.
Adding more exemplars into support set can help to improve the robustness of the weighted mean representation, or lead it to deviate from the representational manifold if high variances exist among exemplars.
Our approach consistently benefits from more support exemplars and gradually enlarges the margin towards the two baselines.
This demonstrates our model is capable to find a better composition when there are more options, therefore preserving the possibility of capability enhancement when more resources are available.
Investigation of Multi-Extent Ensemble Figure 3 shows the functional analyses on multi-extent measurement and its ensemble, where we investigate seven possible options on extent parameter λ , four in a single extent, three in a double extent ensemble, and compare to the full model with four levels of extents plotting in red.
A single extent in the first four rows underperforms the ensemble when using two or four extents.
This confirms our methodological hypothesis that a single extent may not be enough to express the similarity between the query-support snippet pair due to the high variances and that ensembling multiple extents helps the model robustness.
The full model with four extents also performs better than the double extent ensemble shown by the median line of the box plot.
However, adding more extents implies increasing computational costs and hence a trade-off.
R ELATED W ORK
Machine Learning for Code Editing Code editing using machine learning has been developed for a long time and is receiving increasing attention in recent years (Menon et al., 2013; Zhang et al., 2019; Tian et al., 2020).
Depending on how a code snippet is represented, the research can be roughly summarized into two categories, i.e., performing code editing over code token sequences or abstract syntax tree.
In the former category, Yin et al.
; Chen et al.
(2019); Yasunaga & Liang (2020) directly generate the expected edited code token sequence, while Shin et al.
; Vu & Haffari (2018); Dong et al.
; Zhao et al.
(2019) predict the editing operational sequence, which has been demonstrated to bear more sample efficiency.
On the other hand, works in the second category (Yin et al., 2019; Chakraborty et al., 2020; Tarlow et al., 2020; Dinella et al., 2020; Brody et al., 2020; Yao et al., 2021) attempt to edit the trees of programs similarly by either directly generating the edited trees or predicting the tree edit operations.
The problem addressed in this paper is conceptually related to ‘programming by example’ in software engineering that finds the sharing edit operations among different exemplars and applies it to query snippet (Menon et al., 2013; Osera & Zdancewic, 2015; Ferdowsifard et al., 2020; Meng et al., 2013).
Recently, Yin et al.
; Yao et al.
(2021) tackle this problem using deep learning techniques.
Nonetheless, both of the two works have focused on code editing from one shot of exemplar, which may not be sufficient for learning generalizable editing, as we elaborated in Section 1.
This inspires us to investigate few-shot code editing.
Few-shot Learning Few-shot learning has been developed in many low-resource scenes (Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017), and with advance in meta seq2seq learning considering the concept of generalized composition (Gu et al., 2018; Lake, 2019; Nye et al., 2020).
Our work conceptually relates to the matching principle in few-shot learning (Vinyals et al., 2016;
Snell et al., 2017) but with a different compositional task and a specific design for robust multiextent abstract syntax tree matching.
To our best knowledge, we are the first to consider fewshot compositional generalization in code editing, which we believe can inspire development in programming-related applications like code transfer, refactoring, and migration.
C ONCLUSION
In this work, we considered code editing with few exemplars.
Based on previous frameworks on code editing using abstract syntax tree to present the computer code snippets, we proposed an adaptive multi-extent composition method to perform varying intermediate representations between a collective tree and individual nodes.
We leveraged an ensemble approach to gather the query-support matching from multiple extents and delivered a robust composition over support editing representations for the query snippet decoding.
Evaluations on two code editing datasets demonstrated the effectiveness of our method over baselines by a large margin.
To develop this framework into a product, we consider letting the user collect a few exemplars showing their own purpose.
The tool requests an interaction but also offers full flexibility to the user, and is more adapted to a personal development environment.
E THICS S TATEMENT
We discuss the potential ethical considerations in this section.
We do not discover a noteworthy negative societal impact originating from our work.
One minor concern is the research in machine learning for source code may encourage and facilitate the code data collection from open-source projects.
The data may unintentionally involve user privacy like the author’s name, organization, or file date and the collection process may violate the usage license
A E DITING E XAMPLES FOR F IXERS IN PyFixer
We list the 12 kinds of fixer names in PyFixer in Table 3. Fixer [Table] Fixer
[Table]
lib2to
[Table]
Fixer
[Table]
return type(self)
[Table]
type(other
[Table]
and self.__dict__ ==
other.__dict_
[Table]
return isinstance(self
[Table]
type(other)) and self.__dict__ ==
[Table]
__dict_
[Table]
Fixer 15: lib2to3.fixes.fix_map
[Table]
map
[Table]
[Table]
in kwargs
[Table]
values
[Table]
) if a
[Table]
= kwargs[’releaselevel’]])
[Table]
[Table]
in kwargs
[Table]
values
[Table]
if
[Table]
[Table]
[Table]
Fixer 18: lib2to3.fixes.fix_next
[Table]
Fixer
[Table]
lib2to3.fixes.fix_raw
[Table] input [Table] 0:1 and press enter.
".format(str(parsed.ip), str(parsed.p)))
Fixer 34: lib2to3.fixes.fix_zip
C − context.update(zip(inlines_names, kwargs.get(’inlines’, [])))
C + context.update(list(zip(inlines_names, kwargs.get(’inlines’, []))))
Fixer 34: lib2to3.fixes.fix_absolute_import
C − from elks import Elks
C + from .elks import Elks
Fixer 41: lib2to3.fixes.fix_future_standard_library
C − import urlparse
C + import urllib.parse
Fixer 42: lib2to3.fixes.fix_future_standard_library_urllib
C − from urllib2 import HTTPError, urlopen, Request
C + from urllib.request import urlopen, Request
Fixer 42: lib2to3.fixes.fix_print_with_import
C − print(’site_data_dir’, app.locations.site_data_dir)
C + print((’site_data_dir’, app.locations.site_data_dir))
Fixer 49: lib2to3.fixes.fix_unicode_keep_u
C − username =
unicode(origin.getFrom()).split(’/’)[1].replace(" ","")
C + username =
str(origin.getFrom()).split(’/’)[1].replace(" ","")
B E XPERIMENTAL D ETAILS
We collect the PyFixer dataset by separately applying 51 fixers in Python-Future (Ltd, 2019) over 5,959 projects written in Python 2 to make them compatible with Python 3.
We obtain the code pairs by inspecting the changed lines and get the source code before and after the refactoring.
We select 12 types of fixers and extract 200 samples for each fixer from the main data to build the dataset for few-shot learning, while the rest are preserved for model pre-training.
For training, we set the batch size to 16, epochs to 15, learning rate to 1e-4, with a gradient accumulation for every 2 optimized steps.
The extent parameter λ for main experiments are set as an ensemble for 10 , 5 , 2 , and 1 .
The architectural parameters are set the same as Graph2Edit (Yao et al., 2021).
The predictor R is implemented with neural networks in two layers with Leaky ReLU.
The learnable function φ θ ( · , · ) is implemented with one linear layer followed by cosine similarity between the two input terms.
The function ϕ ( · , · ) is a cosine similarity function on edit representation.
The following equation describes the calculation of macro and micro accuracy.
Consider a set of class C , T c is the number of correctly edited samples in class c , N c is the total number of samples in class c .
These two metrics can be expressed by
[Equation]
Note that, in our experiments, an edited code is considered “correct” only when it is exactly the same as the ground truth in both syntax and semantics.
The PyFixer are working under Python 3.8, having the abstract syntax tree grammar from https://docs.python.org/3.8/library/ast.html.
We list the choice of splits for the two datasets below.
[Table] [Table]
We use the fixer’s name in Table 4 and the fixer’s index in Table 5 corresponding to the order in the
Python-Future package.
C A DDITIONAL E XPERIMENT
We extend our experiments by considering the ‘Hit-1-in-5’ and ‘Hit-5-in-5’ cases when editing with one exemplar from the given few exemplars.
In the ‘Hit-1-in-5’ case, we separately decode the query code snippet using the K exemplars and obtain K results.
We count the result of the query sample as correct if any of these K decoding results match with the ground truth.
Similarly, in the ‘Worst’ case, we count it as correct when all of these K results are correct.
Note that both the ‘Hit-1-in-5’ and ‘Hit-5-in-5’ cases are not real in practical usage since we only expect one output from the editing system.
Giving possible answers as many as it can is not user-oriented and not scalable.
The ‘Hit-1-in-5’ and ‘Hit-5-in-5’ cases can be understood as the upper and the lower bound of results if we randomly select one exemplar from the support set
but they are almost not achievable.
We include an experiment using every exemplar in Figure 4.
[Table]
[Table] [Table]
[Table]
D F AILURE C ASE
We analyze the failure case in this section.
A failure case on editing the ‘&&’ logical operation is presented in Table 8.
In this case, most of the exemplars showcase the editorial pattern with complicated context, thus it is hard for our algorithm to extract the sharing editing.
Meanwhile, two of these exemplars concurrently edit with the attribute ‘Any()’ and consequently mislead the query code editing.
E V ISUAL D IAGRAM FOR N ODE A CTIVATION ON Q UERY -S UPPORT M ATCHING
We visualize the node activation in Figure 5.
The value of activation is normalized within the query or support snippet to show the relative magnitude of the value.
Matching on individual nodes facilitates query snippet to find the valuable reference in Support #1 under ‘BinaryExpression’ regardless of the disturbing content under ‘ConditionalExpression.’
Also, there is a faint activation on the multiple operators in Support #2 since the objects on this operation is under ‘MemberAccessExpression.’