M ERGE BERT: P ROGRAM M ERGE C ONFLICT R ESOLUTION VIA N EURAL T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Collaborative software development is an integral part of the modern software development life cycle, essential to the success of large-scale software projects.
When multiple developers make concurrent changes around the same lines of code, a merge conﬂict may occur.
Such conﬂicts stall pull requests and continuous integration pipelines for hours to several days, seriously hurting developer productivity.
In this paper, we introduce MergeBERT, a novel neural program merge framework based on the token-level three-way differencing and a transformer encoder model.
Exploiting restricted nature of merge conﬂict resolutions, we reformulate the task of generating the resolution sequence as a classiﬁcation task over a set of primitive merge patterns extracted from real-world merge commit data.
Our model achieves 63–68% accuracy of merge resolution synthesis, yielding nearly a 3 × performance improvement over existing structured, and 2 × improvement over neural program merge tools.
Finally, we demonstrate that MergeBERT is sufﬁciently ﬂexible to work with source code ﬁles in Java, JavaScript, TypeScript, and C# programming languages, and can generalize zero-shot to unseen languages.
I NTRODUCTION
Collaborative software development relies on version control systems such as git to track changes across ﬁles.
In most projects, developers work primarily in a branch of a software repository, periodically synchronizing their code changes with the main branch via pull requests (Gousios et al., 2016).
When multiple developers make concurrent changes to the same line of code, a merge conﬂict may occur.
According to an empirical study of four large software projects by Zimmermann (2007) up to 46% of all merge commits result in conﬂicts.
Resolving merge conﬂicts is a time-consuming, complicated, and error-prone activity that requires understanding both the syntax and program semantics, often taking more time than developing a code feature itself (Bird & Zimmermann, 2012).
Modern version control systems such as git utilize the diff3 algorithm for performing unstructured line-based three-way merge of input ﬁles (Smith, 1998).
This algorithm aligns the two-way diffs of two versions of the code A and B over the common base O into a sequence of diff “slots”.
At each slot, a change from either A or B is selected.
If both program versions introduce a change at the same slot, a merge conﬂict is produced, and manual resolution of the conﬂicting modiﬁcations is required.
A versatile, production-level merge conﬂict resolution system should be aware of programming language syntax and semantics yet be sufﬁciently ﬂexible to work with any source code ﬁles, irrespective of the programming language.
It should generalize to a wide variety of real-world merge conﬂicts beyond a speciﬁc merge type or a domain of software artifacts.
Inspired by the exceptional performance of transformer models and self-supervised pretraining in natural language understanding and generation tasks (Devlin et al., 2019;
Radford et al., 2019;
Liu et al., 2019; Lewis et al., 2019; Raffel et al., 2020) as well as in the programming language domain (Feng et al., 2020; Svyatkovskiy et al., 2020; Clement et al., 2020; Tufano et al., 2020; Ahmad et al., 2021), we introduce MergeBERT: a neural program merge framework based on token-level three-way differencing and transfer learning.
We select a bidirectional transformer encoder (BERT) as our encoder implementation.
As a bidirectional encoder, BERT allows to attend to code context surrounding the conﬂicting chunks, which is a key advantage over left-to-right language models.
To endow our model with a basic knowledge of programming language syntax and semantics, we adopt a two-step training procedure: (1) unsupervised masked language model pretraining on a massively multilingual source code corpus, (2) supervised ﬁnetuning for the sequence classiﬁcation task.
We transfer weights of the pretrained encoder into a multi-input model architecture that encodes all inputs that a standard diff3 algorithm takes (two two-way diffs of input programs) as well as the edit sequence information, then aggregate them for learning.
While MergeBERT utilizes BERT, other encoder architectures such as LSTM (Hochreiter & Schmidhuber, 1997), or efﬁcient transformer variants like Poolingformer (Zhang et al., 2021) could also be utilized for this task.
The paper contributions are as follows: (1) we introduce MergeBERT a novel transformer-based program merge framework that leverages token-level three-way differencing and formulates the task of generating the resolution sequence as a classiﬁcation task over a set of primitive merge patterns extracted from real-world merge commit data, (2) we effectively transfer knowledge about program syntax and types of source code identiﬁers from millions of software programs to downstream sequence classiﬁcation task of merge conﬂict resolution by using self-supervised pretraining (see section 5), which also makes this approach computationally more feasible, (3) we overcome several limitations of the existing neural program merge models Dinella et al.
(2021) and semi-structured program merge tools like jsFSTMerge and JDime to improve upon the state-of-the-art by 2–3 × (see sections 7 and 8), and ﬁnally, (4) we demonstrate that multilingual MergeBERT is sufﬁciently versatile to work with programs in Java, JavaScript, TypeScript, and C#, and can generalize to unseen languages without retraining.
1. R ELATED W ORK
There have been multiple attempts to improve merge algorithms by restricting the them to a particular programming language or a speciﬁc type of applications (Mens, 2002).
Typically, such attempts result in algorithms that do not scale well or have a low coverage.
Syntactic merge algorithms improve upon diff3 by verifying the syntactic correctness of the merged programs.
Several syntactic program merge techniques have been proposed (Westfechtel, 1991; Asklund, 1999) which are based on parse trees or abstract syntax trees and graphs.
Apel et al.
noted that structured and unstructured merge each has strengths and weaknesses.
They developed a semi-structured merge tool, FSTM ERGE , which switches between approaches (Apel et al., 2010).
They later introduced JD IME , an approach that automatically tunes a mixture of structured and unstructured merge based conﬂict locations (Apel et al., 2012).
A UTO M ERGE tool (Zhu & He, 2018) improves upon the JD IME by adding a built-in matcher to check output resolutions for the merges performed by JD IME .
A UTO M ERGE synthesizes resolutions only when the structured merge algorithm fails to complete a merge.
Trindade Tavares et al.
(2019) later implemented jsFSTMerge by adapting an off-the-shelf grammar for JavaScript and modifying the FSTMerge algorithm itself to address JavaScript speciﬁc issues.
In addition, Pan et al.
(2021) explore using program synthesis to learn repeated merge resolutions within a project.
However, the approach is limited to a single C++ project, and only deals with restricted cases of import statements.
Sousa et al.
(2018) explore the use of program veriﬁcation to certify that a merge obeys a semantic correctness criteria, but does not help resolve merge conﬂicts.
M OTIVATING E XAMPLE
In this section, we formulate the traditional line-level merge conﬂict resolution problem as a classiﬁcation task and show how token-level formulation of diff3 helps to localize the merge conﬂicts.
Fig.
1 provides an example merge conﬂict in JavaScript.
Fig.
1(a) on the left shows the standard diff3 markers “<<<<<<< A.js”, “||||||| O.js”, “==
=
= =
=
=”
and “>>>>>>> B.js”, which denote the conﬂicting regions introduced by programs A , base O , and B respectively.
Here, O represents the most common ancestor of programs A and B in the version control history.
We denote the program text of diff3 conﬂicting regions as A j , B j , O j , where j is a conﬂict index.
The conﬂict index may be omitted when referring to programs consisting of a single conﬂict only.
We refer to the program text outside the conﬂicting chunks, common to all merged programs versions, as a preﬁx and sufﬁx, and denote it respectively as Pref and Suff throughout the paper.
First, MergeBERT represents each line-level merge conﬂict instance at token level (Fig. 1(b)) which localizes conﬂicting regions, yielding a ⊂ A , b ⊂ B , o ⊂ O , and then it invokes the neural model to predict a resolution via classiﬁcation (Fig. 1(c)).
Here, and throughout the paper we will use lower case notations to refer to attributes of token-level differencing (e.g. a , b , and o are conﬂict regions produced by diff3 of token granularity).
Intuitively, token diff3 ﬁrst turns the line-structured text into a list of tokens (including space and line delimiters), applies standard diff3 algorithm to the resulting documents, and reconstructs the merged document at line level.
As a result of token-level merge, the whole “ let x = max(y, ” string is cleanly merged, becoming a part of the program preﬁx, and “ ) ” is prepended to the program sufﬁx.
Observe, that the resolution does not consist of any single line from either A or B since both edits modify a common line present in the base.
Hence, earlier neural approaches such as DeepMerge (Dinella et al., 2021) would not be able to synthesize the resolution.
On the other hand, structured merge techniques (such as jsFSTMergeby Trindade Tavares et al.
(2019)) cannot resolve the conﬂict as the conﬂict appears on a program statement, which leads to side effects (e.g. syntactically incorrect code).
A token-level merge can interleave edits within lines (i.e., tokens in which one edit does not conﬂict with another are trivially merged).
Consider A ’s edit of the var to let keyword.
Such nonconﬂicting edits sufﬁce to demonstrate the above.
Likewise, consider the token-level conﬂict for the arguments to the max function: an appropriate model trained on JavaScript should be able to easily deduce that taking the edit from B (i.e., "11, z") captures the behavior of A ’s edit as well.
The suggested resolution gives an intuitive demonstration for how MergeBERT turns a complex line-level resolution into a simpler token-level classiﬁcation problem.
MergeBERT can deal with non-trivial real-world merges composed of multiple conﬂicting chunks.
We include a complete example of such a merge conﬂict in the Appendix.
[Figure]
B ACKGROUND : D ATA - DRIVEN M ERGE
Dinella et al.
(2021) introduced the data-driven program merge problem as a supervised machine learning problem.
A program merge consists of a 4-tuple of programs ( A , B , O , M ) , where
The base program O is the most common ancestor in the
version
history for programs A and B , 2.
diff3 produces an unstructured line-level conﬂict when applied to ( A , B , O ) , and 3.
M is the program with the developer resolution, having no conﬂicts.
Given a set of such programs and merges ( A , B , O , M ) , the goal of a data-driven merge is to learn a function, merge , that maximizes the set of examples where merge( A , B , O ) =
M .
Moreover, since a program may have multiple unstructured conﬂicts ( A j , B j , O j , M j ) , j=0...
N, the data-driven merge considers the different merge tuples corresponding to the conﬂicting regions independently , and poses the learning problem over all the merge tuples present in ( A , B , O , M ) .
Dinella et al.
(2021) also provide an algorithm for extracting the exact resolution regions for each merge tuple and deﬁne a dataset that corresponds to non-trivial resolutions where the developer does not drop one of the changes in the resolution.
Further, they provide a sequence-to-sequence encoder-decoder based architecture, where a bi-directional gated recurrent unit (GRU) is used for encoding the merge inputs comprising of ( A, B, O ) segments of a merge tuple, and a pointer mechanism is used to restrict the output to only choose from line segments present in the input.
Given the restriction on copying only lines from inputs, the dataset deﬁned in the paper did not consider merges where the resolution required token-level interleaving, such as the conﬂict in Figure 1.
And, lastly, the dataset consists of merge conﬂicts in a single language, JavaScript.
In contrast, our paper addresses both of these limitations.
M ERGE C ONFLICT R ESOLUTION AS A C LASSIFICATION T ASK
In this work, we demonstrate how to exploit the restricted nature of merge conﬂict resolutions (compared to an arbitrary program repair) to leverage discriminative models to perform the task of generating the resolution sequence.
We have empirically observed that a diff3 of token granularity enjoys two useful properties over its line-level counterpart: (i) it helps localize the merge conﬂicts to small program segments, effectively reducing the size of conﬂicting regions, and (ii) most resolutions of merge conﬂicts produced by token diff3 consist entirely of changes from a or b or o or a sequential composition of a followed by b or vice versa.
On the ﬂip side, a token-level merge has the potential to introduce many small conﬂicts.
To balance the trade-off, we start with the line-level conﬂicts as produced by the standard diff3 and perform a token-level merge of only the segments present in the line-level conﬂict.
There are several potential outcomes for such a two-level merge at the line-level:
• A conﬂict-free token-level merge : For example, the edit from A about let is merged since B does not edit that slot as shown in Fig. 1(b).
• A single localized token-level merge conﬂict :
For example, the edit from both A and B for the arguments of max yields a single conﬂict as shown in Fig. 1(b).
• Multiple token-level conﬂicts : Such a case (not illustrated above) can result in several token-level conﬂicts.
Token-level diff3 applied to a merge tuple ( A , B , O , M ) , would generally result in a set of localized conﬂicting regions and resolutions (cid:104) a j , b j , o j , r j (cid:105) j .
We empirically observe that majority of such r j are comprised entirely of ( i ) a j , ( ii ) b j , ( iii ) o j , concatenating ( iv ) a j , b j or ( v ) b j , a j , and variants of ( iv – v ) having the lines present in the base excluded, comprising a total of nine primitive merge resolution patterns (see section 10.1 in Appendix for more details about the primitive merge patterns).
We, therefore, can treat the problem of constructing r j as a classiﬁcation task to predict between these possibilities.
It is important to note that although we are predicting simple resolution strategies at token-level, they translate to complex interleavings at line-level.
Of course, not all line-level conﬂicts are resolved by breaking that conﬂict to tokens—some resolutions which are complex line-based interleavings are not expressible as a choice at the token-level.
The key practical advantage of formulating merge conﬂict resolution as a classiﬁcation task is a signiﬁcant reduction in total FLOPS required to decode a resolution region, as compared to generative models, making this approach an appealing candidate for deployment in IDEs (see section 11.1).
M ERGE BERT: N EURAL P ROGRAM M ERGE F RAMEWORK
MergeBERT is a textual program merge model based on the bidirectional transformer encoder (BERT) model.
It approaches merge conﬂict resolution as a sequence classiﬁcation task given conﬂicting regions extracted with token-level differencing and surrounding code as context.
The key technical innovation to MergeBERT lies in how it breaks program text into an input representation amenable to training with a transformer encoder and how it aggregates various input encodings for classiﬁcation.
MergeBERT exploits the traditional two-step pretraining and ﬁnetuning training procedure.
We use self-supervised masked language modeling (MLM) pretraining on a massively multilingual source
[Figure]
code corpus followed by supervised ﬁnetuning for a classiﬁcation task (section 11.4 provides details about the implementation).
For ﬁnetuning, we construct a multi-input model architecture that encodes pair-wise aligned token sequences of conﬂicting programs A and B with respect to original program
O , as well as corresponding edit sequence steps, then aggregate them for learning.
An overview of
MergeBERT model architecture is shown in Fig.
Given a merge tuple ( A , B , O , M ) with token-level conﬂicting chunks a j , b j , o j , MergeBERT models the following conditional probability distribution:
[Equation]
and consequently, for entire programs:
[Equation]
where N is the number of token-level conﬂicts in the merge tuple ( A , B , O , M ) .
5. R EPRESENTING M ERGE C ONFLICTS
As shown by Dinella et al.
(2021), an effective merge representation needs to be “edit aware” to provide an indication that A and B are edits of the original program O .
Prior work on distributed representations of edits (Yin et al., 2019) describes how to compute a two-way diff using a standard deterministic difﬁng algorithm and represent the resulting pair-wise alignment as a vector consumable by machine learning models.
Given a merge tuple ( A , B , O , M ) , MergeBERT ﬁrst calculates two two-way alignments between the sequence of tokens of conﬂicting regions a (respectively b ) with respect to that of the original program o .
For each pair of aligned token sequence we extract an “edit sequence” that represents how to turn the second sequence into the ﬁrst.
These edit sequences – ∆ ao and ∆ bo – are comprised of the following editing actions (kinds of edits): = represents equivalent tokens, + represents insertions, - represents deletions, ↔ represents a replacement, and ∅ is used as a padding token.
Overall, this produces four token sequences and two edit sequences: ( a | o , o | a , and ∆ ao ) and ( b | o , o | b , and ∆ bo ).
Each token sequence covers the corresponding conﬂicting region and, potentially, surrounding code tokens.
Fig.
2 shows an example of edit sequence.
5. C ONTEXT E NCODING
We pretrain a BERT model E following the masked language modeling objective on a multilingual dataset of source code ﬁles.
In each source code ﬁle, a set of tokens is sampled at random uniform and replaced with [MASK] symbols, and the model aims to reconstruct the original sequence.
We make use of a Byte-Pair Encoding (BPE) unsupervised tokenization procedure to avoid a blowup in the vocabulary size given the sparse nature of code identiﬁers Karampatsis et al.
Besides code tokens, the vocabulary includes the special symbols representing editing steps and the [MASK] symbol.
During ﬁnetuning, we introduce an edit type embedding combining it with token and position embeddings via addition: S = S T + S P + S E .
Edit type embedding helps the model recognize the edit steps, which are not supplied during pretraining.
See Fig.
5 in the Appendix for details.
As shown in Fig.
2, we utilize pretrained encoder model to independently encode each of the four token sequences ( a | o , o | a , b | o , and o | b ) of merged programs, passing edit sequences ( ∆ ao and ∆ bo ) as type embedding indices.
5. M ERGE T UPLE S UMMARIZATION
In standard sequence learning tasks there is one input and one output sequence.
In merge conﬂict resolution setting, there are multiple input programs and one resolution.
To facilitate learning in this setting, we construct MergeBERT as a multi-input encoder neural network, which ﬁrst encodes token sequences a | o , o | a , b | o , and o | b , and then aggregates them into a single hidden summarization state:
[Equation]
where E is the context encoder and E ( x i , ∆) are the embedding tensors for each of the sequences x i ∈ ( a | o , o | a , b | o , o | b ) .
After encoding and aggregation a linear classiﬁcation layer with softmax is applied:
[Equation]
The resulting line-level resolution region is obtained by concatenating the preﬁx pref , predicted token-level resolution r j , and the sufﬁx suff .
Finally, in the case of a one-to-many correspondence between the original line-level and the token-level conﬂicts (see 11.5 in the Appendix for an example and 11 for pseudocode), MergeBERT uses a standard beam-search to decode the most promising predictions.
D ATASET
To create a dataset for pretraining, we clone all non-fork repositories with more than 20 stars in
GitHub that have C, C++, C#, Python, Java, JavaScript, TypeScript, PHP, Go, and Ruby as their top language.
The resulting dataset comprises over 64 million source code ﬁles.
The ﬁnetuning dataset is mined from over 100 thousand open source software repositories in multiple programming languages with merge conﬂicts.
It contains commits from git histories with exactly two parents, which resulted in a merge conﬂict.
We replay git merge on the two parents to see if it generates any conﬂicts.
Otherwise, we ignore the merge from our dataset.
We follow Dinella et al.
(2021) to extract resolution regions—however, we do not restrict ourselves to conﬂicts with less than 30 lines only.
Lastly, we extract token-level conﬂicts and labels from line-level conﬂicts and resolutions.
Tab. 4 provides a summary of the ﬁnetuning dataset.
B ASELINE M ODELS 7. L ANGUAGE M ODEL B ASELINE
Neural language models (LMs) have shown great performance in natural language generation (Radford et al., 2019; Sellam et al., 2020), and have been successfully applied to the domain of source code (Hindle et al., 2012;
Svyatkovskiy et al., 2020;
Feng et al., 2020).
We consider the generative pretrained transformer language model for code (GPT-C) and appeal to the naturalness of software (Allamanis et al., 2018) to construct our baseline approach for the merge resolution synthesis task.
We establish the following baseline: given an unstructured line-level conﬂict produced by diff3 , we take the common source code preﬁx Pref acting as user intent for program merge.
We attempt to generate an entire resolution region token-by-token using beam search.
As an ablation experiment, we repeat this for the conﬂicts produced with the token-level differencing algorithm (Fig. 1 shows details about preﬁx and conﬂicting regions).
7. D EEP M ERGE : N EURAL M ODEL FOR I NTERLEAVINGS
Next, we consider DeepMerge (Dinella et al., 2021): a sequence-to-sequence model based on the bidirectional GRU summarized in section 3.
It learns to generate a resolution region by choosing from line segments present in the input (line interleavings) with a pointer mechanism.
We retrain the
DeepMerge model on our TypeScript dataset.
7. JDIME
Looking for a stronger baseline, we consider JD IME , a Java-speciﬁc merge tool that automatically tunes the merging process by switching between structured and unstructured merge algorithms (Apel et al., 2012).
Structured merge is abstract syntax tree (AST) aware and leverages syntactic information to improve matching precision of conﬂicting nodes.
To compare the accuracy of JD IME to that of
MergeBERT, we use the Java test and complete the following evaluation steps: ﬁrst, we identify the set of merge conﬂict scenarios where JD IME did not report a merge conﬂict, and the standard diff algorithm did.
Second, we compare the JD IME output to the version of the code where the merge conﬂict is resolved.
Third, we calculateJD IME accuracy by identifying the number of merges where
JD IME output ﬁle correctly matches the resolved conﬂict ﬁle.
As a result of its AST matching approach, code generated by JDime is reformatted, and the original order of statements is not always preserved.
In addition, source code comments that are part of conﬂicting code chunks are not merged.
A simple syntactic comparison is too restrictive, and JDime merge output can still be semantically correct.
To accurately identify semantically equivalent merges, we use GumTree Falleri et al.
(2014), an AST differencing tool, to compute ﬁne grained edit scripts between the two merge ﬁles.
By ignoring semantically equivalent differences computed by GumTree (such as moved method declarations) we have a more accurate baseline comparison between the number of semantically equivalent merges generated by JDime and MergeBERT.
7. JS FSTM ERGE
Apel et al.
(2011) introduced FSTMerge, a semi-structured merge engine using an approach similar to JD IME , but that that allows a user to provide an annotated language grammar speciﬁcation for any language.
Trindade Tavares et al.
(2019) implemented jsFSTMerge by adapting an off-the-shelf grammar for JavaScript to address shortcomings of FSTMerge and also modifying the FSTMerge algorithm itself.
For example, statements can be intermingled with function declarations at the same syntactic level, and statement order must be preserved while function order does not.
jsFSTMerge allows for certain types of nodes to maintain their relative order ( e.g. , statements) while others may be order independent ( e.g. , function declarations) even if they share the same parent node.
For cases where jsFSTMerge produces a resolution that does not match the user resolution, we manually inspect the output for semantic equivalence (e.g., reordered import statements).
E VALUATION
We evaluate MergeBERT’s accuracy of resolution synthesis.
Our evaluation metrics are precision and recall of verbatim string match (modulo whitespaces or indentation) of the decoded predictions to the user resolution extracted from real-world merge resolutions.
This deﬁnition is rather restrictive as a predicted resolution might differ from the true user resolution by, for instance, only the order of statements, being semantically equivalent otherwise.
As such, this evaluation approach gives a lower bound of the MergeBERT model performance.
In addition to the precision and recall, we estimate the fraction of syntactically correct (or parseable) source code suggestions to ﬁlter out merge resolutions with syntax errors.
8. B ASELINE M ODEL E VALUATIONS
[Table]
As seen in Tab. [Table] performance of language model baselines on the task of merge resolution synthesis is relatively low, suggesting that the naturalness hypothesis is insufﬁcient to capture the developer intent when merging programs.
This is perhaps not surprising given the notion of precision that does not tolerate even a single token mismatch.
MergeBERT tool is based on two core components: token-level diff3 and bidirectional transformer encoder with classiﬁcation head.
As seen, the token-level differencing alone gives a high precision of 82.4%, with a relatively low recall of only 36.1%.
With the neural transformer model, the recall is increased up to a total of 68.2%.
Note, as a deterministic algorithm token-level diff3 can only provide a single suggestion.
DeepMerge precision of merge resolution synthesis is quite admirable, showing 55.0% top-1 precision.
However, it was only able to produce predictions for 63.8% of the test conﬂicts, failing to generate predictions for merge conﬂicts which are not representable as a line interleaving.
This type of merge conﬂicts comprises almost a third of the test set, resulting in a recall of 35.1% which signiﬁcantly lower than MergeBERT, and 50% lower F-score.
As an ablation, we train a DeepMerge model starting with conﬂicting regions of token granularity.
As seen in Tab.
1 overall performance increases, resulting in a 13% higher F-score, while still falling short of MergeBERT by 34% F-score (7% lower precision and 60% lower recall).
[Table]
As can be seen from Tab. 5, jsFSTMerge is only able to produce a resolution for 22.8% of conﬂicts, and the produced resolution is correct only 15.8% of the time.
This is in line with the conclusions of the creators of jsFSTMerge that semi-structured merge approaches may not be as advantageous for dynamic scripting languages (Trindade Tavares et al., 2019).
Because jsFSTMerge may produce reformatted code, we manually examined cases where a resolution was produced but did not match [Table] the user resolution (our oracle).
If the produced resolution was semantically equivalent to the user resolution, we classiﬁed it as correct.
Tab. 3 shows the detailed evaluation results of the MergeBERT.
As seen, multilingual variant of MergeBERT yields 63 .
6 − 68 .
5% top-1 and 75 .
2 − 76 .
8% top-3 precision of verbatim match and relatively high recall values.
Overall, the multilingual variant of the model generates results comparable to the monolingual versions on the languages present in the training set and shows the potential for zero-shot generalization to unseen languages.
We test the zero-shot generalization property on merge conﬂicts in Scala 1 programming language and obtain an encouraging 64.8% topprecision of merge resolution synthesis.
C ONCLUSION
This paper introduces MergeBERT, a transformer-based program merge framework that leverages token-level differencing and reformulates the task of generating the resolution sequence as a classiﬁcation task over a set of primitive merge patterns extracted from real-world merge commit data.
MergeBERT exploits pretraining over massive amounts of code and then ﬁnetuning on speciﬁc programming languages, achieving 64–69% precision and 63–68% recall of merge resolution synthesis.
Lastly, MergeBERT is ﬂexible and effective, capable of resolving more conﬂicts than the existing tools in multiple programming languages.
Our work focuses on helping software developers resolve merge conﬂicts and improve their productivity.
The ﬁnetuning approach that lies at the core of this tool promotes the re-usability of pretrained transformer models for software engineering tasks.
A PPENDIX 10. P RIMITIVE M ERGE R ESOLUTION T YPES
Given a merge tuple ( A , B , O , M ) with line-level conﬂicting regions
A i B i , O i , i=0...N, and tokenlevel conﬂicting regions a ij b ij o ij corresponding to a line-level conﬂict i , we deﬁne following nine basic merge resolution types which serve as labels for supervised classiﬁcation task:
Take changes a ij proposed in program A (developer branch A) as resolution, 2.
Take changes b ij proposed in program B as resolution, 3.
Take changes o ij in the base reference program O as resolution, 4.
Take a string concatenation of changes in a ij and b ij as resolution, 5.
Take a string concatenation of changes in b ij and a ij as resolution (reverse order as compared to the previous), 6.
Take changes a ij proposed in program A , excluding the lines also present in the base reference program O , as resolution, 7.
Take changes b ij proposed in program B , excluding the lines present in the base, as resolution, 8.
Take a string concatenation of changes in a ij and b ij , excluding the lines present in the base, as resolution, 9.
Take a string concatenation of changes in b ij and a ij , excluding the lines present in the base, as resolution (reverse order as compared to the previous),
We use a data-driven approach to identify these 9 primitive merge resolution patterns based on the analysis of the real-world merge conﬂict resolutions from GitHub.
Our analysis shows that over 99% of all the merge conﬂicts can be represented using these labels.
While the above nine resolution types are primitive, they form a basis sufﬁcient to cover a large class of real-world merge resolutions in modern version control systems, including arbitrary combinations or interleavings of lines.
Fig.
3 shows the label distribution in our dataset for TypeScript programming language.
The plot on the left shows the label distribution obtained for the standard (line-level) diff3 conﬂicting regions.
As seen, nearly 60% of all cases are trivial – take changes from branch A or B. Arguably, these cases can be resolved without machine learning and are easily addressed by take ours or take theirs merge resolution strategies.
The plot on the right shows the label distribution obtained for tokenlevel differencing algorithm.
It excludes trivial (take A or take B) merge resolutions.
Note, that “take A” merge resolution at token-level does not correspond to “take ours” or “take theirs” merge resolution strategy, and can map to any label at line-level, thus representing a non-trivial merge scenario stimulating for machine learning studies.
It is important to stress, these primitive merge resolution types are not strictly deﬁned templates dictating which syntactic structures should be selected from input programs.
For instance, a label “take changes proposed in program A “ can correspond to a single code token as well as an entire method signature or body.
As such, the merge types are not restrictive in their representation power of merge conﬂicts.
10. S UMMARY OF THE F INETUNING D ATASET
Tab. 4 shows the summary of the ﬁnetuning merge conﬂict resolution dataset.
[Table]
M ERGE R ESOLUTION D ECODING
Each model prediction yields a probability distribution p ( r j | a j , b j , o j ) over primitive merge patterns given a token-level conﬂict.
In case of a one-to-many correspondence between original line-level and the token-level conﬂicts (see, for instance, Fig. 7) to approximate the original p ( R | A, B, O )
we decode the most promising combination from the predicted solution space.
This can be conceptualized as a maximum cost path search on a matrix, which we approach via a beam search algorithm.
As a result, the model prediction for each line-level conﬂict consists of either a label for a token-level conﬂict or a combination of labels for multiple token-level conﬂicts representing the best prediction for each token-level conﬂict within the line-level conﬂict.
Given these labels for each line-level conﬂict and the contents of the merged ﬁle, MergeBERT generates the code corresponding to the resolution region.
The contents of the merged ﬁle includes the conﬂict in question and its surrounding regions.
Therefore, for each conﬂicting line, MergeBERT chooses between the versions of code based on the labels the model produced and generates the resolution code by concatenating them. Afterwards, MergeBERT checks the syntax of the generated resolved code, and in case of correctness, outputs it as the candidate merge conﬂict resolution. In case of multiple line-level conﬂicts in the merged ﬁle, MergeBERT reﬁnes the contents of the merged ﬁle that serves as the surrounding region of the conﬂict. More speciﬁcally, for each line-level b ← { r 0 , 0 } (cid:46) Initialize beam {state, logprob} { a j , b j , o j , r j } j =0 ...
N ← token_diff3 ( A , B , O , M ) , (cid:46) Perform token-level differencing for j ← 0 to N do if a j = ∅ ∧ b j = ∅ ∧
o j = ∅
then (cid:46)
In case token-level merge results in a clean merge return pref j :: r j :: suff j elsefor ( r kj , p kj )
∈
TopK ( mergebert ( r j | a j , b j , o j )) do b ← b ∪ { r + pref j :: r kj :: suff j , p + p kj } (cid:46) Update beam for each token-level conﬂict j end for b ← TopM ( b ) (cid:46)
Prune candidates to keep top-M end if end for
R ← b [0] (cid:46) Get resolution region string return R conﬂict, MergeBERT replaces the other conﬂicts in the the merged ﬁle contents with the code it previously generated as their predicted resolutions.
For this purpose, MergeBERT updates the contents of the merged ﬁle after resolving each line-level conﬂict with the code it generates as the conﬂict resolution based on the model prediction.
11. I NFERENCE C OST
Computational efﬁciency is an important constraint inﬂuencing machine learning design decisions in production environments (e.g. deployment in IDE, GitHub action).
In the following, we discuss inference costs and ﬂoating point operations per second (FLOPs) of MergeBERT as compared to the language model baseline.
In this paper, we reformulate the task of merge conﬂict region as a classiﬁcation problem.
This provides a major speedup during inference, due to a smaller number of inference calls necessary to decode a resolution.
Indeed, in most cases MergeBERT requires only 1 inference call to resolve a merge conﬂict, with up to 3 calls in the worst case, based on our dataset.
The cost of a single inference call on a 16GB Tesla V100 GPU is 60 ms.
The end-to-end time to resolve a merge conﬂict (including tokenization, alignment, and edit sequence extraction) is 105 ms on average, and up toms in the worst case.
With GPT-C language model, the resolution region is decoded token-by-token via the beam search algorithm.
The average time to decode a single token (in our experiments we use beam width of 5, and 1024 tokens context length, with past hidden state caching optimization enabled) on a 16GB
Tesla V100 GPU is about 15 ms.
With token-level differencing, the resolution size is 70 tokens on average (up to 1584 tokens maximum, in our dataset), which yields 1.1 seconds on average and up to 23.8 seconds in the worst case (the largest conﬂict) to generate resolution token sequence.
Overall, end-to-end inference time required to resolve a merge conﬂict (including parsing and tokenization) is 2.3 seconds on average and up to 48.5 seconds for the largest conﬂict.
From the user experience prospective in IDE, inference times of over 10 seconds are prohibitively slow.
11.1. F LOATING P OINT O PERATIONS PER S ECOND
In the following, we identify main operations in the transformer encoder, for the multi-input MergeBERT architecture (see Fig.
2 for reference):
• Self-attention: 600 MFLOPs x 4 inputs (encoder weights are shared for all inputs), • Feed-forward layer: 1200 MFLOPs x 4 inputs.
Contribution of the lightweight pooling (aggregation) and classiﬁcation layers are negligibly small.
With a total of 6 transformer encoder layers this yields: 43200 MFLOPs per forward pass.
For the GPT-C transformer decoder-only model we get:
• Self-attention: 600 MFLOPs • Feed-forward layer: 1200 MFLOPs
with a total of 12 encoder layers this yields: 21600 MFLOPs per inference call, and for 6 encoder layers: 10800 MFLOPs.
With larger FLOPs per a single forward pass as compared to generative approach, with MergeBERT we gain a signiﬁcant reduction in total FLOPS required to decode resolution region as a result of needing to performing orders of magnitude less calls (1–3 calls with MergeBERT as compared to 70–1584 with a language model), making this approach an appealing candidate for deployment in
IDE.
11. I MPACT OF P RETRAINING D ETAILS
The effect of self-supervised pretraining is two-fold: (1) it speeds up the time to solution as a result of faster model convergence – we observe a 20% higher F-score after 5 training epochs – and 32 times larger ﬁnetuning training throughput, and (2) it yields 14% overall higher F-score as compared to a model trained from scratch.
While experiments with various pretraining objectives (Guo et al., 2021; Rozière et al., 2021) is outside of the scope of this work, as a baseline, we utilized the CodeBERT public checkpoint for a downstream task of merge conﬂict resolution which utilizes a similar pretraining objective as ours.
The resulting model showed a comparable F-score, and a likely explanation for the difference is that the CodeBERT is pretrained on the CodeSearchNet dataset, which does not include C# and
TypeScript programming languages used in this study.
The F-scores of merge resolution synthesis are quoted for TypeScript test set as a function of epoch.
Finetuning performance with CodeBERT-base 2 publicly available checkpoint is quoted for reference
11. C ONTEXT E NCODING D ETAILS
Fig.
5 illustrates how input representations are extracted.
As an ablation, we have repeated MergeBERT model training excluding the edit type information.
As seen, adding edit type embedding information improves the F-score by 7%.
11. I MPLEMENTATION D ETAILS
We pretrain a BERT model with 6 encoder layers, 12 attention heads, and a hidden state size of 768.
The vocabulary is constructed using byte-pair encoding method (Sennrich et al., 2016) and x
S x = max ( y , )
S = S max S (
S y
S ,
S
S )
S S
S S
S S
S S
S = S =
S = S =
S = S =
S ßà
S = a| o
Token
Position
Edit x
S x = max ( y , )
S = S max S (
S y
S ,
S
S )
S S
S S
S S
S S
S = S =
S = S =
S = S =
S ßà
S = o| a
Token
Position
Edit
[Table]
the vocabulary size is 50000.
We set the maximum sequence length to 512.
Input sequences cover conﬂicting regions and surrounding code (i.e., fragments of Pref and Suff ) up to a maximum length of 512 BPE tokens.
The backbone of our implementation is HuggingFace’s RobertaModel and RobertaForSequenceClassification classes in PyTorch, which are modiﬁed to turn the model into a multi-input architecture shown in Fig.
We train the model with Adam stochastic optimizer with weight decay ﬁx using a learning rate of 5e-5, 512 batch size and 8 backward passes per allreduce on 64M ﬁles in C, C++, C#, Python, Java,
JavaScript, TypeScript, PHP, Go and Ruby programming languages.
The training was performed on 64 NVIDIA Tesla V100 with 32GB memory for 21 days; we utilized mixed precision.
Finetuning was performed on 4 NVIDIA Tesla V100 GPUs with 16GB memory for 6 hours.
In the inference phase, the model prediction for each line-level conﬂict consists of one or more tokenlevel predictions.
Given the token-level predictions and the contents of the merged ﬁle, MergeBERT generates the code corresponding to the resolution region.
The contents of the merged ﬁle include the conﬂict in question and its surrounding regions.
Afterward, MergeBERT checks the syntax of the generated code with tree-sitter 3 parser and outputs it as the candidate merge conﬂict resolution only in case of correctness.
The model prediction for each line-level conﬂict consists of either a label for a token-level conﬂict or a combination of labels for multiple token-level conﬂicts representing the best prediction for each token-level conﬂict within the line-level conﬂict.
Given these labels for each line-level conﬂict and the contents of the merged ﬁle, MergeBERT generates the code corresponding to the resolution region.
The contents of the merged ﬁle includes the conﬂict in question and its surrounding regions.
Therefore, MergeBERT, for each conﬂicting line, chooses between the versions of code based on the labels the model produced and generates the resolution code by concatenating them.
Afterwards,
MergeBERT checks the syntax of the generated resolved code, and in case of correctness, outputs it as the candidate merge conﬂict resolution.
In case of multiple line-level conﬂicts in the merged ﬁle, MergeBERT reﬁnes the contents of the merged ﬁle that serves as the surrounding region of the conﬂict.
More speciﬁcally, MergeBERT replaces the other conﬂicts in the the merged ﬁle contents with the code it previously generated as their predicted resolutions.
For this purpose, as shown in Fig. 6, MergeBERT updates the contents of the merged ﬁle after resolving each line-level conﬂict with the code it generates as the conﬂict resolution based on the model prediction. [Figure] [Figure]
11. M ULTIPLE C ONFLICTING R EGIONS
MergeBERT can deal with non-trivial real-world merges, composed of multiple conﬂicting chunks.
To provide an example of such a merge conﬂict, we include Fig. 7.
MergeBERT correctly predicts a concatenation of changes proposed by developers A and B for the ﬁrst token-level chunk and a concatenation of changes proposed by developers B and A (in the reverse order) for the second chunk
[Figure]
[Figure]