Paper: Coeditor: Leveraging Repo-level Diffs for Code Auto-editing (ICLR-2024)

Reviewer comment (Experiments):
There can be many reference blocks, spanning up to 16.4K tokens, so the work uses a relatively short "Query" of up to 1,024 tokens. Other LMs, such as InCoder and StarCoder, all have much larger context sizes. It is not clear from the writing whether the other models were "allowed" to use their full context window, or were provided with the same limited section of context. If a completion took place halfway through an 8K token file, one would expect StarCoder, for instance, to do substantially better with the full file context than with a 1K token window.

Author response:
__Q1: Were the baseline LMs evaluated with a 1K context, same as Coeditor, or with their maximum number of tokens (when possible)?__

We let all baseline LMs utilize their full context by including as much surrounding code as possible. The context size of each model is detailed in Table 3.

Reviewer comment (Experiments):
A seemingly unrelated form of "reference" comes from the "signatures" (section 3.2). This idea does not have anything to do with code editing itself, but rather with providing an LM access to the project-level context. This seems to be very impactful (Tab. 5). It looks like the baseline LMs were not provided with this information, which seems like an oversight. For one, there are reasonable ways to prompt pretrained language models with such information even if that wasn't present at pretraining time. For another, the performance of this component has little, if anything, to do with the stated goal of the work (leveraging diffs for edit prediction). The fact that it boosts performance so strongly makes the results much less interesting; it suggests that a large component of the performance does not come from the presence of diffs, but from a more banal form of information that related work has already explored. The paper should either include carefully calibrated experiments with baselines using the same information or exclude this from its main contributions in results such as Tab. 3.

Author response:
__Q2: Was signature information only used in Coeditor? If so, is this intended to be a core part of the contribution? Please provide ablations with other models using this information, and with Coeditor fully trained without this information in any case.__

We would first like to clarify that all our ablation results were obtained by retraining Coeditor with the corresponding information removed at training time (as mentioned at the beginning of Section 4.3.)

The signature information (based on usage-analysis) is only used by Coeditor in our evaluation and is part of the contribution of this work. While prior work has explored the use of signatures in tasks like code completion [1] and type inference [2], their value in code editing has not been studied before. In fact, our initial hypothesis was that signatures might be less critical in editing tasks, given that the model has access to the old version of the code, which can aid in inferring signatures from existing usages.

However, our ablation studies (Table 5) uncovered a significant finding: both recent editing history and signature information are crucial for editing performance, and neither alone is sufficient. In particular, the first row of Table 5 shows that not having editing history (while still keeping the signatures and a 16K context size) leads to the largest drop in performance, reducing overall EM from 42.1% to 26.1%. This result counters the statement that “a large component of the performance does not come from the presence of diffs, but from a more banal form of information that related work has already explored”. 

To understand why the combination of signatures and recent editing history is so useful for this task, we manually inspected examples where Coeditor was successful and found that the editing history often clarifies the desired types of changes, while the signature information aids in identifying where similar changes should be applied. For example, if the model sees that the user has added a statement `obj.some_attribute *= 2`  and needs to predict a similar change `obj.another_attribute *= 2` ,  it often requires seeing what other attributes are defined in `obj`’s class signature in order for the model to predict which other attributes need to be similarly mutated.

[1] Toufique Ahmed et al. "Improving Few-Shot Prompts with Relevant Static Analysis Products." arXiv preprint arXiv:2304.06815 (2023).

[2] Jiayi Wei et al. "TypeT5: Seq2seq Type Inference using Static Analysis." The Eleventh International Conference on Learning Representations. 2023.

Reviewer comment (Evaluation):
Relatedly, it is not clear why there is such a large performance gap between Tab. 5 and Tab. 3. The former uses the validation set, so one might expect that the latter is based on test data. The text in 4.3 does mention that the model was only halfway trained here. But the gap between 42.1% and 60.4% is really very large. Is this not the same type of task as in Tab. 3? And how should we expect the gaps between ablations to scale to the numbers of Tab. 3? In particular, would the impact of the retrieval blocks (criticized above) be proportional or even disproportionally larger?

Author response:
__Q: I would still appreciate clarification on why Tab. 5 shows a best-case EM rate of 42.1% while Tab. 3 shows a much higher value of 60.4%.__

This is because these two tables show results on different tasks. Tab. 3 shows the EM rate on the PYCOMMITS-ONELINE dataset, which is a code completion task involving only one missing line. Tab. 5 shows the single-round EM rate on PYCOMMITS, which is an editing task requiring identifying and performing multiple missing changes. Because the latter is a fundamentally harder problem, the EM rate is lower.

Reviewer comment (Experiments):
Were the baseline LMs evaluated with a 1K context, same as Coeditor, or with their maximum number of tokens (when possible)?

Author response:
__Q1: Were the baseline LMs evaluated with a 1K context, same as Coeditor, or with their maximum number of tokens (when possible)?__

We let all baseline LMs utilize their full context by including as much surrounding code as possible. The context size of each model is detailed in Table 3.

Reviewer comment (Experiments):
The edit granularity choice and dataset can introduce bias: developers often commit many intermediate commits that get squashed into a single commit before pushing and changes can get overwritten.

Author response:
This is a good point: in practice, commits in a pull request are often squashed before merging, resulting in training commits that are larger than what users typically encounter at editing time. There are several potential remedies to this issue. For example, at inference time, we could extend our approach to include editing history beyond the last commit, in order to better match the content of a squashed commit. Future work can also explore using LLMs to identify squashed commits and decompose them into smaller, semantically independent units. Notably, however, many squashed commits collected from open source repos can still be relatively small and contain closely related changes. Therefore, training our model on a substantial volume of such commits may already provide a robust training signal that is sufficient for teaching the model to utilize editing histories.

Reviewer comment (Experiments):
With the move towards an extension, was using more granular edit events considered, what about navigation events or other similar side-channel information? Is the main concern the context size or are there other trade-offs that moved the authors closer to git diff, line-level information?

Author response:
Our choice of using commit history is mainly motivated by data availability. Acquiring detailed user edit events entails installing specialized tools for data collection, posing scalability and privacy challenges that are beyond the scope of this work. A promising future direction, however, could involve a hybrid approach that combines large-scale pre-training (using the commit-based data proposed by this work) followed by fine-tuning with a smaller dataset of detailed edit events.

Reviewer comment (Experiments):
I would be curious to see how CodePlan + Coeditor perform on the PyCommits dataset (especially since the CodePlan paper partially features the results using the Coeditor model), although I realized that CodePlan was published after the ICLR submission deadline. Is it possible to add?

Author response:
Regarding combining CodePlan with Coeditor for the PyCommits dataset, it's important to note CodePlan and Coditor’s different focuses. Coeditor, along with the PyCommits dataset, focuses on making precise changes within specific code regions. CodePlan focuses on orchestrating large-scope changes across multiple files, based on some clear specification and utilizing a local editing model similar to Coeditor. This difference in their operational scopes means PyCommits would not be an ideal evaluation for CodePlan + Coeditor, as CodePlan’s planning strength is not needed for PyCommits' localized edits.

Reviewer comment (Experiments):
Another interesting baseline could be BluePencil model focusing on repetitive edits (see "On the fly synthesis of edit suggestions" by Miltner et. al) this is also available through IntelliCode Suggestions VS extension.

Author response:
BluePencil and Coeditor share a common goal of predicting user edits based on recent user changes. However, BluePencil uses rule-based program synthesis techniques and is limited to suggesting repetitive changes. In contrast, Coeditor has the capability to produce novel changes beyond BluePencil's scope. Furthermore, BluePencil's implementation is specific to C# programs, while Coeditor has been trained exclusively for Python. This difference in target programming language precludes a direct comparison of the two tools.

Reviewer comment (Evaluation):
The primary issue of the paper is over claim when it compares with SOTA code infilling models. Statements such as "our method achieves 60.4% exact match accuracy using a 220M parameter model, whereas text-davinci-003—the best performing code infilling model we have compared, which has 175 billion parameters—achieves only 30.5%." is extremely misleading.

Author response:
Our comparison with infill models is detailed in Section 4. We extracted single-line code completion problems from real-world edits and fed the prefix and suffix of each problem to the infill model, assessing whether it can accurately output the correct middle span. We discuss other prompting strategies below under Q3.

Note that the single-round editing evaluation has a structure that is designed to be fair to infilling models as follows: if the ground truth changes in a code unit $u$ contain multiple changed lines, we only select the last changed line as the code completion target and remove it from $u$, and we apply the remaining code changes to $u$. This creates a normal completion problem such that the resulting code (after adding the missing line) is the after-commit version of $u$. 
For example, if the ground truth is
```
-def sort_by_name(persons):
+def sort_by_name(persons, reversed=False):
     persons.sort(key=lambda p: p.name)
+    if reversed:
+        persons.reverse()
     return persons
```
We create the corresponding code completion problem as
```
def sort_by_name(persons, reversed=False):
    persons.sort(key=lambda p: p.name)
    if reversed:
        <target> persons.reverse() </target>
    return persons

```
This is similar to the last round in a multi-round editing setting, arguably the *easiest part* of a multi-round editing loop for the baseline models, which were not pre-trained to handle a partially edited context.

Reviewer comment (Reproducibility):
The paper didn't explain clearly how the generic infill models are prompted to solve the code editing tasks.

Author response:
Given that the single-line code completion task is closely related to the original infilling task, we did not employ few-shot prompting for the text-davinci-003 model. Instead, we utilized the maximum available context size (4K tokens) to store as much surrounding code as feasible.  The reported performance of text-davinci-003 was obtained by directly calling OpenAI’s infilling API, which expects a prefix and a suffix.

We did experiment with different prompt formats on GPT-3.5-turbo, as described below (in the next comment).

Reviewer comment (Experiments):
There is no way we can fairly compare the proposed approach with the baseline models compared in this work.

Author response:
Our comparison with infill models is detailed in Section 4. We extracted single-line code completion problems from real-world edits and fed the prefix and suffix of each problem to the infill model, assessing whether it can accurately output the correct middle span. We discuss other prompting strategies below under Q3.

Note that the single-round editing evaluation has a structure that is designed to be fair to infilling models as follows: if the ground truth changes in a code unit $u$ contain multiple changed lines, we only select the last changed line as the code completion target and remove it from $u$, and we apply the remaining code changes to $u$. This creates a normal completion problem such that the resulting code (after adding the missing line) is the after-commit version of $u$. 
For example, if the ground truth is
```
-def sort_by_name(persons):
+def sort_by_name(persons, reversed=False):
     persons.sort(key=lambda p: p.name)
+    if reversed:
+        persons.reverse()
     return persons
```
We create the corresponding code completion problem as
```
def sort_by_name(persons, reversed=False):
    persons.sort(key=lambda p: p.name)
    if reversed:
        <target> persons.reverse() </target>
    return persons

```
This is similar to the last round in a multi-round editing setting, arguably the *easiest part* of a multi-round editing loop for the baseline models, which were not pre-trained to handle a partially edited context.

Reviewer comment (Experiments):
Why Coeditor is trained with a fixed batch size of 1? The Nvidia GPU used to train the model has 48GB memory which should be good to accommodate batch size > 1 with a 220M param model, right?

Author response:
In our training setup, we utilize about 20% of the available GPU memory, achieving roughly 70% average GPU utilization. While a larger batch size is feasible, the block-sparse attention pattern (which varies for each training example) coupled with the diverse context sizes of different editing problems (ranging from no reference blocks to 15K reference block tokens), limited the benefits of larger batch sizes for our task.

Reviewer comment (Experiments):
Does the PyCommits dataset composed of Github projects that have permissive licenses?

Author response:
Yes, as stated in Section 3.4's last paragraph, we constructed the PyCommits dataset from the commit history of 1,650 Python projects with permissive licenses (MIT, Apache, BSD) sourced from GitHub.

Reviewer comment (Experiments):
How the generic infill models are prompted to solve the code editing tasks? Is few-shot prompting used for the models? For example, [1] used demonstration augmented prompting for Codex model (when finetuning is not possible).

Author response:
Given that the single-line code completion task is closely related to the original infilling task, we did not employ few-shot prompting for the text-davinci-003 model. Instead, we utilized the maximum available context size (4K tokens) to store as much surrounding code as feasible.  The reported performance of text-davinci-003 was obtained by directly calling OpenAI’s infilling API, which expects a prefix and a suffix.

We did experiment with different prompt formats on GPT-3.5-turbo, as described below (in the next comment).

Reviewer comment (Experiments):
Can we use instruction finetuned version of the code generation models for code editing task?

Author response:
While there is recent work focusing on instruction-based editing, our approach focuses on a markedly different setting that doesn't require user instructions and is more similar to traditional code completion tools. Combining explicit developer instructions with editing history into our model could be an interesting direction for future work.

Reviewer comment (Experiments):
Instead of a seq2seq model, can we use a decoder-only LM for the editing task?

Author response:
A decoder-only LM could potentially be adapted for the task presented in this work. However, some specific designs, like how T5’s mask tokens are used to model editing as masked span infilling, would likely require some adaptation for decoder-only models to work well on this task.

Reviewer comment (Experiments):
Is it possible to evaluate Coeditor on the PIE benchmark [1]?

Author response:
The focus of this work is predicting probable code edits based on recent programmer changes made elsewhere, whereas the PIE benchmark focuses exclusively on performance-improving program rewrites without conditioning. Therefore, the PyCommits dataset differs fundamentally in its objectives from the PIE dataset.