From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper presents a method for repairing vulnerable source code that is based on an encoder-decoder Transformer architecture, but additional supervision is provided indicating if each input token appears inside a "vulnerable statement" (one that needs repairing). The learned masks are then used to modify the attention operations, with the intuition being that these masks focus attention in a similar way as object queries are used in Vision Transformers. While the authors presented a number of helpful additional experiments in the rebuttal, there remain concerns about the clarity of the contributions. Reviewers are unconvinced that the framing of vulnerability queries is directly analogous to object queries, and the significance of the architectural contribution remains somewhat unclear, as the gains are models and only in one experimental setting, and there is some mixing in several comparisons of contributions from using the pretraining set of Chen et al. with the architectural modifications. To match the narrative being presented, I'd suggest the authors focus on demonstrating the robustness of improvements from just the architectural component, ideally holding fixed the pretraining data and demonstrating improvements across more kinds of repair tasks.
justification_for_why_not_higher_score: Remaining questions about robustness of improvements and clarity of contributions.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper5884/Reviewer_VFto
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims to improve existing automated vulnerability repair problem by considering the prediction of precise parts of vulnerable code. By making an analogy to object detection problem in computer vision, they introduce the concept of vulnerability query counter to object query and adopt cross-attention mechanism to pay attention to vulnerable code areas. Besides, they also propose to learn an extra model to predict vulnerable areas, the prediction is used to construct a vulnerability query mask and further incorporated into both encoder and decoder’s self-attention. Evaluation is conducted on a read-world dataset which achieves outperforming results. 
strength_and_weaknesses: Strength: 
(1) The research problem is important among code repair community. 
(2) The idea of incorporating predicting vulnerable areas into code repair is interesting and the intuition makes sense.


Weakness: 
(1) The concept of vulnerability query is somewhat weird as the authors cannot fully elaborate what it exactly is, especially when making analogy to object query. The vulnerability query is designed and utilized in the decoder part, how it functions is also not clearly described.
(2) Some important baselines are missing and more metrics should be reported. There are a handful of works such as [1,2] which also use seq2seq models to predict program repair but these baselines are missing in experiments. Sequence based metrics should also be reported like BLEU, METEOR as they are widely used in previous code repair works.
(3) The ablation study is not sufficient. While the authors highlight the importance of proposed vulnerability query, there is no ablation study to demonstrate the effectiveness of it.

[1] Li, Yi, Shaohua Wang, and Tien N. Nguyen. "Dlfix: Context-based code transformation learning for automated program repair." Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 2020.
[2] Jiang, Nan, Thibaud Lutellier, and Lin Tan. "Cure: Code-aware neural machine translation for automatic program repair." 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021.

clarity,_quality,_novelty_and_reproducibility: Quality: Weak. The effectiveness of the design is not well demonstrated.
Clarity: Good. The writing of the paper is good and clearly shows the proposed method.
Originality: Good. The work sheds some light on the solution to code repair.

summary_of_the_review: This work presents the key idea of providing code repair model with the precise location of vulnerable code and proposes two techniques: vulnerability query cross-match and vulnerability query mask. But the rationale and effectiveness of vulnerability query is not well explained and proved. And important baselines are also missing during the experimental evaluation. We believe that this paper can be further improved to be solid work.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5884/Reviewer_Vf8F
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims at improving the transformer based vulnerable code repair techniques. Inspired by vision transformer based object detection approach, it proposes to incorporate a learnable  ‘vulnerability query mask’ into both encoder output and the cross-attention of existing transformer based models. Experimental results show the proposed approach provides an increased percentage of perfect predictions compared to state-of-the art approaches.
strength_and_weaknesses: Strength

- Code repair is a challenging task and improving the accuracy of the transformer based models is a reasonable direction.
- Ablation studies are performed to demonstrate the effectiveness of added vulnerability query mask.

Weakness

-  The improvement over the existing transformer based models proposed in this work has limited novelty. Proposed vulnerability is a direct mapping of the vision transformer setup.

- The evaluation results do not discuss types of errors used for training and evaluation. It is also not clear if the base models using text-to-text transformers (e.g., TFix) are fine tuned with the same training dataset. What implementation of CodeBERT is used for the evaluation of vulnerability repair?



clarity,_quality,_novelty_and_reproducibility: Clarity: Overall, the paper is easy to follow. 

Novelty: The novelty of this work is incorporating vulnerability query mask that could potentially improve existing transformer based code repair models.

Reproducibility: Model configurations are shared to enable reproducibility of the proposed work.

summary_of_the_review: The proposed approach does not demonstrate significant novelty. Also, the effectiveness of the proposed approach should be investigated more elaborately by examining the error types used for training and evaluation.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5884/Reviewer_urCv
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a transformer model for generating repair for software vulnerabilities. It trains a separate token-level classifier model to generate masks called vulnerability queries and incorporates these masks in both the encoder and decoder parts. Through experimental evaluation on Big-Vul and CVEFixes, it shows that this helps the model obtain better repairs compared to baseline models.
strength_and_weaknesses: Strengths
-----
* The problem of fixing software vulnerabilities is important in practice and paper performs evaluation on two of the most common vulnerability datasets against relevant baselines.
* It achieves a few percentage improvement across different beam sizes.

Weaknesses
----
* The modeling contribution of the paper is minimal. It incorporates a simple masking strategy that labels vulnerable code tokens and uses the masks in the self/cross attention computations.
* The test set is split arbitrarily into 80-20 train+validation and test. The test set is small enough (5.4K samples) that a five fold experiment can be conducted. Can you report results on the entire set of examples similar to Table 2?
* The training setup and accuracy of the vulnerability query mask prediction model are not reported.

Clarifications
--------
* If you are generating token-wise labels as query masks then why and how do you use global pooling (see the sentence before Eq 2)?
* Which layers do you apply the query marks to in the encoder and the decoder?
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly (except for some points above). The authors have given a link their code and models so it should be possible to reproduce the results. The paper makes a simple modification to the transformer architecture and has minimal modeling contribution.
summary_of_the_review: The paper addresses a specific software engineering problem and makes a simple modification to transformers to encode domain knowledge. On one split of the test data, it gets a few percentage point improvement over baselines.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5884/Reviewer_oYMK
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper applies a Vision Tranformer (VIT)-based approach to automated
vulnerability repair. The method is called Vulnerability Query based Software
Vulnerability Repair (VQR). The authors introduce vulnerability queries similar
to the object queries of VIT and cross-match them with vulnerable code areas
through a cross-attention mechanism to generate more accurate repairs. They also
learn a vulnerability query mask to focus on vulnerable code areas and include
it into the cross-attention, and also into the self-attention in the encoder to
learn embeddings that emphasize the vulnerable areas of the program. Their
method outperforms existing state-of-the-art methods based on a real-world dataset
of 5417 vulnerabilities.

strength_and_weaknesses: ### Strengths

The paper is based on a good idea, is very well written, and presents convincing
results with many comparisons to state-of-the-art and an ablation study.

### Weaknesses and Questions

My main concern is that I'm not sure about the analogy between VIT and VQR,
particularly between object queries and vulnerability queries. Object queries
are used by the model to predict objects in a given area of the image, are
learned throughout the training process (as an embedding), and are the same for
each image (apart from being updated by learning). In contrast, it seems to me
based on 3.2.2 that vulnerability queries are just inputs to the decoder. This
is reinforced by their initialization: they are initialized by the repair tokens,
which are different for each example (they are the targets).

If I'm not mistaken then some of the parallels between VIT and VQR and some of the
arguments in the paper do not hold, and the state-of-the-art results are
obtained probably because of the architecture from VIT and not the queries,
which makes it necessary to rewrite significant parts of the paper, including
the title.

I believe that a minus sign is missing from the cross-entropy loss (2). Also,
maybe it would be clearer to explicitly state that this is a cross-entropy loss.

$x_i = [t_1, \dots, t_n]$ is sometimes referred to as a sequence of tokens, and
sometimes as a sequence of token embeddings.

It could be helpful to expand the VIT abbreviation in the paper.

There are some small typos:
- in the Abstract, "the real-world 5417 vulnerabilities"
- at the start of 3.1, "Assuming"


clarity,_quality,_novelty_and_reproducibility: ### Clarity

The paper is very well written.

### Quality

The paper is of very high quality.

### Novelty

The main idea which is an application of Vision Transformers to code
vulnerability detection is novel.

### Reproducibility

The complete source code is provided.

summary_of_the_review: I really liked the paper, but I have a serious concern about the vulnerability
queries which are at the heart of it. If I'm mistaken regarding this issue it
probably needs to be made clearer in the paper.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold