Paper: Exploiting Code Symmetries for Learning Program Semantics (ICLR-2024)

Reviewer comment (Reproducibility):
The one part of the paper that could be more clear is in the precise discussion of the implementation of SymC in a Transformer. Specifically, it is unclear what the relationship between this distance matrix and the standard use of positional encodings is.

Author response:
SymC does not use absolute positional embeddings fed to the input sequences. Instead, it follows a similar setting as relative positional embedding, which biases the attention computation, i.e., the product of query and key, with the program structure. Unlike regular relative positional embedding, where the distance between the pair of nodes is still computed based on their relative distance in the linear sequence, e.g., the distance between token 2 and token 5 is (5-2=3), we compute the relative distance based on the customized distance matrix using PDG, and we prove the self-attention biased by this distance matrix preserves the automorphism of PDG, thus the semantics-preserving code permutations.

Reviewer comment (Evaluation):
I would appreciate some discussion of why the F1/AUC results in Table 2 are not monotonic in the percentage of semantics-preserving permutations.

Author response:
We do observe more statements involved in permutations do not necessarily imply more incorrect predictions. One possible explanation is that permutations involving a higher percentage of statements could represent more extensive changes in the code. Such changes might more closely resemble variations found in real-world code distributions. In contrast, single permutations, which only involve altering a single pair of statements, may be less representative of typical real-world scenarios. It is thus hard to gauge the correlation between model performance and permutation percentage.

Reviewer comment (Evaluation):
The evaluation also lacks any quantification of variance in the results (e.g., standard error across different training trials).

Author response:
We will update the paper by including variance. In our new results, see our responses above to Reviewer Srm2, we have included the variance by running the experiments multiple times with different random seeds.

Reviewer comment (Reproducibility):
Does SymC use positional embeddings?

Author response:
SymC does not use absolute positional embeddings fed to the input sequences. Instead, it follows a similar setting as relative positional embedding, which biases the attention computation, i.e., the product of query and key, with the program structure. Unlike regular relative positional embedding, where the distance between the pair of nodes is still computed based on their relative distance in the linear sequence, e.g., the distance between token 2 and token 5 is (5-2=3), we compute the relative distance based on the customized distance matrix using PDG, and we prove the self-attention biased by this distance matrix preserves the automorphism of PDG, thus the semantics-preserving code permutations.

Reviewer comment (Evaluation):
Why are the F1/AUC results in Table 2 not monotonic in the percentage of semantics-preserving permutations?

Author response:
We do observe more statements involved in permutations do not necessarily imply more incorrect predictions. One possible explanation is that permutations involving a higher percentage of statements could represent more extensive changes in the code. Such changes might more closely resemble variations found in real-world code distributions. In contrast, single permutations, which only involve altering a single pair of statements, may be less representative of typical real-world scenarios. It is thus hard to gauge the correlation between model performance and permutation percentage.

Reviewer comment (Experiments):
Could the authors provide examples of code where the relaxation of the interpretation graph to the program dependence graph is too conservative?

Author response:
As described in our response to Reviewer 6HuJ, consider the case in binary analysis:

```asm
…
1       mov rcx, [rbx]
2       mov [rax], rdx
…
```
We have 2 nodes in PDG, i.e., {1,2} denoting 2 instructions, and the edge {1->2} because we can not easily eliminate the possibility that `[rbx]` and `[rax]` would point to the same memory location. This can lead to the spurious edge, rendering the permutation of 1 and 2 an invalid automorphism. In memory-bound applications, memory-accessing instructions can be very frequent. For these programs, the size of the automorphism group can indeed be very small (not many valid semantics-preserving permutations), restricting the number of possible permutations.

Reviewer comment (Novelty):
The paper could benefit from a more comprehensive comparison with related works, such as DOBF (https://arxiv.org/abs/2102.07492), which exploits code symmetry in pretraining through a deobfuscation objective, and CodeT5 (https://arxiv.org/abs/2109.00859), which leverages code symmetry in pretraining with identifier-aware data augmentation. These related works were not discussed or compared to the proposed method in the paper.

Author response:
**Q1 & Weakness 1: The paper could benefit from a more comprehensive comparison with related works, such as DOBF (https://arxiv.org/abs/2102.07492), which exploits code symmetry in pretraining through a deobfuscation objective, and CodeT5 (https://arxiv.org/abs/2109.00859), which leverages code symmetry in pretraining with identifier-aware data augmentation. These related works were not discussed or compared to the proposed method in the paper.**

Thanks for introducing the related works to us. As shown in our response to Reviewer nM8R, Q2, we include the new results comparing SymC to DOBF, CodeT5, and GraphCodeBERT. These baselines are pre-trained, while SymC is not. Therefore, to enable a more fair comparison, we consider both pre-trained and non-pre-trained baselines and also a pre-trained version of SymC (similar to the experiment setup in Section 6.3, Figure 4(b)). In particular, the pre-trained version of SymC was pre-trained using masked language modeling on the functions provided by code2seq (java-small in the paper). 

Not pre-trained:

|               | Original Sample (F1) | Permuted Samples (F1) | Invariance Violation Rate (the lower the better) |
|---------------|-----------------|-------------------------|----------------------|
| SymC          | 36.3            | 36.3                    | 0%                   |
| GraphCodeBERT | 20.83           | 20.62                   | 31%                  |
| DOBF          | 16.34           | 20.07                   | 41%                  |
| CodeT5        | 25.35           | 25.42                   | 1%                |


Pre-trained: 

|               | Original Sample (F1) | Permuted Samples (F1) | Invariance Violation Rate (the lower the better) |
|---------------|-----------------|-------------------------|----------------------|
| SymC          | 37.1            | 37.1                    | 0%                   |
| GraphCodeBERT | 28.65           | 30.27                   | 13%                  |
| DOBF          | 34.64           | 33.71                   | 16%                  |
| CodeT5        | 41.06           | 41.73                   | 10.7%                |


The results in the above tables show that, without pre-training, SymC outperforms the second-best model, CodeT5, by 43.2% and 42.8%, in original and permuted test samples, respectively. The baselines have 24.3% violation rate on average when the samples are permuted, while SymC enjoys provable invariance by construction. In addition, we observe that pre-trained CodeT5 has 10.7x higher violation rate than that when it is not pre-trained. This is likely due to that the model learns spurious positional biases not robust to statement permutations during pre-training.

We will add more discussion to the related work, and complete Table 2.

Reviewer comment (Experiments):
The evaluation framework relies heavily on four artificial tasks created by the authors, omitting well-established, practical benchmarks used commonly in the field. For instance, important code generation tasks like OpenAI HumanEval (https://huggingface.co/datasets/openai_humaneval) and MBPP (https://huggingface.co/datasets/mbpp), as well as code translation, clone detection, defect detection, and code repair tasks from CodeXGLUE (https://github.com/microsoft/CodeXGLUE), are all relevant to the domain but were not considered. This absence of evaluation on existing benchmarks and comparison with related works raises questions about the paper's soundness and the model's real-world applicability.

Author response:
**Q2 & Weakness 2: The evaluation framework relies heavily on four artificial tasks created by the authors, omitting well-established, practical benchmarks used commonly in the field. For instance, important code generation tasks like OpenAI HumanEval (https://huggingface.co/datasets/openai_humaneval) and MBPP (https://huggingface.co/datasets/mbpp), as well as code translation, clone detection, defect detection, and code repair tasks from CodeXGLUE (https://github.com/microsoft/CodeXGLUE), are all relevant to the domain but were not considered. This absence of evaluation on existing benchmarks and comparison with related works raises questions about the paper's soundness and the model's real-world applicability.**

* Regarding “artificial tasks”

Our analysis tasks are classic program analysis tasks, and they are all adopted in practice. For example, binary code similarity detection is used for detecting vulnerabilities [1] and patching them [2], function signature prediction is used to retrofit control flow integrity [3, 4], and memory region prediction is used to facilitate malware analysis and vulnerability detection [5, 6]. We followed the baselines by using the standard datasets and experimental setup.

[1] Pewny, Jannik, Behrad Garmany, Robert Gawlik, Christian Rossow, and Thorsten Holz. "Cross-architecture bug search in binary executables." In 2015 IEEE Symposium on Security and Privacy, pp. 709-724. IEEE, 2015.

[2] Duan, Yue, Xuezixiang Li, Jinghan Wang, and Heng Yin. "Deepbindiff: Learning program-wide code representations for binary diffing." In Network and distributed system security symposium. 2020.

[3] Van Der Veen, Victor, Enes Göktas, Moritz Contag, Andre Pawoloski, Xi Chen, Sanjay Rawat, Herbert Bos, Thorsten Holz, Elias Athanasopoulos, and Cristiano Giuffrida. "A tough call: Mitigating advanced code-reuse attacks at the binary level." In 2016 IEEE Symposium on Security and Privacy (SP), pp. 934-953. IEEE, 2016.

[4] Ge, Xinyang, Nirupama Talele, Mathias Payer, and Trent Jaeger. "Fine-grained control-flow integrity for kernel software." In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 179-194. IEEE, 2016.

[5] Balakrishnan, Gogul, Thomas Reps, David Melski, and Tim Teitelbaum. "Wysinwyx: What you see is not what you execute." In Working Conference on Verified Software: Theories, Tools, and Experiments, pp. 202-213. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005.

[6] Guo, Wenbo, Dongliang Mu, Xinyu Xing, Min Du, and Dawn Song. "{DEEPVSA}: Facilitating Value-set Analysis with Deep Learning for Postmortem Program Analysis." In 28th USENIX Security Symposium (USENIX Security 19), pp. 1787-1804. 2019.

Reviewer comment (Reproducibility):
The paper does not discuss potential limitations of the SymC model, such as its requirement for input code to be processed by a parser and a static analysis tool. This requirement may limit the model's applicability when dealing with incomplete or syntactically incorrect code, such as in code completion tasks or when faced with an empty Python block. While it is acceptable to establish certain assumptions for input code, these assumptions must be explicitly discussed rather than overlooked.

Author response:
**Weakness 3: The paper does not discuss potential limitations of the SymC model, such as its requirement for input code to be processed by a parser and a static analysis tool. This requirement may limit the model's applicability when dealing with incomplete or syntactically incorrect code, such as in code completion tasks or when faced with an empty Python block. While it is acceptable to establish certain assumptions for input code, these assumptions must be explicitly discussed rather than overlooked.**

As shown in our response to Reviewer 6HuJ, statically analyzing the input code incurs additional overhead, e.g., 250.2ms per sample, but still much less than the existing model. That being said, we agree that efficiently analyzing incomplete or syntactically incorrect code on the fly can be an interesting future direction. We will add the discussion of limitations in the paper.

Reviewer comment (Experiments):
Can the authors provide a comparative analysis of SymC with related works such as DOBF and CodeT5 that also leverage code symmetry?

Author response:
* New results

However, we agree with the reviewer that there are many ML4code tasks, like defect prediction, that can benefit from code symmetry. To demonstrate that effect, we follow the reviewer’s suggestion by incorporating the defect prediction and include our preliminary results in the following. 

Particularly, we consider defects4j dataset [7] and follow CodeXGLUE to formulate the task as a binary classification (buggy/non-buggy) given the function. We ensure the number of buggy and non-buggy samples are the same, and measure the prediction accuracy on the original and permuted samples. Our baselines include GraphCodeBERT [8], DOBF [9], CodeT5 [10], UnixCoder [11], and CodeBert [12].

|  | Original Sample | Permuted Sample |
|---|---|---|
| SymC | 68.8$\pm$1.6 | 68.8$\pm$1.6 |
| GraphCodeBERT | 61.7$\pm$1.7 | 61.7$\pm$2.2 |
| DOBF | 62.4$\pm$1.03 | 61.5$\pm$1.2 |
| CodeT5 | 63.3$\pm$2.1 | 60$\pm$7.5 |
| UnixCoder | 67.1$\pm$2.2  | 67.1$\pm$1.6 |
| CodeBERT | 62.2$\pm$1.2 | 61.7$\pm$2.1 |
| GraphCodeBERT-not-pre-trained | 60.6$\pm$2.4 | 61$\pm$1.7 |
| DOBF-not-pre-trained | 59.2$\pm$1  | 59.9$\pm$1.4 |
| CodeT5-not-pre-trained | 59.2$\pm$2.7 | 59.2$\pm$2.7 |
| UnixCoder-not-pre-trained | 63.1$\pm$2.6 | 63.1$\pm$2.6 |
| CodeBERT-not-pre-trained | 62.2$\pm$4.2 | 62.4$\pm$4.6 |

The above table shows that SymC outperforms all the baselines, even when they are pre-trained on large-scale code datasets, by 11.4% on average. For example, SymC outperforms the second-best model, UnixCoder, by 2.5% in accuracy on permuted samples. Similar to function name prediction, all the baselines are susceptible to simple statement permutations. We will add more discussions to the related works and experiments in the paper.

[7] Just, René, Darioush Jalali, and Michael D. Ernst. "Defects4J: A database of existing faults to enable controlled testing studies for Java programs." In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pp. 437-440. 2014.

[8] Guo, Daya, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou et al. "GraphCodeBERT: Pre-training code representations with data flow." arXiv preprint arXiv:2009.08366 (2020).

[9] Roziere, Baptiste, Marie-Anne Lachaux, Marc Szafraniec, and Guillaume Lample. "Dobf: A deobfuscation pre-training objective for programming languages." arXiv preprint arXiv:2102.07492 (2021).

[10] Wang, Yue, Weishi Wang, Shafiq Joty, and Steven CH Hoi. "Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation." arXiv preprint arXiv:2109.00859 (2021).

[11] Guo, Daya, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. "Unixcoder: Unified cross-modal pre-training for code representation." arXiv preprint arXiv:2203.03850 (2022).

[12] Feng, Zhangyin, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou et al. "Codebert: A pre-trained model for programming and natural languages." arXiv preprint arXiv:2002.08155 (2020).

Reviewer comment (Evaluation):
Why were the evaluation tasks limited to four artificial tasks created by the authors?

Author response:
**Q2 & Weakness 2: The evaluation framework relies heavily on four artificial tasks created by the authors, omitting well-established, practical benchmarks used commonly in the field. For instance, important code generation tasks like OpenAI HumanEval (https://huggingface.co/datasets/openai_humaneval) and MBPP (https://huggingface.co/datasets/mbpp), as well as code translation, clone detection, defect detection, and code repair tasks from CodeXGLUE (https://github.com/microsoft/CodeXGLUE), are all relevant to the domain but were not considered. This absence of evaluation on existing benchmarks and comparison with related works raises questions about the paper's soundness and the model's real-world applicability.**

* Regarding “artificial tasks”

Our analysis tasks are classic program analysis tasks, and they are all adopted in practice. For example, binary code similarity detection is used for detecting vulnerabilities [1] and patching them [2], function signature prediction is used to retrofit control flow integrity [3, 4], and memory region prediction is used to facilitate malware analysis and vulnerability detection [5, 6]. We followed the baselines by using the standard datasets and experimental setup.

[1] Pewny, Jannik, Behrad Garmany, Robert Gawlik, Christian Rossow, and Thorsten Holz. "Cross-architecture bug search in binary executables." In 2015 IEEE Symposium on Security and Privacy, pp. 709-724. IEEE, 2015.

[2] Duan, Yue, Xuezixiang Li, Jinghan Wang, and Heng Yin. "Deepbindiff: Learning program-wide code representations for binary diffing." In Network and distributed system security symposium. 2020.

[3] Van Der Veen, Victor, Enes Göktas, Moritz Contag, Andre Pawoloski, Xi Chen, Sanjay Rawat, Herbert Bos, Thorsten Holz, Elias Athanasopoulos, and Cristiano Giuffrida. "A tough call: Mitigating advanced code-reuse attacks at the binary level." In 2016 IEEE Symposium on Security and Privacy (SP), pp. 934-953. IEEE, 2016.

[4] Ge, Xinyang, Nirupama Talele, Mathias Payer, and Trent Jaeger. "Fine-grained control-flow integrity for kernel software." In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 179-194. IEEE, 2016.

[5] Balakrishnan, Gogul, Thomas Reps, David Melski, and Tim Teitelbaum. "Wysinwyx: What you see is not what you execute." In Working Conference on Verified Software: Theories, Tools, and Experiments, pp. 202-213. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005.

[6] Guo, Wenbo, Dongliang Mu, Xinyu Xing, Min Du, and Dawn Song. "{DEEPVSA}: Facilitating Value-set Analysis with Deep Learning for Postmortem Program Analysis." In 28th USENIX Security Symposium (USENIX Security 19), pp. 1787-1804. 2019.

Reviewer comment (Experiments):
The paper needs more evaluations, e.g. an evaluation of the robustness of SYMC using the adversarial attack methods based on code transformations.

Author response:
**Q1: As stated in the paper, current code LLMs struggle with generalization to new code. Have you tried to evaluate the robustness of SYMC using the adversarial attack methods based on code transformations? The evaluation may make your method more convincing.**

Thanks for bringing up the robustness angle against adversarial attacks. The key approach we started off from this paper is the development of a code model with **provable robustness by construction** against semantics-preserving code permutations. Guided by this framework, we proved that our approach is verifiably robust against any possible adversarial permutations, and thus not bound by minimal perturbations assumed in many adversarial attacks [1]. We leave exploring other code symmetry groups and their robustness guarantees as future work.

[1] Pierazzi, Fabio, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. "Intriguing properties of adversarial ML attacks in the problem space." In 2020 IEEE Symposium on Security and Privacy (S&P), pp. 1332-1349. IEEE, 2020.

Indeed, when evaluating *other code transformations beyond permutations*, we agree that adversarial attacks can potentially lead to a greater number of violations when they are guided properly, e.g., by the input gradient. Therefore, we evaluate SymC and the newly suggested code models against the adversarial attack for function name prediction, i.e., Averloc [2]. The adversarial transformations implemented in Averloc are comprehensive, and include the adversarial code transformations proposed in prior work by Yefet et al. [3], e.g., variable renaming, dead code insertion, etc., with additional transformations such as loop unrolling. We follow the setting in Averloc by computing the adversarial attacks against a seq2seq model trained by the Averloc authors, and evaluate SymC and the baselines (GrahCodeBERT as suggested by the reviewer, and DOBF and CodeT5 as suggested by reviewer Srm2) on the generated adversarial examples. This ensures a fair comparison by evaluating all the models on the same set of adversarial examples. The baselines are trained on the same exact training set as SymC.

|               | Original Sample (F1) | Adversarial Sample (F1) | Invariance Violation (the lower the better) |
|---------------|-----------------|--------------------|----------------------|
| SymC          | 52.9            | 47.5               | 26%                   |
| GraphCodeBERT | 52.56           | 42.89              | 51%                  |
| DOBF          | 51.59          | 39.68              | 51%                  |
| CodeT5        | 44.21           | 36.66              | 47%                |

The table shows that SymC outperforms the second-best baseline, GraphCodeBERT, by 10.7% and 49%, in F1 and violation rate on the adversarial examples, respectively, against adversarial transformations (beyond semantics-preserving permutations). This indicates the strong robustness of SymC against adversarial code transformations, even though the attacks are not statement permutations. 

[2] Henkel, Jordan, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. "Semantic robustness of models of source code." In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 526-537. IEEE, 2022.

[3] Yefet, Noam, Uri Alon, and Eran Yahav. "Adversarial examples for models of code." Proceedings of the ACM on Programming Languages 4, no. OOPSLA (2020): 1-30.

Reviewer comment (Experiments):
As stated in the paper, current code LLMs struggle with generalization to new code. Have you tried to evaluate the robustness of SYMC using the adversarial attack methods based on code transformations? The evaluation may make your method more convincing.

Author response:
**Q1: As stated in the paper, current code LLMs struggle with generalization to new code. Have you tried to evaluate the robustness of SYMC using the adversarial attack methods based on code transformations? The evaluation may make your method more convincing.**

Thanks for bringing up the robustness angle against adversarial attacks. The key approach we started off from this paper is the development of a code model with **provable robustness by construction** against semantics-preserving code permutations. Guided by this framework, we proved that our approach is verifiably robust against any possible adversarial permutations, and thus not bound by minimal perturbations assumed in many adversarial attacks [1]. We leave exploring other code symmetry groups and their robustness guarantees as future work.

[1] Pierazzi, Fabio, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. "Intriguing properties of adversarial ML attacks in the problem space." In 2020 IEEE Symposium on Security and Privacy (S&P), pp. 1332-1349. IEEE, 2020.

Indeed, when evaluating *other code transformations beyond permutations*, we agree that adversarial attacks can potentially lead to a greater number of violations when they are guided properly, e.g., by the input gradient. Therefore, we evaluate SymC and the newly suggested code models against the adversarial attack for function name prediction, i.e., Averloc [2]. The adversarial transformations implemented in Averloc are comprehensive, and include the adversarial code transformations proposed in prior work by Yefet et al. [3], e.g., variable renaming, dead code insertion, etc., with additional transformations such as loop unrolling. We follow the setting in Averloc by computing the adversarial attacks against a seq2seq model trained by the Averloc authors, and evaluate SymC and the baselines (GrahCodeBERT as suggested by the reviewer, and DOBF and CodeT5 as suggested by reviewer Srm2) on the generated adversarial examples. This ensures a fair comparison by evaluating all the models on the same set of adversarial examples. The baselines are trained on the same exact training set as SymC.

|               | Original Sample (F1) | Adversarial Sample (F1) | Invariance Violation (the lower the better) |
|---------------|-----------------|--------------------|----------------------|
| SymC          | 52.9            | 47.5               | 26%                   |
| GraphCodeBERT | 52.56           | 42.89              | 51%                  |
| DOBF          | 51.59          | 39.68              | 51%                  |
| CodeT5        | 44.21           | 36.66              | 47%                |

The table shows that SymC outperforms the second-best baseline, GraphCodeBERT, by 10.7% and 49%, in F1 and violation rate on the adversarial examples, respectively, against adversarial transformations (beyond semantics-preserving permutations). This indicates the strong robustness of SymC against adversarial code transformations, even though the attacks are not statement permutations. 

[2] Henkel, Jordan, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. "Semantic robustness of models of source code." In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 526-537. IEEE, 2022.

[3] Yefet, Noam, Uri Alon, and Eran Yahav. "Adversarial examples for models of code." Proceedings of the ACM on Programming Languages 4, no. OOPSLA (2020): 1-30.

Reviewer comment (Experiments):
Have you tried to compare with "Graphcodebert: Pre-training code representations with data flow", which is a state-of-the-art method considering the inherent structure of code, in your evaluation?

Author response:
**Q2: Have you tried to compare with "GraphCodeBERT: Pre-training code representations with data flow", which is a state-of-the-art method considering the inherent structure of code, in your evaluation?**

Thanks for pointing out the related works. We will add more discussion to GraphCodeBERT and other related works. To answer your specific question, we include the new evaluation results we have obtained during the rebuttal. As shown in the following, we compare SymC to the baselines including GraphCodeBERT. Significantly, SymC outperforms GrahCodeBert by 19.9%, even when it is pre-trained with substantial data augmentation effort. More importantly, SymC is provably robust to all permuted samples by construction i.e., 0% violation rate, while GraphCodeBERT (even pre-trained) has 13% of the labels changed under semantics-preserving permutations.

| | Original Sample (F1) | Permuted Samples (F1) | Invariance Violation Rate (the lower the better) |
| ----------|----------|----------|----------|
| SymC | 36.3 | 36.3 | 0 |
| GraphCodeBERT | 28.65 | 30.27 | 13 |
| GraphCodeBERT-no-pretrain | 20.83 | 20.62 | 31 |

Reviewer comment (Theory):
Page 5, "PDG (VPDG,EPDG) is a super graph of IG, sharing the same vertices but having a superset of edges (EPDG ⊇ EIG), because we consider all memory accesses as aliasing, making PDG a conservative construction of IG", If you "consider all memory accesses as aliasing", which is apparently a very weak encoding of the program semantics, it seems there would be too many aliases in the programs, making most statements unexchangeable to accomplish semantics-preserving statement permutations?

Author response:
**Q3: Page 5, "PDG ($V_{PDG}$,$E_{PDG}$) is a super graph of $\mathcal{IG}$, sharing the same vertices but having a superset of edges ($E_{PDG}\subseteq E_{\mathcal{IG}}$), because we consider all memory accesses as aliasing, making PDG a conservative construction of $\mathcal{IG}$", If you "consider all memory accesses as aliasing", which is apparently a very weak encoding of the program semantics, it seems there would be too many aliases in the programs, making most statements unexchangeable to accomplish semantics-preserving statement permutations?**

This is a great point. We will include additional investigation into this problem by adding concrete numbers showing the percentage of memory access instructions in the program. We agree there would be a tradeoff between efficiency and precision, but we employ the conservative strategy in this paper for simplicity and efficiency. This could also depend on the purposes of the program, e.g., computation-heavy programs employ arithmetic operations and leverage the CPU registers more. We observe that highly optimized binaries often have fewer memory accesses due to the compiler optimizations that aim to make the most efficient use of CPU registers.

Reviewer comment (Writing):
Page 6, "Each entry dij is a 2-value tuple (pij , nij), indicating the shortest path from the lowest common ancestor of Vi and Vj , denoted as Tij , to Vi and Vj , respectively", is pij the positive distances and nij the negative distances as denoted in the next paragraph? Also, what do you mean by positive distances and negative distances?

Author response:
**Q4: Page 6, "Each entry $d_{ij}$ is a 2-value tuple ($p_{ij}$, $n_{ij}$), indicating the shortest path from the lowest common ancestor of $V_i$ and $V_j$ , denoted as $T_{ij}$ , to $V_i$ and $V_j$ , respectively", is $p_{ij}$ the positive distances and $n_{ij}$ the negative distances as denoted in the next paragraph? Also, what do you mean by positive distances and negative distances?**

Sorry for the confusion. Yes, the positive and negative distance corresponds to $p_{ij}$ and $n_{ij}$, respectively. The positive distance denotes the distance from node $V_i$ to the lowest common ancestor of node i and j, $T_{ij}$. The negative distance denotes the distance from node $V_j$ to $T_{ij}$. We will make the explanation more clear in the paper.

Reviewer comment (Presentation):
Page 2, "SYMC enforces its output to stay invariant via keeping its learned representation G-equivariant, where the code representation (e1, e2, e3, e4) is transformed into (e2, e3, e4, e1) xxx", should "(e2, e3, e4, e1)" be "(e2, e1, e3, e4)" as shown in Figure 1a?

Author response:
**Q5: Page 2, "SYMC enforces its output to stay invariant via keeping its learned representation G-equivariant, where the code representation $(e_1, e_2, e_3, e_4)$ is transformed into $(e_2, e_3, e_4, e_1)$ xxx", should "$(e_2, e_3, e_4, e_1)$" be "$(e_2, e_1, e_3, e_4)$" as shown in Figure 1a?**

Great catch! We have fixed the mistake in the updated paper.

Reviewer comment (Presentation):
Page 9, the lines labeled 2nd, 4th, 6th in Figure 4a. Which lines are for Aut(PDG)-equivariant self-attention layers and which are for the Aut(PDG)-invariant ones?

Author response:
**Q6: Page 9, the lines labeled 2nd, 4th, 6th in Figure 4a. Which lines are for Aut(PDG)-equivariant self-attention layers and which are for the Aut(PDG)-invariant ones?**

Figure 4a shows the performance drop when we set the subsequent self-attention layers Aut(PDG)-*invariant*, not Aut(PDG)-*equivariant*, starting from 2nd, 4th, 6th self-attention layer (see caption of Figure 4). This is to demonstrate having equivariance in the code representation learning is important, compared to making it invariant throughout all the layers. We will clarify this in the paper.

Reviewer comment (Presentation):
Hard to understand exactly what program interpretation graphs and program dependence graphs look like, which is crucial to the paper.

Author response:
**Q1 & Weakness 1: Could you illustrate an example program interpretation graph and program dependence graphs? This would be quite helpful for understanding.**

Consider the following Java example:

```java
……
1       int a = 1;
2       int b = 2;
3       if (a > b) {
4           return 1;
5       } else {
6           return 0;
7       }
……
```
We will have 5 nodes in IG, i.e., {1,2,3,4,6} denoting 5 statements, and the edges {1->3, 2->3, 3->4, 3->6}. In such a case, IG is equivalent to PDG.

Consider the case in binary analysis:

```asm
…
1       mov rcx, [rbx]
2       mov [rax], rdx
…
```
We have 2 nodes in IG, i.e., {1,2} denoting 2 instructions, and there is no edge in actual IG, as there is no dataflow from 1 to 2. However, in PDG, we conservatively consider the edge {1->2} by marking them as a write-after-read dependence. This is because we can not easily eliminate the possibility that `[rbx]` and `[rax]` would point to the same memory location.

Reviewer comment (Reproducibility):
Experimental details are lacking. What is the training procedure for SymC, is it just supervised training on the downstream task? How about for the other models? For Function Name prediction, do the LLMs take in just the text as input, and what exactly does SymC take as input there?

Author response:
**Weakness 2: Experimental details are lacking. What is the training procedure for SymC, is it just supervised training on the downstream task? How about the other models? For Function Name prediction, do the LLMs take in just the text as input, and what exactly does SymC take as input there?**

SymC adopts only supervised training and trains the model from scratch *without any pre-training*. While Figure 4(b) demonstrates that adding pre-training brings extra improvement, we aim to demonstrate having symmetry-preserving architecture alone leads to better performance than many pre-trained models.

For other baselines, we respect their own training paradigm. In particular, PalmTree, GPT4, and WizardCoder are pre-trained on programs in the wild. code2vec, code2seq, and GGNN use supervised training. The new models that we have experimented with during the rebuttal, i.e., CodeBert, GraphCodeBERT, UnixCoder, DOBF, and CodeT5 (see our responses to reviewer Srm2), are pre-trained with large-scale code datasets. 

For function name prediction, both LLMs and SymC take as input the function (function body and function signature) with the function name stripped.

Reviewer comment (Experiments):
Computation of graphs associated to code may be costly and restrictive.

Author response:
**Q2 & Weakness 3: How costly is obtaining the code graphs? Computation of graphs associated to code may be costly and restrictive.**

To incorporate the inductive bias of code, many code models involve extracting and encoding code structures, including our baselines, as computing PDGs statically is not overly expensive. Below, we add the runtime performance (in milliseconds) per code sample of SymC and PalmTree on binary analysis, training, and inference, using the same exact hardware (an AMD EPYC 7502 processor, 128 virtual cores, 256GB RAM, Nvidia RTX 3090 GPUs). 

| | Graph Construction | Train | Inference |
| ----------|----------|----------|----------|
| SymC | 250.2ms | 15.4ms | 2ms |
| PalmTree | 4460ms | 36ms | 17ms |

SymC’s cheap computation of PDG incurs 17.8x less runtime overhead. It is also an interesting research problem to incorporate optimization, e.g., caching, to improve the efficiency of the PDG computation during inference. We will update the paper to include both source and binary analysis tasks.

Reviewer comment (Presentation):
Could you illustrate example program interpretation graph and program dependence graphs? This would be quite helpful for understanding.

Author response:
**Q1 & Weakness 1: Could you illustrate an example program interpretation graph and program dependence graphs? This would be quite helpful for understanding.**

Consider the following Java example:

```java
……
1       int a = 1;
2       int b = 2;
3       if (a > b) {
4           return 1;
5       } else {
6           return 0;
7       }
……
```
We will have 5 nodes in IG, i.e., {1,2,3,4,6} denoting 5 statements, and the edges {1->3, 2->3, 3->4, 3->6}. In such a case, IG is equivalent to PDG.

Consider the case in binary analysis:

```asm
…
1       mov rcx, [rbx]
2       mov [rax], rdx
…
```
We have 2 nodes in IG, i.e., {1,2} denoting 2 instructions, and there is no edge in actual IG, as there is no dataflow from 1 to 2. However, in PDG, we conservatively consider the edge {1->2} by marking them as a write-after-read dependence. This is because we can not easily eliminate the possibility that `[rbx]` and `[rax]` would point to the same memory location.

Reviewer comment (Experiments):
How costly is obtaining the code graphs?

Author response:
**Q2 & Weakness 3: How costly is obtaining the code graphs? Computation of graphs associated to code may be costly and restrictive.**

To incorporate the inductive bias of code, many code models involve extracting and encoding code structures, including our baselines, as computing PDGs statically is not overly expensive. Below, we add the runtime performance (in milliseconds) per code sample of SymC and PalmTree on binary analysis, training, and inference, using the same exact hardware (an AMD EPYC 7502 processor, 128 virtual cores, 256GB RAM, Nvidia RTX 3090 GPUs). 

| | Graph Construction | Train | Inference |
| ----------|----------|----------|----------|
| SymC | 250.2ms | 15.4ms | 2ms |
| PalmTree | 4460ms | 36ms | 17ms |

SymC’s cheap computation of PDG incurs 17.8x less runtime overhead. It is also an interesting research problem to incorporate optimization, e.g., caching, to improve the efficiency of the PDG computation during inference. We will update the paper to include both source and binary analysis tasks.

Reviewer comment (Reproducibility):
Why does SymC require "40,162 lines of code"? I'm curious as to what makes it require so much.

Author response:
**Q3: Why does SymC require "40,162 lines of code"? I'm curious as to what makes it require so much.**

Besides implementing the model and algorithm (~4.5k lines), we include the lines of code for doing program analysis (both source and binary code) for obtaining PDGs (2.4k lines) and generating ground truths for the analysis tasks (3k lines). We also include the code for setting up the baselines (4.3k lines), implementing program transformations (2.8k lines), data processing (1k), and all the scripts for experiments and result analysis (22k lines). We will clarify in the paper by providing a breakdown for each of these components.