Paper: COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuits (ICLR-2024)

Reviewer comment (Theory):
While the authors do consider the finite-sample error induced by the finite size of the calibration set, they fail to account for the finite-sample error due to the Monte-Carlo sampling (they use 10000 samples) when estimating the expectations under the randomized smoothing framework. Therefore, the resulting sets are only asymptotically valid.

Author response:
> Q1: Consideration of finite-sample error induced by Monte-Carlo sampling in randomized smoothing. How does COLEP perform when accounting for finite-sample errors when estimating randomized smoothing expectations?

We thank you for bringing up this interesting point! We added the following discussions as well as theoretical and empirical results considering the finite-sample errors of randomized smoothing in the revised manuscript.

RSCP+ [1] points out the neglect of finite-sample errors of Monte-Carlo sampling by randomized smoothing in RSCP [2]. A naive solution is to apply union bound over all calibration samples, but it leads to a loose prediction set as shown in RSCP+.  Therefore, RSCP+ proposes a subtle solution by **viewing the empirical mean $\hat{S}\_{\text{RS}}$ as the non-conformity scores** and thus only need to bound the finite-sample error of the empirical mean $\hat{S}\_{\text{RS}}$ on the test sample $(x\_{n+1},y\_{n+1})$ (Corollary 2 in [1]). Specifically, they apply Hoeffding’s inequality and Empirical Bernstein’s inequality to bound the finite-sample error of empirical mean $\hat{S}\_{\text{RS}}$ compared to the true mean $S\_{\text{RS}}$ of the test sample $(x\_{n+1},y\_{n+1})$ (Theorem C.6 in [1]). In this way, RSCP+ does not need to apply union over calibration samples and one concentration bound over the test sample is sufficient.

Since COLEP applies randomized smoothing to the model output in the learning component instead of the end-to-end non-conformity scores, we need to deal with the finite-sample error in a slightly different way, but the general spirit aligns with RSCP+.
COLEP applies randomized smoothing to get the bound of output probabilities of the main model and knowledge models, and then propagates the bound to the probability bound after the reasoning component (Theorem 1) and the bound of non-conformity scores (Theorem 2).
Therefore, to consider the finite-sample errors, we need to define a **$\beta$-confidence worst-case non-conformity score** based on Equation (12) in Theorem 2. Concretely, in Equation (12), we replace the original probability upper bound $\mathbb{U}\_{\delta}[\cdot]$ with a $\beta$-confidence version $\mathbb{U}\_{\beta,\delta}[\cdot]$, denoting the upper bound of non-conformity scores with confidence $1-\beta$, and similarly for the lower bound. In this way, during the proof of Theorem 2, we need to apply the union bound in the derivation from Equation (53) to Equation (54) by considering the event probability $\mathbb{P}[\hat{\pi}\_j^{\text{COLEP}}(\tilde{X}\_{n+1}) \le \mathbb{U}\_{\beta,\delta}[\hat{\pi}\_j^{\text{COLEP}}(X\_{n+1})] ] \ge 1 - 2\beta$. Similarly, we consider the event related to the lower bound in the derivation from Equation (57) to Equation (58). Finally, to counter the effect of finite-sample error and maintain a $1-\alpha$ coverage, we need to compute the $1-\alpha+2\beta$ quantile in Equation (11) in Theorem 2. We provided the rigorous theorem statement and proofs in **Theorem 6 in Appendix F.3**. We separated the theoretical result with finite-sample errors since Theorem 2 also allows for deterministic certification methods such as bound propagation methods which do not lead to finite-sample errors. We added related discussions and references in the remarks of Theorem 2.

We also empirically considered the finite-sample errors induced by randomized smoothing expectation approximation and provided the updated results in Figure 2 and Figure 3 in the revised manuscript. We consider the finite-sample errors of RSCP following RSCP+ (Corollary 2 in [1]) and the finite-sample errors of COLEP following Theorem 6 in Appendix F.3. We performed 100k Monte-Carlo sampling for a smaller error in practice and provided all the details in the experiment section of the revised manuscript. We generally observe an error of $[0.002,0.005]$ induced by finite samples of randomized smoothing. One can further reduce the finite-sample errors by increasing the Monte-Carlo sample sizes or applying better concentration bounds. From the updated results in Figure 2 and Figure 3, we can still conclude that COLEP with the reasoning component demonstrates better robustness and prediction efficiency than SOTA baseline RSCP.

*[1] Provably Robust Conformal Prediction with Improved Efficiency, ICLR 2024 submission (https://openreview.net/forum?id=BWAhEjXjeG)*

*[2] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

Reviewer comment (Theory):
As pointed out by a concurent ICLR submission (https://openreview.net/forum?id=BWAhEjXjeG) RSCP also suffers from the same issue and it is non-trivial to correct for this. The same issue is also discussed among the reviewers and the RSCP authors on their respective openreview page (https://openreview.net/forum?id=9L1BsI4wP1H). Naively, one would have to apply a union bound over all examples in the calibration set and the test example such that each of the randomized smoothing expectations hold simultaneously. There is also a subtler alternative solution. When correcting RSCP for this error the resulting sets end up returning all labels, i.e. they are useless. It is not clear to which degree COLEP suffers from the same issue. In any case, the "fix" proposed by the concurent submission is orthogonal and can be applied to COLEP as well. Still, this finite-sample issue needs to be addressed in the paper, especially given that the goal is to produce a sound certificate.

Author response:
We thank you for the reply! In the updated version, we showed the results of RSCP+ (with error correction). [1] reports a very high set size of baseline RSCP+ because they use a small Monte-Carlo sample size $N_{\text{MC}}=256$, according to the paragraph of hyperparameters in Section 5 of [1]. However, we use a large Monte-Carlo sample size of 100K in the updated evaluations. [1] also shows the sensitivity of the selection of Monte-Carlo sample sizes to the certification results in Table 3 of [1]. We selected a large Monte-Carlo sample size to reduce the influence of finite-sample error and focused more on the effectiveness of the reasoning component. Following the suggestion, we provided results of comparisons between COLEP (without finite-sample error as Theorem 2) and COLEP+ (with finite-sample errors as Theorem 6) in Table 5 in Appendix J.3. The results show that with the sample size of 100K, we generally observe a small gap ($0.002\sim0.005$) induced by finite-sample errors.

*[1] Provably Robust Conformal Prediction with Improved Efficiency, ICLR 2024 submission (https://openreview.net/forum?id=BWAhEjXjeG)*

*[2] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

Reviewer comment (Evaluation):
The second weakness is in the PGD attack. The authors state "For fair comparisons, we apply PGD attack (Madry et al., 2018) with the same parameters on CP, RSCP, and COLEP". However, it is unclear whether the PGD attack is adaptive, i.e. it takes the learning-reasoning component into account. An adaptive attacker can in fact know that a reasoning component is used, and thus can try to perturb the input such that both the class and the concept probabilities are suitably changed as to fool the entire pipeline. While this is not critical, not using adaptive attacks makes the conclusions drawn from Figure 3 less reliable.

Author response:
> Q2: Clarification of adaptive PGD attack against the learning-reasoning pipeline.

We thank you for the valuable comment. We added a clear clarification that we consider an adaptive PGD attack against the complete learning-reasoning pipeline in Section 7. Formally, the attack process can be formulated as the optimization problem: $\max\_{\|\eta\|\_2 \le \delta} \ell\_\text{CE}(\hat{\pi}^{\text{COLEP}}(x),y)$, where $\ell\_\text{CE}$ is the cross-entropy loss and $\hat{\pi}^{\text{COLEP}}(x)$ is the class probabilities given sample $x$ computed by Equation (6) which takes the reasoning component into consideration.

Reviewer comment (Evaluation):
How does COLEP perform when accounting for finite-sample errors when estimating randomized smoothing expectations?

Author response:
> Q1: Consideration of finite-sample error induced by Monte-Carlo sampling in randomized smoothing. How does COLEP perform when accounting for finite-sample errors when estimating randomized smoothing expectations?

We thank you for bringing up this interesting point! We added the following discussions as well as theoretical and empirical results considering the finite-sample errors of randomized smoothing in the revised manuscript.

RSCP+ [1] points out the neglect of finite-sample errors of Monte-Carlo sampling by randomized smoothing in RSCP [2]. A naive solution is to apply union bound over all calibration samples, but it leads to a loose prediction set as shown in RSCP+.  Therefore, RSCP+ proposes a subtle solution by **viewing the empirical mean $\hat{S}\_{\text{RS}}$ as the non-conformity scores** and thus only need to bound the finite-sample error of the empirical mean $\hat{S}\_{\text{RS}}$ on the test sample $(x\_{n+1},y\_{n+1})$ (Corollary 2 in [1]). Specifically, they apply Hoeffding’s inequality and Empirical Bernstein’s inequality to bound the finite-sample error of empirical mean $\hat{S}\_{\text{RS}}$ compared to the true mean $S\_{\text{RS}}$ of the test sample $(x\_{n+1},y\_{n+1})$ (Theorem C.6 in [1]). In this way, RSCP+ does not need to apply union over calibration samples and one concentration bound over the test sample is sufficient.

Since COLEP applies randomized smoothing to the model output in the learning component instead of the end-to-end non-conformity scores, we need to deal with the finite-sample error in a slightly different way, but the general spirit aligns with RSCP+.
COLEP applies randomized smoothing to get the bound of output probabilities of the main model and knowledge models, and then propagates the bound to the probability bound after the reasoning component (Theorem 1) and the bound of non-conformity scores (Theorem 2).
Therefore, to consider the finite-sample errors, we need to define a **$\beta$-confidence worst-case non-conformity score** based on Equation (12) in Theorem 2. Concretely, in Equation (12), we replace the original probability upper bound $\mathbb{U}\_{\delta}[\cdot]$ with a $\beta$-confidence version $\mathbb{U}\_{\beta,\delta}[\cdot]$, denoting the upper bound of non-conformity scores with confidence $1-\beta$, and similarly for the lower bound. In this way, during the proof of Theorem 2, we need to apply the union bound in the derivation from Equation (53) to Equation (54) by considering the event probability $\mathbb{P}[\hat{\pi}\_j^{\text{COLEP}}(\tilde{X}\_{n+1}) \le \mathbb{U}\_{\beta,\delta}[\hat{\pi}\_j^{\text{COLEP}}(X\_{n+1})] ] \ge 1 - 2\beta$. Similarly, we consider the event related to the lower bound in the derivation from Equation (57) to Equation (58). Finally, to counter the effect of finite-sample error and maintain a $1-\alpha$ coverage, we need to compute the $1-\alpha+2\beta$ quantile in Equation (11) in Theorem 2. We provided the rigorous theorem statement and proofs in **Theorem 6 in Appendix F.3**. We separated the theoretical result with finite-sample errors since Theorem 2 also allows for deterministic certification methods such as bound propagation methods which do not lead to finite-sample errors. We added related discussions and references in the remarks of Theorem 2.

We also empirically considered the finite-sample errors induced by randomized smoothing expectation approximation and provided the updated results in Figure 2 and Figure 3 in the revised manuscript. We consider the finite-sample errors of RSCP following RSCP+ (Corollary 2 in [1]) and the finite-sample errors of COLEP following Theorem 6 in Appendix F.3. We performed 100k Monte-Carlo sampling for a smaller error in practice and provided all the details in the experiment section of the revised manuscript. We generally observe an error of $[0.002,0.005]$ induced by finite samples of randomized smoothing. One can further reduce the finite-sample errors by increasing the Monte-Carlo sample sizes or applying better concentration bounds. From the updated results in Figure 2 and Figure 3, we can still conclude that COLEP with the reasoning component demonstrates better robustness and prediction efficiency than SOTA baseline RSCP.

*[1] Provably Robust Conformal Prediction with Improved Efficiency, ICLR 2024 submission (https://openreview.net/forum?id=BWAhEjXjeG)*

*[2] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

Reviewer comment (Theory):
In section 3 you have "assume that the data samples are drawn i.i.d. (thereby exchangeably)". Is the i.i.d. assumptions necessary for the learning-reasonig component, or can this be relaxed to just exchangeability as with vanilla CP?

Author response:
> Q3: Are the i.i.d. assumptions necessary for the learning-reasoning component, or can this be relaxed to just exchangeability as with vanilla CP?

We thank you for the valuable questions. In Section 3 of the revised manuscript, we made it clear that we **only need the exchangeability assumptions** as vanilla CP. Specifically, we only assume the clean test sample is drawn **exchangeably** to the calibration samples and the adversarial sample is within bounded perturbations from the clean test sample.

Reviewer comment (Experiments):
Is the PGD attack adaptive, i.e. takes the reasoning component into account?

Author response:
> Q2: Clarification of adaptive PGD attack against the learning-reasoning pipeline.

We thank you for the valuable comment. We added a clear clarification that we consider an adaptive PGD attack against the complete learning-reasoning pipeline in Section 7. Formally, the attack process can be formulated as the optimization problem: $\max\_{\|\eta\|\_2 \le \delta} \ell\_\text{CE}(\hat{\pi}^{\text{COLEP}}(x),y)$, where $\ell\_\text{CE}$ is the cross-entropy loss and $\hat{\pi}^{\text{COLEP}}(x)$ is the class probabilities given sample $x$ computed by Equation (6) which takes the reasoning component into consideration.

Reviewer comment (Experiments):
Empirical evaluation is quite limited, which is quite unfortunate given that including more settings beyond datasets as simple as CIFAR-10, as well as more benchmarks, would likely let the guarantees afforded by the learning-reasoning setup shine even more: both in terms of the ability to construct rules specific to each considered setting, as well as the resulting adaptivity compared to other conformal scores/methods.

Author response:
We thank you for your valuable comments and suggestions on the evaluation part. We made the following updates to clarify and strengthen the evaluation of COLEP.

1. We avoided the confusion induced by improper namings in Section 7. APS score is indeed not the SOTA method, and thus, we called it a baseline of conformal prediction and only called RSCP [1] as the SOTA approach in adversarially robust conformal prediction in the revised version.

2. We followed the suggestion to compare with more baselines of various conformal prediction scores to show the robustness of COLEP. Specifically, we further evaluate the RAPS score [2], which improves the APS score by adding a regularization term regarding the ranking of the ground truth label. We also evaluate the recent clustered conformal prediction (CCP) [3], which computes the quantile and performs calibration at the cluster level. We compare COLEP with these baselines on AwA2 under PGD attack. The results in the following Table 1 show that (1) conformal prediction with vanilla conformity scores APS, RAPS, and CCP leads to a broken coverage in the adversary setting, while adversarially robust conformal prediction method RSCP and COLEP still maintains the valid coverage, and (2) compared to RSCP, COLEP demonstrates a better tradeoff between the coverage and prediction efficiency.

3. We added the following clarifications and more evaluations to demonstrate the scalability of COLEP. Besides the relatively small-scale but standard benchmark CIFAR-10 and GTSRB, we also conducted evaluations on AwA2 [4] in different settings in Figure 2 and Figure 3 in Section 7. AwA2 dataset contains 37,322 images of 50 different animal classes, which are extracted from the ImageNet dataset and with a high resolution of $256 \times 256$. The dataset also has 85 attribute labels (concept labels) for each animal class and is suitable for evaluations of knowledge-intensive tasks. In total, we construct 187 implication rules in total with 4 types of knowledge encoded in 4 PCs separately. Evaluations on AwA2 are already the largest scale in the related literature on learning-reasoning [5] and neuro-symbolic [6]. The scalability of COLEP comes from the usage of probabilistic circuits with linear inference time with respect to the graph size instead of the intractable Markov logic network with NP-complete time complexity used by prior works [5,6]. To consolidate the scalability to AwA2, we further added more evaluations with multiple perturbation bounds in the following Table 1. The results together with results in Figure 2 and Figure 3 on AwA2 demonstrate the scalability and robustness of COLEP to achieve an impressive prediction coverage and efficiency in the adversary setting. 

We also admit that it is challenging for evaluations on the full ImageNet with 1000 classes due to the expensive costs of training thousands of concept models and the lack of knowledge rules at the current stage, which is also discussed as one limitation in Appendix A. However, we deem that the principles and analysis in COLEP shed light on the effectiveness of the reasoning component to achieve a more robust conformal prediction. It is also interesting for future work to leverage the CLIP model for concept detection and the large language models for the design of knowledge rules, which can fully unleash the power of COLEP on more large-scale evaluations.

Table 1. Marginal coverage / average set size of multiple conformal prediction baselines, SOTA method RSCP, and COLEP under PGD Attack and AutoAttack with $\ell_2$ bounded perturbations $0.25$ and $0.50$ on AwA2. The nominal level of coverage is $0.9$.

|  | APS | RAPS | CCP | RSCP | COLEP |
| --- | ----------- | ----------- | ----------- | ----------- | ----------- | 
| $\delta=0.25$ | 0.8325 / 3.22 | 0.8363 / 3.20 | 0.8427 / 3.53 | 0.945 / 6.41 | 0.962 / 4.74 |
| $\delta=0.50$ | 0.7829 / 2.83 | 0.7821 / 2.89 | 0.7894 / 2.93 | 0.932 / 5.83 | 0.942 / 4.42 |

*[1] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

*[2] Angelopoulos, Anastasios, et al. "Uncertainty sets for image classifiers using conformal prediction." ICLR 2021.*

*[3] Ding, Tiffany, et al. "Class-Conditional Conformal Prediction With Many Classes." NeurIPS 2023.*

*[4] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." TPAMI 2018.*

*[5] Yang, Zhuolin. Improving certified robustness via statistical learning with logical reasoning. NeurIPS 2022.*

*[6] Wu, Tailin, et al. "Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time." NeurIPS 2022.*

Reviewer comment (Experiments):
I was quite confused when the authors repeatedly call the APS score of Romano et al (2020) 'SOTA'. It is certainly one of the existing reasonable conformal approaches to use, but by no means state of the art when it comes to adaptivity and especially set size. For instance, its generalization and improvement, the RAPS nonconformity score introduced in "Uncertainty Sets for Image Classifiers using Conformal Prediction", may still be considered close to the SOTA frontier for many domains — and it would be very pertinent to include it, along with further improved conformal methods developed thereafter.

Author response:
We thank you for your valuable comments and suggestions on the evaluation part. We made the following updates to clarify and strengthen the evaluation of COLEP.

1. We avoided the confusion induced by improper namings in Section 7. APS score is indeed not the SOTA method, and thus, we called it a baseline of conformal prediction and only called RSCP [1] as the SOTA approach in adversarially robust conformal prediction in the revised version.

2. We followed the suggestion to compare with more baselines of various conformal prediction scores to show the robustness of COLEP. Specifically, we further evaluate the RAPS score [2], which improves the APS score by adding a regularization term regarding the ranking of the ground truth label. We also evaluate the recent clustered conformal prediction (CCP) [3], which computes the quantile and performs calibration at the cluster level. We compare COLEP with these baselines on AwA2 under PGD attack. The results in the following Table 1 show that (1) conformal prediction with vanilla conformity scores APS, RAPS, and CCP leads to a broken coverage in the adversary setting, while adversarially robust conformal prediction method RSCP and COLEP still maintains the valid coverage, and (2) compared to RSCP, COLEP demonstrates a better tradeoff between the coverage and prediction efficiency.

3. We added the following clarifications and more evaluations to demonstrate the scalability of COLEP. Besides the relatively small-scale but standard benchmark CIFAR-10 and GTSRB, we also conducted evaluations on AwA2 [4] in different settings in Figure 2 and Figure 3 in Section 7. AwA2 dataset contains 37,322 images of 50 different animal classes, which are extracted from the ImageNet dataset and with a high resolution of $256 \times 256$. The dataset also has 85 attribute labels (concept labels) for each animal class and is suitable for evaluations of knowledge-intensive tasks. In total, we construct 187 implication rules in total with 4 types of knowledge encoded in 4 PCs separately. Evaluations on AwA2 are already the largest scale in the related literature on learning-reasoning [5] and neuro-symbolic [6]. The scalability of COLEP comes from the usage of probabilistic circuits with linear inference time with respect to the graph size instead of the intractable Markov logic network with NP-complete time complexity used by prior works [5,6]. To consolidate the scalability to AwA2, we further added more evaluations with multiple perturbation bounds in the following Table 1. The results together with results in Figure 2 and Figure 3 on AwA2 demonstrate the scalability and robustness of COLEP to achieve an impressive prediction coverage and efficiency in the adversary setting. 

We also admit that it is challenging for evaluations on the full ImageNet with 1000 classes due to the expensive costs of training thousands of concept models and the lack of knowledge rules at the current stage, which is also discussed as one limitation in Appendix A. However, we deem that the principles and analysis in COLEP shed light on the effectiveness of the reasoning component to achieve a more robust conformal prediction. It is also interesting for future work to leverage the CLIP model for concept detection and the large language models for the design of knowledge rules, which can fully unleash the power of COLEP on more large-scale evaluations.

Table 1. Marginal coverage / average set size of multiple conformal prediction baselines, SOTA method RSCP, and COLEP under PGD Attack and AutoAttack with $\ell_2$ bounded perturbations $0.25$ and $0.50$ on AwA2. The nominal level of coverage is $0.9$.

|  | APS | RAPS | CCP | RSCP | COLEP |
| --- | ----------- | ----------- | ----------- | ----------- | ----------- | 
| $\delta=0.25$ | 0.8325 / 3.22 | 0.8363 / 3.20 | 0.8427 / 3.53 | 0.945 / 6.41 | 0.962 / 4.74 |
| $\delta=0.50$ | 0.7829 / 2.83 | 0.7821 / 2.89 | 0.7894 / 2.93 | 0.932 / 5.83 | 0.942 / 4.42 |

*[1] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

*[2] Angelopoulos, Anastasios, et al. "Uncertainty sets for image classifiers using conformal prediction." ICLR 2021.*

*[3] Ding, Tiffany, et al. "Class-Conditional Conformal Prediction With Many Classes." NeurIPS 2023.*

*[4] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." TPAMI 2018.*

*[5] Yang, Zhuolin. Improving certified robustness via statistical learning with logical reasoning. NeurIPS 2022.*

*[6] Wu, Tailin, et al. "Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time." NeurIPS 2022.*

Reviewer comment (Experiments):
CIFAR-10 does not quite sound like the natural domain to apply potentially highly complex learning-reasoning boosts, so it would be wise to show the performance of the framework on Imagenet; as a byproduct, it would help to convince the reader of the tractability of useful learning-reasoning in more challenging settings.

Author response:
We thank you for your valuable comments and suggestions on the evaluation part. We made the following updates to clarify and strengthen the evaluation of COLEP.

1. We avoided the confusion induced by improper namings in Section 7. APS score is indeed not the SOTA method, and thus, we called it a baseline of conformal prediction and only called RSCP [1] as the SOTA approach in adversarially robust conformal prediction in the revised version.

2. We followed the suggestion to compare with more baselines of various conformal prediction scores to show the robustness of COLEP. Specifically, we further evaluate the RAPS score [2], which improves the APS score by adding a regularization term regarding the ranking of the ground truth label. We also evaluate the recent clustered conformal prediction (CCP) [3], which computes the quantile and performs calibration at the cluster level. We compare COLEP with these baselines on AwA2 under PGD attack. The results in the following Table 1 show that (1) conformal prediction with vanilla conformity scores APS, RAPS, and CCP leads to a broken coverage in the adversary setting, while adversarially robust conformal prediction method RSCP and COLEP still maintains the valid coverage, and (2) compared to RSCP, COLEP demonstrates a better tradeoff between the coverage and prediction efficiency.

3. We added the following clarifications and more evaluations to demonstrate the scalability of COLEP. Besides the relatively small-scale but standard benchmark CIFAR-10 and GTSRB, we also conducted evaluations on AwA2 [4] in different settings in Figure 2 and Figure 3 in Section 7. AwA2 dataset contains 37,322 images of 50 different animal classes, which are extracted from the ImageNet dataset and with a high resolution of $256 \times 256$. The dataset also has 85 attribute labels (concept labels) for each animal class and is suitable for evaluations of knowledge-intensive tasks. In total, we construct 187 implication rules in total with 4 types of knowledge encoded in 4 PCs separately. Evaluations on AwA2 are already the largest scale in the related literature on learning-reasoning [5] and neuro-symbolic [6]. The scalability of COLEP comes from the usage of probabilistic circuits with linear inference time with respect to the graph size instead of the intractable Markov logic network with NP-complete time complexity used by prior works [5,6]. To consolidate the scalability to AwA2, we further added more evaluations with multiple perturbation bounds in the following Table 1. The results together with results in Figure 2 and Figure 3 on AwA2 demonstrate the scalability and robustness of COLEP to achieve an impressive prediction coverage and efficiency in the adversary setting. 

We also admit that it is challenging for evaluations on the full ImageNet with 1000 classes due to the expensive costs of training thousands of concept models and the lack of knowledge rules at the current stage, which is also discussed as one limitation in Appendix A. However, we deem that the principles and analysis in COLEP shed light on the effectiveness of the reasoning component to achieve a more robust conformal prediction. It is also interesting for future work to leverage the CLIP model for concept detection and the large language models for the design of knowledge rules, which can fully unleash the power of COLEP on more large-scale evaluations.

Table 1. Marginal coverage / average set size of multiple conformal prediction baselines, SOTA method RSCP, and COLEP under PGD Attack and AutoAttack with $\ell_2$ bounded perturbations $0.25$ and $0.50$ on AwA2. The nominal level of coverage is $0.9$.

|  | APS | RAPS | CCP | RSCP | COLEP |
| --- | ----------- | ----------- | ----------- | ----------- | ----------- | 
| $\delta=0.25$ | 0.8325 / 3.22 | 0.8363 / 3.20 | 0.8427 / 3.53 | 0.945 / 6.41 | 0.962 / 4.74 |
| $\delta=0.50$ | 0.7829 / 2.83 | 0.7821 / 2.89 | 0.7894 / 2.93 | 0.932 / 5.83 | 0.942 / 4.42 |

*[1] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

*[2] Angelopoulos, Anastasios, et al. "Uncertainty sets for image classifiers using conformal prediction." ICLR 2021.*

*[3] Ding, Tiffany, et al. "Class-Conditional Conformal Prediction With Many Classes." NeurIPS 2023.*

*[4] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." TPAMI 2018.*

*[5] Yang, Zhuolin. Improving certified robustness via statistical learning with logical reasoning. NeurIPS 2022.*

*[6] Wu, Tailin, et al. "Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time." NeurIPS 2022.*

Reviewer comment (Experiments):
Having just three relatively simple datasets and only two comparison methods, Romano et al and Gendler et al, appears insufficient to fully explore the relative advantages of learning-reasoning.

Author response:
We thank you for your valuable comments and suggestions on the evaluation part. We made the following updates to clarify and strengthen the evaluation of COLEP.

1. We avoided the confusion induced by improper namings in Section 7. APS score is indeed not the SOTA method, and thus, we called it a baseline of conformal prediction and only called RSCP [1] as the SOTA approach in adversarially robust conformal prediction in the revised version.

2. We followed the suggestion to compare with more baselines of various conformal prediction scores to show the robustness of COLEP. Specifically, we further evaluate the RAPS score [2], which improves the APS score by adding a regularization term regarding the ranking of the ground truth label. We also evaluate the recent clustered conformal prediction (CCP) [3], which computes the quantile and performs calibration at the cluster level. We compare COLEP with these baselines on AwA2 under PGD attack. The results in the following Table 1 show that (1) conformal prediction with vanilla conformity scores APS, RAPS, and CCP leads to a broken coverage in the adversary setting, while adversarially robust conformal prediction method RSCP and COLEP still maintains the valid coverage, and (2) compared to RSCP, COLEP demonstrates a better tradeoff between the coverage and prediction efficiency.

3. We added the following clarifications and more evaluations to demonstrate the scalability of COLEP. Besides the relatively small-scale but standard benchmark CIFAR-10 and GTSRB, we also conducted evaluations on AwA2 [4] in different settings in Figure 2 and Figure 3 in Section 7. AwA2 dataset contains 37,322 images of 50 different animal classes, which are extracted from the ImageNet dataset and with a high resolution of $256 \times 256$. The dataset also has 85 attribute labels (concept labels) for each animal class and is suitable for evaluations of knowledge-intensive tasks. In total, we construct 187 implication rules in total with 4 types of knowledge encoded in 4 PCs separately. Evaluations on AwA2 are already the largest scale in the related literature on learning-reasoning [5] and neuro-symbolic [6]. The scalability of COLEP comes from the usage of probabilistic circuits with linear inference time with respect to the graph size instead of the intractable Markov logic network with NP-complete time complexity used by prior works [5,6]. To consolidate the scalability to AwA2, we further added more evaluations with multiple perturbation bounds in the following Table 1. The results together with results in Figure 2 and Figure 3 on AwA2 demonstrate the scalability and robustness of COLEP to achieve an impressive prediction coverage and efficiency in the adversary setting. 

We also admit that it is challenging for evaluations on the full ImageNet with 1000 classes due to the expensive costs of training thousands of concept models and the lack of knowledge rules at the current stage, which is also discussed as one limitation in Appendix A. However, we deem that the principles and analysis in COLEP shed light on the effectiveness of the reasoning component to achieve a more robust conformal prediction. It is also interesting for future work to leverage the CLIP model for concept detection and the large language models for the design of knowledge rules, which can fully unleash the power of COLEP on more large-scale evaluations.

Table 1. Marginal coverage / average set size of multiple conformal prediction baselines, SOTA method RSCP, and COLEP under PGD Attack and AutoAttack with $\ell_2$ bounded perturbations $0.25$ and $0.50$ on AwA2. The nominal level of coverage is $0.9$.

|  | APS | RAPS | CCP | RSCP | COLEP |
| --- | ----------- | ----------- | ----------- | ----------- | ----------- | 
| $\delta=0.25$ | 0.8325 / 3.22 | 0.8363 / 3.20 | 0.8427 / 3.53 | 0.945 / 6.41 | 0.962 / 4.74 |
| $\delta=0.50$ | 0.7829 / 2.83 | 0.7821 / 2.89 | 0.7894 / 2.93 | 0.932 / 5.83 | 0.942 / 4.42 |

*[1] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

*[2] Angelopoulos, Anastasios, et al. "Uncertainty sets for image classifiers using conformal prediction." ICLR 2021.*

*[3] Ding, Tiffany, et al. "Class-Conditional Conformal Prediction With Many Classes." NeurIPS 2023.*

*[4] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." TPAMI 2018.*

*[5] Yang, Zhuolin. Improving certified robustness via statistical learning with logical reasoning. NeurIPS 2022.*

*[6] Wu, Tailin, et al. "Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time." NeurIPS 2022.*

Reviewer comment (Writing):
The overbearing nature of the notation used, and the overall presentation, is a weakness. It took many hours to internalize the type of guarantees that learning-reasoning components lead to, and even the notation itself. Moreover, especially the multi-subscript-superscript notation involved in both theorem statements and proofs related to the learning-reasoning component was very hard to parse and even read.

Author response:
We thank you for your valuable comments and suggestions. We make the following updates to enhance the reading of the paper.

1.  We include more comprehensive and structured notation review tables in Appendix K for a better understanding of the analysis of COLEP. Specifically, we provide an overview of notations related to model definitions in Table 5, notations related to conformal prediction in Table 6, notations related to certification of Section 5 in Table 7, and notations related to analysis of Section 6 in Table 8. All the used notations in COLEP are currently provided in the notation tables in a structured manner, which is emphasized at the beginning of Section 4. In this way, readers can look up the meanings of notations easily and quickly via the tables.

2. We also improved the overall certification flow in Figure 4 in Appendix E. We restructured the flow and included more illustrations to improve the readability. We also added a brief overview description of the certification flow for a better understanding. The certification setting is that the inference time adversaries can add bounded perturbations to the test sample and violate the data exchangeability assumption, compromising the guaranteed coverage. The certification generally achieves the following three goals. (1) We can preserve the guaranteed coverage using a prediction set that takes the perturbation bound into account (in Theorem 2), achieved by computing the probability bound of models before the reasoning component (by randomized smoothing) and the bound after the reasoning component (by Theorem 1). (2) We prove the worst-case coverage (a lower bound) if we use the standard prediction set as before (in Theorem 3). (3) We theoretically show that COLEP can achieve better prediction coverage (in Theorem 4) and prediction accuracy (in Theorem 5) than a data-driven model without the reasoning component. 

3. We follow the suggestion to separate important definitions from the paragraphs for ease of backtracking the original definitions. Concretely, we separate the definitions of $\hat{\pi}_j(x)$ and $\hat{\pi}^{(l)}(x)$ in Section 4.1 and the definition of $F(\mu)$ in Section 4.2.

We believe that these enhancements will make it simpler for readers to understand the provided analysis.

Reviewer comment (Writing):
The notational review in the last appendix was a nice touch, but only marginally helpful as it only listed some of the relevant notation.

Author response:
We thank you for your valuable comments and suggestions. We make the following updates to enhance the reading of the paper.

1.  We include more comprehensive and structured notation review tables in Appendix K for a better understanding of the analysis of COLEP. Specifically, we provide an overview of notations related to model definitions in Table 5, notations related to conformal prediction in Table 6, notations related to certification of Section 5 in Table 7, and notations related to analysis of Section 6 in Table 8. All the used notations in COLEP are currently provided in the notation tables in a structured manner, which is emphasized at the beginning of Section 4. In this way, readers can look up the meanings of notations easily and quickly via the tables.

2. We also improved the overall certification flow in Figure 4 in Appendix E. We restructured the flow and included more illustrations to improve the readability. We also added a brief overview description of the certification flow for a better understanding. The certification setting is that the inference time adversaries can add bounded perturbations to the test sample and violate the data exchangeability assumption, compromising the guaranteed coverage. The certification generally achieves the following three goals. (1) We can preserve the guaranteed coverage using a prediction set that takes the perturbation bound into account (in Theorem 2), achieved by computing the probability bound of models before the reasoning component (by randomized smoothing) and the bound after the reasoning component (by Theorem 1). (2) We prove the worst-case coverage (a lower bound) if we use the standard prediction set as before (in Theorem 3). (3) We theoretically show that COLEP can achieve better prediction coverage (in Theorem 4) and prediction accuracy (in Theorem 5) than a data-driven model without the reasoning component. 

3. We follow the suggestion to separate important definitions from the paragraphs for ease of backtracking the original definitions. Concretely, we separate the definitions of $\hat{\pi}_j(x)$ and $\hat{\pi}^{(l)}(x)$ in Section 4.1 and the definition of $F(\mu)$ in Section 4.2.

We believe that these enhancements will make it simpler for readers to understand the provided analysis.

Reviewer comment (Presentation):
Diagram in Figure 4, describing the overall certification flow, felt confusing and didn't really help, making the reviewer resort to understanding text only.

Author response:
We thank you for your valuable comments and suggestions. We make the following updates to enhance the reading of the paper.

1.  We include more comprehensive and structured notation review tables in Appendix K for a better understanding of the analysis of COLEP. Specifically, we provide an overview of notations related to model definitions in Table 5, notations related to conformal prediction in Table 6, notations related to certification of Section 5 in Table 7, and notations related to analysis of Section 6 in Table 8. All the used notations in COLEP are currently provided in the notation tables in a structured manner, which is emphasized at the beginning of Section 4. In this way, readers can look up the meanings of notations easily and quickly via the tables.

2. We also improved the overall certification flow in Figure 4 in Appendix E. We restructured the flow and included more illustrations to improve the readability. We also added a brief overview description of the certification flow for a better understanding. The certification setting is that the inference time adversaries can add bounded perturbations to the test sample and violate the data exchangeability assumption, compromising the guaranteed coverage. The certification generally achieves the following three goals. (1) We can preserve the guaranteed coverage using a prediction set that takes the perturbation bound into account (in Theorem 2), achieved by computing the probability bound of models before the reasoning component (by randomized smoothing) and the bound after the reasoning component (by Theorem 1). (2) We prove the worst-case coverage (a lower bound) if we use the standard prediction set as before (in Theorem 3). (3) We theoretically show that COLEP can achieve better prediction coverage (in Theorem 4) and prediction accuracy (in Theorem 5) than a data-driven model without the reasoning component. 

3. We follow the suggestion to separate important definitions from the paragraphs for ease of backtracking the original definitions. Concretely, we separate the definitions of $\hat{\pi}_j(x)$ and $\hat{\pi}^{(l)}(x)$ in Section 4.1 and the definition of $F(\mu)$ in Section 4.2.

We believe that these enhancements will make it simpler for readers to understand the provided analysis.

Reviewer comment (Presentation):
Instead of structured definitions of the main notations and concepts (using appropriate LaTeX environments), everything was crammed into the paragraphs, thus diluting the structure of the presentation and making it hard to backtrack to the context in which a piece of notation was originally defined.

Author response:
We thank you for your valuable comments and suggestions. We make the following updates to enhance the reading of the paper.

1.  We include more comprehensive and structured notation review tables in Appendix K for a better understanding of the analysis of COLEP. Specifically, we provide an overview of notations related to model definitions in Table 5, notations related to conformal prediction in Table 6, notations related to certification of Section 5 in Table 7, and notations related to analysis of Section 6 in Table 8. All the used notations in COLEP are currently provided in the notation tables in a structured manner, which is emphasized at the beginning of Section 4. In this way, readers can look up the meanings of notations easily and quickly via the tables.

2. We also improved the overall certification flow in Figure 4 in Appendix E. We restructured the flow and included more illustrations to improve the readability. We also added a brief overview description of the certification flow for a better understanding. The certification setting is that the inference time adversaries can add bounded perturbations to the test sample and violate the data exchangeability assumption, compromising the guaranteed coverage. The certification generally achieves the following three goals. (1) We can preserve the guaranteed coverage using a prediction set that takes the perturbation bound into account (in Theorem 2), achieved by computing the probability bound of models before the reasoning component (by randomized smoothing) and the bound after the reasoning component (by Theorem 1). (2) We prove the worst-case coverage (a lower bound) if we use the standard prediction set as before (in Theorem 3). (3) We theoretically show that COLEP can achieve better prediction coverage (in Theorem 4) and prediction accuracy (in Theorem 5) than a data-driven model without the reasoning component. 

3. We follow the suggestion to separate important definitions from the paragraphs for ease of backtracking the original definitions. Concretely, we separate the definitions of $\hat{\pi}_j(x)$ and $\hat{\pi}^{(l)}(x)$ in Section 4.1 and the definition of $F(\mu)$ in Section 4.2.

We believe that these enhancements will make it simpler for readers to understand the provided analysis.

Reviewer comment (Experiments):
The experimental section needs strengthening; please see above for suggestions/guidelines on directions.

Author response:
We thank you for your valuable comments and suggestions on the evaluation part. We made the following updates to clarify and strengthen the evaluation of COLEP.

1. We avoided the confusion induced by improper namings in Section 7. APS score is indeed not the SOTA method, and thus, we called it a baseline of conformal prediction and only called RSCP [1] as the SOTA approach in adversarially robust conformal prediction in the revised version.

2. We followed the suggestion to compare with more baselines of various conformal prediction scores to show the robustness of COLEP. Specifically, we further evaluate the RAPS score [2], which improves the APS score by adding a regularization term regarding the ranking of the ground truth label. We also evaluate the recent clustered conformal prediction (CCP) [3], which computes the quantile and performs calibration at the cluster level. We compare COLEP with these baselines on AwA2 under PGD attack. The results in the following Table 1 show that (1) conformal prediction with vanilla conformity scores APS, RAPS, and CCP leads to a broken coverage in the adversary setting, while adversarially robust conformal prediction method RSCP and COLEP still maintains the valid coverage, and (2) compared to RSCP, COLEP demonstrates a better tradeoff between the coverage and prediction efficiency.

3. We added the following clarifications and more evaluations to demonstrate the scalability of COLEP. Besides the relatively small-scale but standard benchmark CIFAR-10 and GTSRB, we also conducted evaluations on AwA2 [4] in different settings in Figure 2 and Figure 3 in Section 7. AwA2 dataset contains 37,322 images of 50 different animal classes, which are extracted from the ImageNet dataset and with a high resolution of $256 \times 256$. The dataset also has 85 attribute labels (concept labels) for each animal class and is suitable for evaluations of knowledge-intensive tasks. In total, we construct 187 implication rules in total with 4 types of knowledge encoded in 4 PCs separately. Evaluations on AwA2 are already the largest scale in the related literature on learning-reasoning [5] and neuro-symbolic [6]. The scalability of COLEP comes from the usage of probabilistic circuits with linear inference time with respect to the graph size instead of the intractable Markov logic network with NP-complete time complexity used by prior works [5,6]. To consolidate the scalability to AwA2, we further added more evaluations with multiple perturbation bounds in the following Table 1. The results together with results in Figure 2 and Figure 3 on AwA2 demonstrate the scalability and robustness of COLEP to achieve an impressive prediction coverage and efficiency in the adversary setting. 

We also admit that it is challenging for evaluations on the full ImageNet with 1000 classes due to the expensive costs of training thousands of concept models and the lack of knowledge rules at the current stage, which is also discussed as one limitation in Appendix A. However, we deem that the principles and analysis in COLEP shed light on the effectiveness of the reasoning component to achieve a more robust conformal prediction. It is also interesting for future work to leverage the CLIP model for concept detection and the large language models for the design of knowledge rules, which can fully unleash the power of COLEP on more large-scale evaluations.

Table 1. Marginal coverage / average set size of multiple conformal prediction baselines, SOTA method RSCP, and COLEP under PGD Attack and AutoAttack with $\ell_2$ bounded perturbations $0.25$ and $0.50$ on AwA2. The nominal level of coverage is $0.9$.

|  | APS | RAPS | CCP | RSCP | COLEP |
| --- | ----------- | ----------- | ----------- | ----------- | ----------- | 
| $\delta=0.25$ | 0.8325 / 3.22 | 0.8363 / 3.20 | 0.8427 / 3.53 | 0.945 / 6.41 | 0.962 / 4.74 |
| $\delta=0.50$ | 0.7829 / 2.83 | 0.7821 / 2.89 | 0.7894 / 2.93 | 0.932 / 5.83 | 0.942 / 4.42 |

*[1] Gendler, Asaf, et al. "Adversarially robust conformal prediction." ICLR 2021.*

*[2] Angelopoulos, Anastasios, et al. "Uncertainty sets for image classifiers using conformal prediction." ICLR 2021.*

*[3] Ding, Tiffany, et al. "Class-Conditional Conformal Prediction With Many Classes." NeurIPS 2023.*

*[4] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." TPAMI 2018.*

*[5] Yang, Zhuolin. Improving certified robustness via statistical learning with logical reasoning. NeurIPS 2022.*

*[6] Wu, Tailin, et al. "Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time." NeurIPS 2022.*

Reviewer comment (Theory):
Is the F(mu) prefixed function with H rules? Is mu observed only for the training? (the knowledge graph encodes the relationship between, e.g., "stop" and shape info, but the goal is to predict if it is a stop sign). I am confused about how pi_j(x) is calculated when mu is unknown.

Author response:
> Q1: More clarifications about assignment $\mu$, factor function value $F(\mu)$, and the computation of $\pi^{\text{COLEP}}_j(x)$ accordingly.

We included the following clarification of definitions in Section 4.2. $F(\mu)$ is indeed a fixed factor function once $H$ knowledge rules and weights are determined. Note that $\mu$ is an **index variable** of the summation process in Equation (6) (instead of observation), and thus we do not need to observe the values of $\mu$. For a particular index vector $\mu$ in the summation process, $F(\mu)$ computes the summation of the weights of satisfied knowledge rules, denoting how well the index vector $\mu$ aligns with the specified knowledge rules. 

Let’s consider the illustrative example of stop sign classification model with an auxiliary knowledge model of octagon shape classification in Section 4.2. The index variable $\mu \in \\{0,1\\}^2$ denotes a possible assignment of Bernoulli random variables $\text{IsStopSign}$ and $\text{IsOctagon}$. 
Therefore, we have that $F(\mu=[0,0])=F(\mu=[0,1])=F(\mu=[1,1])=e^w$ and $F(\mu=[1,0])=e^0$ since only the assignment $\mu=[1,0]$ (i.e., $\text{IsStopSign}=1, \text{IsOctagon}=0$) violates the knowledge rule $\text{IsStopSign} \rightarrow \text{IsOctagon}$ with weight $w$. Then according to Equation (6), we can compute $\pi_j^{\text{COLEP}}(x)$ (i.e., $p(\text{IsStopSign}=1)$) by marginalizing the Bernoulli variable $\text{IsOctagon}$ as follows:
$$ \pi_j^{\text{COLEP}}(x) = \dfrac{p(\text{IsStopSign}=1)p(\text{IsOctagon}=0)F(\mu=[1,0]) +  p(\text{IsStopSign}=1)p(\text{IsOctagon}=1)F(\mu=[1,1])}{p(\text{IsStopSign}=1)p(\text{IsOctagon}=0)F(\mu=[1,0]) + p(\text{IsStopSign}=1)p(\text{IsOctagon}=1)F(\mu=[1,1]) + p(\text{IsStopSign}=0)p(\text{IsOctagon}=0)F(\mu=[0,0]) + p(\text{IsStopSign}=0)p(\text{IsOctagon}=1)F(\mu=[0,1])} $$, where $p(\cdot)$ is the likelihood estimate of the main model and knowledge models without the reasoning component.

Reviewer comment (Experiments):
It seems that the gain of COLEP originates from utilizing the prior knowledge graph through F(u) and u: The knowledge graphs incorporated are side information with both the graph relationship and u remained true against attacks. How robust does the method perform against contaminated graphs/u?

Author response:
> Q2: Discussion of the robustness of COLEP with contaminated knowledge graph.

We thank you for the interesting question! It is interesting to consider cases where the knowledge graph is contaminated by adversarial attackers or contains noises and misinformation by the neglect of designers. According to the response to Q1, $\mu$ is an index variable instead of observations, and thus, the robustness of COLEP mainly originates from the knowledge rules via factor function $F(\cdot)$ denoting how well each assignment conforms to the specified knowledge rules. Since the knowledge rules and weights are specified by practitioners and fixed during inferences, the PCs encoding the knowledge rules together with the main model and knowledge models (learning + reasoning component in Figure 1) can be viewed as a complete model. According to the adversarial attack settings in the literature[1,2], the model weights can be well protected and not manipulated by attackers in practical cases, and thus, the PC graph (i.e., source of side information) which is a part of the complete model can not be easily manipulated by adversarial attackers.

On the other hand, it is an interesting question to discuss the case when the knowledge rules may be contaminated and contain misinformation due to the neglect of designers. We can view the problem of checking knowledge rules and correcting misinformation as a pre-processing step before the employment of COLEP. The parallel module of knowledge checking and correction is explored by a line of research [3,4,5], which basically detects and corrects false logical relations via semantic embeddings and consistency-checking techniques. It is also interesting to leverage more advanced large language models to check and correct human-designed knowledge rules. We deem that the process of knowledge-checking is independent and parallel to COLEP, which may take additional efforts before the deployment of COLEP but significantly benefits in achieving a much more robust system based on our analysis and evaluations. We added the related discussions in Appendix A. 


*[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).*

*[2] Gendler, Asaf, et al. "Adversarially robust conformal prediction." International Conference on Learning Representations. 2021.*

*[3] Melo, André, and Heiko Paulheim. "An approach to correction of erroneous links in knowledge graphs." CEUR Workshop Proceedings. Vol. 2065. RWTH Aachen, 2017.*

*[4] Caminhas, Daniel D. "Detecting and correcting typing errors in open-domain knowledge graphs using semantic representation of entities." (2019).*

*[5] Chen, Jiaoyan, et al. "Correcting knowledge base assertions." Proceedings of the Web Conference 2020.*

Reviewer comment (Theory):
Is it easy to calculate max|η|2≤δ ˆ πj (x + η) and min|η|2≤δ ˆ πj (x + η) for general probability assignment function pi_j(.)? The two main Theorems are from a brute-forth search of the worst-case scenario under perturbations, which are intuitively correct but it seems more important to show the feasibility of achieving this for general functions.

Author response:
> Q3: Computation of the bound of class probabilities for more general functions.

As illustrated in Section 5.1, the computation of the bound of class probabilities consists of two steps: 
- (1) bound computation of class probabilities after the learning component (i.e., $\min\_{\| \eta \|_2 \le \delta} \pi\_j(x+\eta)$ and $\max\_{\| \eta \|_2 \le \delta} \pi\_j(x+\eta)$)
- (2) bound computation of class probabilities after the reasoning component (i.e., $\min\_{\|\eta\|\_2 \le \delta} \pi\_j^{\text{COLEP}}(x+\eta)$ and $\max\_{\|\eta\|\_2 \le \delta} \pi\_j^{\text{COLEP}}(x+\eta)$). 

We can compute the bound in step (1) for any general functions $\pi_j$ with randomized smoothing. We can compute the bound in step (2) based on the results of step (1) according to Theorem 1, which holds for general probabilistic circuits that can encode arbitrary logic rules. Therefore, COLEP allows for bound computation for general deep neural networks in the learning component and general probabilistic circuits in the reasoning component.

Reviewer comment (Reproducibility):
How are the weights w chosen in F(mu)?

Author response:
> Q4: Selection of weight $w$ in $F(\mu)$.

We thank you for the interesting question. The weight $w$ of logical rules is specified based on the utility of the corresponding knowledge rule. In Section 6, we theoretically show that the robustness of COLEP correlates with the utility of knowledge rules regarding the uniqueness of the rule, defined in Equation (17). Specifically, a universal knowledge rule which generally holds for almost all class labels is of low quality and does not benefit the robustness and should be assigned with a low weight $w$. For example, if all the traffic signs in the dataset are of octagon shape, then the implication rule $\text{IsStopSign} \implies \text{IsOctagon}$ should be useless and assigned with a low weight $w$, which also aligns with the intuition. 

In the empirical evaluations, since we already designed knowledge rules of high quality (i.e., good uniqueness) as shown in Section 7, we do not need to spend great efforts selecting weights for each knowledge rule individually. We assume that all the knowledge rules have the same weight $w$ and select a proper weight $w$ for all the knowledge rules via grid search. The results in the following Table 1 inspire us to fix $w$ as $1.5$, and the evaluations demonstrate that it is sufficient to achieve good robustness in different settings. One can also model the weight selection problem as a combinatorial optimization, which is non-trivial but can better unleash the power of COLEP. We leave a more intelligent and automatic approach of knowledge weight selection for future work. We included the discussions in Section 7 and Appendix J.

Table 1. Certified coverage of COLEP on GTSRB with different weights $w$ of knowledge rules under perturbation bound $\delta=0.25$.

| w | 0.0 | 0.5 | 1.0 | 1.5 | 2.0 | 3.0 | 5.0 | 10.0 |
| --- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Certified Coverage | 0.816 | 0.832 | 0.837 | **0.842**  | 0.840 | 0.823 | 0.829 | 0.827 |

Reviewer comment (Writing):
The reasoning part seems to be missing significant details to understand how the PCs are working together to reduce the error produced by the main classifier. The paper makes a strong assumption that the reader will be familiar with semantics of PC structures. There should be some brief introduction to these models.

Author response:
We thank you for the valuable comment. We included the following details about the PC structure and its benefits to the robustness in Section 4.2 in the revised manuscript.

PC defines a joint distribution over a set of random variables (e.g., $\text{IsStopSign}$, $\text{IsOctagon}$) and computes the joint probability of them in the graph efficiently by performing forwarding passes. We can encode the knowledge rules with a 3-layer PC as shown in Figure 1: (1) the bottom layer consists of leaf nodes representing Bernoulli variables; (2) the second layer consists of product nodes computing the joint probability of an assignment by multiplying the likelihood of individual Bernoulli variables and the factor values (e.g., $\hat{\pi}_{s} \times \hat{\pi}^{o} \times F([1,1])$); and (c) the top layer is a sum node computing the summation of the products. With the PC structure, one can efficiently compute the marginal probabilities in Equation (6) with two forwarding passes. The details and proofs are shown in Appendix C.1.

The reasoning component can correct the output probability with encoded knowledge rules and improve the robustness of the prediction. Consider the following concrete example. Suppose that the adversarial example of a speed limit sign misleads the main classifier to detect it as a stop sign with a large probability (e.g., $\hat{\pi}_s=0.9$), but the octagon classifier is not misled and correctly predicts that the speed limit sign does not have an octagon shape (e.g., $\hat{\pi}_o=0.0$). Then according the computation as Equation (6), the corrected probability of detecting a stop sign given the attacked speed limit image is $\hat{\pi}^{\text{COLEP}}_s=0.9/(0.1e^w+0.9)$, which is down-weighted from the original wrong prediction $\hat{\pi}_s=0.9$ as we have a positive logical rule weight $w>0$. Therefore, the reasoning component can correct the wrong output probabilities with the knowledge rules and lead to a more robust prediction framework. Besides the intuition reflected by the simple example, we also theoretically show that the reasoning component benefits in achieving better prediction coverage and accuracy compared to a single DNN without the reasoning component.

Reviewer comment (Reproducibility):
The leaf weights (factor weights) are assumed to be prespecified and then there are also the Bernoulli parameters estimated by the neural networks. It is unclear how these estimated parameters, predictions (classes and concepts) and user defined knowledge rules are combined together to make robust decisions. The coefficients of the component PCs $\beta_k$ should be more clearly specified in the main manuscript.

Author response:
Thank you for the suggestion! We incorporated more specifications for PC coefficients $\beta_r$ in the main manuscript in Section 4.2. Concretely, we can formulate the corrected conditional class probability $\pi_j^{\text{COLEP}}$ with multiple PCs as $\pi_j^{\text{COLEP}}(x) = \sum_{r \in [R]} \beta_{r} \pi_j^{(r)}(x)$, where $\pi_j^{(r)}$ is the output class probabilities of the $r$-th PC. The core of this formulation is the mixture model involving a latent variable $r_{\text{PC}}$ representing the PCs. In short, we can write $\pi_j^{(r)}(x)$ as $\pi_j^{(r)}(x)=\mathbb{P}[Y=y|X=x, r_{\text{PC}}=r]$, and $\pi_j(x)$ as the marginalized probability over the latent variable $r_{\text{PC}}$ as $\pi_j(x)=\sum_{r \in [R]} \mathbb{P}[r_{\text{PC}}=r] \cdot \pi_j^{(r)}(x)$. Hence, the coefficient $\beta_{r}$ for the $r$-th PC are determined by $\mathbb{P}[r_{\text{PC}}=r]$. Although we lack direct knowledge of this probability, we can estimate it using the data by examining how frequently each PC $r$ correctly predicts the outcome across the given examples, similarly as in the estimation of prior class probabilities for Naive Bayes classifiers.

Reviewer comment (Theory):
The theorems in the main paper need a more detailed discussion. Having only a short paragraph for each theorem doesn't fully explain the complexities and nuances involved, which doesn't give the paper's theoretical contributions their due.

Author response:
We thank you for the valuable suggestion. We added more details in the remarks of theorems in the revised manuscript. Concretely, we made the following additions: 

- updated details of how we can get the probability bounds of the learning component and how we leverage them for computations of the bounds after the reasoning component in remarks of Theorem 1 

- details of the meanings of different terms such as certified prediction set and worst-case non-conformity score and the corresponding references to the equations in the remark of Theorem 2

- illustrations of the technical difference between Theorem 3 and Theorem 2 in the remark of Theorem 3 for a better understanding

- discussions of the physical meanings of the quantities affecting the bounds such as the utility of models $T_{j,\mathcal{D}}^{(r)},Z_{j,\mathcal{D}}^{(r)}$ and the utility of knowledge rules $U_j^{(r)}$ in the remark of Theorem 4 and the practical insights implied by the results

- discussions of the influence of the utility of models $T_{j,\mathcal{D}}^{(r)},Z_{j,\mathcal{D}}^{(r)}$ and the utility of knowledge rules $U_j^{(r)}$ on the prediction accuracy of COLEP and the practical implications in the remark of Theorem 5

Reviewer comment (Experiments):
To make the empirical evaluation stronger, the authors could consider using more datasets or testing against different types of attacks. This will help confirm that COLEP is robust and works well in various situations.

Author response:
We thank you for the valuable suggestions. To evaluate the robustness of COLEP under various types of attacks, we further consider AutoAttack [1], which uses an ensemble of four diverse attacks to reliably evaluate the robustness: (1) APGD-CE attack, (2) APGD-DLR attack, (3) FAB attack, and (4) square attack. APGD-CE and APGD-DLR attacks are step-size-free versions of PGD attacks with cross-entropy loss and DLR loss, respectively. FAB attack optimizes the adversarial sample by minimizing the norm of the adversarial perturbations. Square attack is a query-efficient black-box attack. We evaluate the prediction coverage and averaged set size of COLEP and SOTA method RSCP with $\ell_2$ bounded perturbation of $0.25$ on GTSRB, CIFAR-10, and AwA2. The desired coverage is $1-\alpha=0.9$ in the evaluation. The results in the following Table 1 show that under diverse and stronger adversarial attacks, COLEP still achieves higher marginal coverage than the nominal level $0.9$ and also demonstrates a better prediction efficiency (i.e., smaller size of prediction sets) than RSCP. We incorporated the results in Appendix J.2 and referred to it in Section 7 in the main text.

Table 1. Marginal coverage / average set size of RSCP and COLEP under AutoAttack [1] with $\ell_2$ bounded perturbation of $0.25$ on GTSRB, CIFAR-10, and AwA2.

| | GTSRB | CIFAR-10 | AwA2 |
| --- | ----------- | ----------- | -- |
| RSCP | 0.9682 / 2.36 | 0.9102 / 3.56 | 0.9230 / 5.98  |
| COLEP | **0.9702** / **2.13** | **0.9232** / **2.55** | **0.9520** / **4.57** |

*[1] Croce, Francesco, and Matthias Hein. "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." International conference on machine learning. PMLR, 2020.*

Reviewer comment (Experiments):
How many PCs were considered in each of the experiments? Can the authors give an idea on the complexity of the PCs?

Author response:
We thank you for the valuable question. In summary, we have 3 PCs and 28 knowledge rules in GTSRB, 3 PCs and 30 knowledge rules in CIFAR-10, and 4 PCs and 187 knowledge rules in AwA2. Based on the domain knowledge in each dataset, we encode a set of implication rules into PCs. In each PC, we encode a type of implication knowledge with disjoint attributes. In GTSRB, we have a PC of shape knowledge (e.g., “octagon'', “square''), a PC of boundary color knowledge (e.g., “red boundary'', “black boundary''), and a PC of content knowledge (e.g., “digit 50'', “Turn Left''). In CIFAR-10, we have a PC of category knowledge (e.g., “animal'', “transportation''), a PC of space knowledge (e.g., “in the sky'', “in the water'', “on the ground''), and a PC of feature knowledge (e.g., “has legs'', “has wheels''). In AwA2, we have a PC of superclass knowledge (e.g., “procyonid'', “big cat''), a PC of predation knowledge (e.g., “fish'', “meat'', “plant''), a PC of appearance (e.g., “big ears'', “small eyes''), and a PC of fur color knowledge (e.g., “green'', “gray'', “white''). For example, in the PC of shape knowledge in GTSRB, we can encode the implication rules related to the shape knowledge such as $\text{IsStopSign} \implies \text{IsOctagon}$ (stop signs are octagon), $\text{IsSpeedLimit} \implies \text{IsSquare}$ (speed limit signs are square), and  $\text{IsTurnLeft} \implies \text{IsRound}$ (turn left signs are round). We included the statistics of PCs and knowledge rules in Section 7 and more details in Appendix J.1 in the revised manuscript.

Reviewer comment (Experiments):
Can the scope of the evaluation be extended to include the performance of COLEP against other forms of adversarial attacks?

Author response:
We thank you for the valuable suggestions. To evaluate the robustness of COLEP under various types of attacks, we further consider AutoAttack [1], which uses an ensemble of four diverse attacks to reliably evaluate the robustness: (1) APGD-CE attack, (2) APGD-DLR attack, (3) FAB attack, and (4) square attack. APGD-CE and APGD-DLR attacks are step-size-free versions of PGD attacks with cross-entropy loss and DLR loss, respectively. FAB attack optimizes the adversarial sample by minimizing the norm of the adversarial perturbations. Square attack is a query-efficient black-box attack. We evaluate the prediction coverage and averaged set size of COLEP and SOTA method RSCP with $\ell_2$ bounded perturbation of $0.25$ on GTSRB, CIFAR-10, and AwA2. The desired coverage is $1-\alpha=0.9$ in the evaluation. The results in the following Table 1 show that under diverse and stronger adversarial attacks, COLEP still achieves higher marginal coverage than the nominal level $0.9$ and also demonstrates a better prediction efficiency (i.e., smaller size of prediction sets) than RSCP. We incorporated the results in Appendix J.2 and referred to it in Section 7 in the main text.

Table 1. Marginal coverage / average set size of RSCP and COLEP under AutoAttack [1] with $\ell_2$ bounded perturbation of $0.25$ on GTSRB, CIFAR-10, and AwA2.

| | GTSRB | CIFAR-10 | AwA2 |
| --- | ----------- | ----------- | -- |
| RSCP | 0.9682 / 2.36 | 0.9102 / 3.56 | 0.9230 / 5.98  |
| COLEP | **0.9702** / **2.13** | **0.9232** / **2.55** | **0.9520** / **4.57** |

*[1] Croce, Francesco, and Matthias Hein. "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." International conference on machine learning. PMLR, 2020.*

Reviewer comment (Experiments):
Why wasn't the effect of varying the parameter $\alpha$ and $\rho$ on COLEP's performance analyzed?

Author response:
We thank you for the valuable suggestions. We further evaluate COLEP with multiple nominal levels of coverage $1-\alpha$ and with various perturbation bounds $\delta$. We provide the results with multiple coverage levels $1-\alpha$ in the following Table 2, and the results with multiple perturbation bounds $\delta$ in Table 3. The results demonstrate that for different parameters of $\alpha$ and $\delta$, (1) COLEP always achieves a higher marginal coverage than the nominal level, indicating the validity of the certification in COLEP, and (2) COLEP shows a better trade-off between the prediction coverage and prediction efficiency (i.e., average set size) than the SOTA baseline RSCP. We incorporated the results in Appendix J.2 and referred to it in Section 7 in the main text.

Table 2. Marginal coverage / average set size of RSCP and COLEP under PGD Attack with multiple desired coverage levels $1-\alpha$ with $\ell_2$ bounded perturbation of $0.25$ on GTSRB.

|  | $1-\alpha=0.85$ | $1-\alpha=0.9$ | $1-\alpha=0.95$ |
| --- | ----------- | ----------- | -- |
| RSCP | 0.9029 / 1.63 | 0.9682 / 2.36 | 0.9829 / 2.96  |
| COLEP | **0.9053** / **1.34** | **0.9702** / **2.13** | **0.9859** / **2.53** |

Table 3. Marginal coverage / average set size of RSCP and COLEP under PGD Attack with multiple $\ell_2$ bounded perturbations on GTSRB. The nominal level of coverage is $0.9$.

| | $\delta=0.125$ | $\delta=0.25$ | $\delta=0.5$ |
| --- | ----------- | ----------- | -- |
| RSCP | 0.9320 / *1.34* | 0.9682 / 2.36 | 0.9712 / 2.78  |
| COLEP | **0.9891** / 1.47 | **0.9702** / **2.13** | **0.9780** / **2.69** |