Paper: Causal Unsupervised Semantic Segmentation (ICLR-2024)

Reviewer comment (Theory):
The causal diagram is not solid. Why the path $T \rightarrow Y$ could be omit?

Author response:
**A1.** We kindly argue that our causal diagram design is solid and it is highly related with unsupervised setup. We have not omitted $T \rightarrow Y$ pathway, this pathway means directly use the feature representation for semantic segmentation prediction, and it is equivalent with the vanilla DINO results in Table 1.

The vanilla DINO results show that naively using $T$ is sub-optimal performance, thus we point out that the unawareness of the clustering targets in unsupervised setup makes difficulties during the segmentation. 

Accordingly, our design of causal diagram defines with an explicit ***“Mediator $M$"*** between $T$ and $Y$ ($T \rightarrow M \rightarrow Y$) to handle the difficulties in unsupervised learning. We highlight again that $M$ generates concept prototypes as priors with concrete causal background, and provides explicit distance criteria for the pixel grouping.

Reviewer comment (Writing):
The specific explanation about $U$ is missing. It is not convincing to use question as definition.

Author response:
**A2.** We respectfully highlight again that we have clearly stated $U$ as ***"How can we define what to cluster and how to do so under an unsupervised setting?"*** in Introduction. As we described, the $U$ indicates the lack of awareness regarding what and how to cluster for each pixel representation, especially when aiming for the desired level of granularity. 

For further understanding, let us assume that we have a pre-trained feature representation for unsupervised segmentation (from DINO). The main problem for conducting USS is that we don’t know ***“what concepts have to be grouped and segmented for semantic segmentation”***. As an extension of your Q1, just maximizing correspondence “continuous” feature representation ($T \rightarrow Y$) directly provides sub-optimal performance because grouping feature representation in continuous embedding space is challenging tasks to conduct, especially in the case we don’t know any segmentation information to cluster ($U$: what \& how to cluster). We hope this additional explanation can address any misunderstanding regarding $U$.

Reviewer comment (Presentation):
Is there any instance or example for explaining Figure 2?

Author response:
**A4.**  ***Yes, in Figure 1!*** The other baselines in the Figure 1 directly maximize “continuous” feature representation, thus suffering low segmentation quality with respect for the targeted-level objects in the images. Note that this is provoked due to $U$ in unsupervised setup. 

As stated in the caption of Figure 1, for instance, in column 4 of the figure, other baselines are suffering from segment human head and whole body part and often assigning different unsupervised predictions for these elements. Notably, our approach successfully group the targeted *person* objects, mainly due to the consolidation of feature representations from small fractions to the targeted level, utilizing concept prototypes in the clusterbook.

Reviewer comment (Evaluation):
Is is feasible to direct evaulate the mIoU based on results of the concept with respect to the index on clusterBook?

Author response:
**A6.** We would like to kindly remind that this is ***"unsupervised"*** evaluation. The evaluation protocol should utilize annotations in the datasets to directly measure the quality of unsupervised feature representation. That is the reason all other unsupervised baselines also use matching procedure (*e.g.,* Hungarian Matching) and probing prediction head with stop gradients ***during evalutations***. As you asked, it is possible to measure mIoU scores with respect to the clusterbook index. However, we cannot derive any meaningful evaluation outcomes for the cluterbook index, which is not ***TRUE*** annotations for the datasets.

Reviewer comment (Reproducibility):
This paper lacks a detailed explanation of the specific methodologies used for constructing the concept clusterbook and conducting concept-wise self-supervised learning.

Author response:
**A1.** Based on Answer 1 and 2 of Reviewer 1Vow and Answer 1 and 3 of Reviewer zoBW, we will add technical details and insightful writings included in Appendix, once we take a potential next step (cam-ready) due to extra-one page for enough space.

However, in Appendix B, we have already described a concrete implementation for training details of CAUSE about (B.1) Maximizing Modularity, (B.2) Transformer-based Segmentation Head, (B.3) Anchor Selection, (B.4) Positive \& Negative Concept Selection, (B.5) Concept Bank, and (B.6) Image Resolution and Augmentation.

Especially, Appendix B.2 ***"Transformer-based Segmentation Head"*** has included the propagation detail of this segmentation head and which paper inspired to this segmentation head. Moreover, we did not use pre-trained model for segmentation head but train from scratch. We would like to kindly remind the paragraph in Appendix B.2.

Reviewer comment (Presentation):
This paper does not discuss the limitations or potential drawbacks of the proposed framework, which would have been useful for readers to understand the scope and applicability of the approach.

Author response:
**A2.** We would like to respectfully argue that we have described Appendix D ***"Discussion and Limitations"***. We kindly remind that as we stated in the last paragraph of *Categorical Analysis* of Sec 4.2: "We include further discussions and limitations in Appendix D".

Reviewer comment (Experiments):
The comparison with recent and state-of-the-art methods in unsupervised semantic segmentation is missing, which could provide a comprehensive evaluation of the proposed framework.

Author response:
**A3.** We respectfully disagree that we have already compared CAUSE with SOTA methods for unsupervised semantic segmentation on Papers-With-Code. To the best of our knowledge, we have comprehensively compared with whole existing unsupervised semantic segmentation baselines: MaskContrast (*ICCV 2022*), HSG (*CVPR 2022*), ReCo+ (*NeurIPS 2022*), STEGO (*ICLR 2022*), TranFGU (*ECCV 2022*), DINOSAUR (*ICLR 2023*), HP (*CVPR 2023*). Please note that compared with STEGO and HP, we even extend the comparison targets to object-centric unsupervised semantic segmentation in Pascal VOC 2012 and COCO-81 and compare CAUSE performance with DeepSpectral (*CVPR 2022*), Leopart (*CVPR 2022*), ACseg (*CVPR 2023*), COMUS (*ICLR 2023*).

Please reply specific missing references for the further comparison. We are willing to add them in the experiment tables in the main manuscript.

Reviewer comment (Evaluation):
Please discuss the limitations or potential drawbacks of the proposed framework. It would be helpful for readers to understand the scope and applicability of the approach.

Author response:
**A2.** We would like to respectfully argue that we have described Appendix D ***"Discussion and Limitations"***. We kindly remind that as we stated in the last paragraph of *Categorical Analysis* of Sec 4.2: "We include further discussions and limitations in Appendix D".

Reviewer comment (Reproducibility):
Please provide more details on the implementation, such as the specific architectures used for the segmentation head and the pre-trained model.

Author response:
**A1.** Based on Answer 1 and 2 of Reviewer 1Vow and Answer 1 and 3 of Reviewer zoBW, we will add technical details and insightful writings included in Appendix, once we take a potential next step (cam-ready) due to extra-one page for enough space.

However, in Appendix B, we have already described a concrete implementation for training details of CAUSE about (B.1) Maximizing Modularity, (B.2) Transformer-based Segmentation Head, (B.3) Anchor Selection, (B.4) Positive \& Negative Concept Selection, (B.5) Concept Bank, and (B.6) Image Resolution and Augmentation.

Especially, Appendix B.2 ***"Transformer-based Segmentation Head"*** has included the propagation detail of this segmentation head and which paper inspired to this segmentation head. Moreover, we did not use pre-trained model for segmentation head but train from scratch. We would like to kindly remind the paragraph in Appendix B.2.

Reviewer comment (Theory):
Please discuss the computational complexity of the proposed framework. Please provide any insights or analysis on the computational efficiency of the approach.

Author response:
**A4.** Borrowing Answer 2 for the Question 2 Reviewer zoBW: (***Calculation of affinity matrix is time consuming?***), we will discuss the computational complexity of two step procedures.

For Step 1, the potential computational burden can be thought of as calculation of affinity matrix over spatial resolution size in features. However, we calculate affinity matrix $\mathcal{A}$ in batch-wise manners. For example, we have pre-trained features $T\in\mathbb{R}^{b\times hw \times c}$ with batch number $b$. Then, its affinity matrix is nothing but $\mathcal{A}\in\mathbb{R}^{b \times hw \times hw}$. Even, we optimize the maximization of modularity through just ***"one epoch"*** with Adam optimizer, as we mentioned in the last paragraph of Sec 3.2. Furthermore, Appendix B.1, "Maximizing Modularity" has already described how to efficiently compute cluster assignment matrix using trace property. For our experiment environment (CPU: Intel(R) Xeon(R) Gold 6230R, GPU: RTX 3090 $\times$ 4EA, and RAM: 256GB), in the end, we take about within 10 min, 30 min, 45 min, 1 hour, and 10 min to perform modularity maximization on Cityscapes, COCO-Stuff, COCO-81, COCO-171, and Pascal-VOC 2012. Based on these computation time in modularity maximization, therefore, calculating affinity matrix and conducing modularity maximization requires negligible computation.

For Step 2, the computationally expensive part is line 7 in Algorithm 2, which calculates the likelihood of semantic groups $Y$ (*i.e.,* segmentation head features) to act concept-wise self-supervised learning. This is because, once we select all of anchor feature points $y$ in $Y =\lbrace y\in\mathbb{R}^{r}\rbrace^{hw}$, we should collect all positive and negative features corresponding to each anchor feature point of which the number is $hw$. For example, we have segmentation head features $Y\in\mathbb{R}^{40(h)\times 40(w) \times 90(r)}$ and assume that no matter which anchor feature point $y$ in $Y$ is selected, there exist the $100$ number of positive ($p$) or negative ($n$) feature points, respectively. Then, we must deal with the number $40(h)\times 40(w) \times \lbrace100 (p)+100 (n)\rbrace = 320,000$ of the combinations between anchor feature points, positive, and negative features. Furthermore, if we consider batch-wise computation with $16$ batches, the highly striking number $320,000\times 16 = 5,120,000$ of combinations are needed. To effectively avoid this desperate computation, we *stochastically sampling* only $6.25$% of all anchor feature points as already illustrated in Appendix B.3 ***"Anchor Selection"***. This efficiently reduce $5,120,000 \times \frac{6.25}{100}=320,000$ which is surprisingly equal to one batch computation despite using full $16$ batches. From our effort trying to much lesson surplus computation, we can reach 30-45 min, 2-3 hour, 3-4 hour, 4-5 hour, and 30-45 min for Cityscapes, COCO-Stuff, COCO-81, COCO-171, and Pascal VOC 2012 with total two or three epochs. Note that, Distributed Data Parallel (DDP) [R1] with $16$ batches per GPU and Mixed Precision (float32-float16) [R2] are together used to technically more reduce computation in this environment setting: CPU: Intel(R) Xeon(R) Gold 6230R, GPU: RTX 3090 $\times$ 4EA, and RAM: 256GB.

Beyond, the paragraph of *Implementation* in Sec 4.1 and Appendix B.2 ***"Transformer-based segmentation head"*** describe we use a single layer transformer-based segmentation head even with a single attention head. This is attributed to our primary intention making a simple yet effective model design for CAUSE. Furthermore, please see the attached code documents for full details and requirements.

---

**References**

* [R1] Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704 (2020).

* [R2] Mixed Precision Training. International Conference on Learning Representations. 2018.

Reviewer comment (Theory):
The link between the proposed method and the causal inference is not very clear. The authors do pay much attention on theory and formulation of frontdoor adjustment, however is there explicit link between it with the proposed pipeline? That is, between equation 1 and the algorithm 1 and 2.

Author response:
**Q1. The link between the proposed method and the causal inference is not very clear. The authors do pay much attention on theory and formulation of frontdoor adjustment, however is there explicit link between it with the proposed pipeline? that is between equation 1 and the algorithm 1 and 2.**

**A1.** We believe that the link between unsupervised semantic segmentation (USS) and causal inference is clearly explained in ***Causal Perspective on USS*** (Section 3.1) beyond the theoretical background. In Eq. (1), we would like to kindly remind that each underbraced step, denoted as Step 1 and Step 2, is directly correlated with Algorithm 1 and Algorithm 2, respectively.

That is, Step 1 of Eq. (1) corresponds to $p(m|t)$ in Eq. (2), and Step 2 of Eq. (1) corresponds to $p(Y|t',m)p(t')$ in Eq. (2). Increasing both probabilities gets the probability $p(Y| \text{do}(T))$ of frontdoor adjustment to be improved.

Here, Algorithm 1 is the procedure of finding concept prototypes $M$ in rich pre-trained features $T$, thus it is the efforts of improving $p(m|t)$ for all concepts $m \in M$ and all spatial feature points $t\in T$. Furthermore, Algorithm 2 is the procedure of enhancing the likelihood $p(Y|t', m)=\prod_{y\in Y}p(y| t', m)$, for which Eq. (4) represents $p(y| t', m)$ equals to the probability of Noise-Contrastive Estimation.

Furtheremore, the footprint writing in page number 5 explains the concrete reasoning why maximizing the likelihood for the given $M$ is directly leading to increasing causal effect between $T$ and $Y$.

Reviewer comment (Reproducibility):
Calculation of affinity matrix is time consuming? How many samples does it use while calculating the affinity matrix? Does the codebook update while training, or is it fixed based on pretrained DINO features? In addition, how to guarantee that the codebook spans different levels, how to decide the proper granularity?

Author response:
**Q2. Calculation of affinity matrix is time consuming? how many samples does it use while calculating the affinity matrix?**

**A2.** No. We calculate affinity matrix $\mathcal{A}$ in batch-wise manners. For example, we have pre-trained features $T\in\mathbb{R}^{b\times hw \times c}$ with batch number $b$. Then, its affinity matrix is nothing but $\mathcal{A}\in\mathbb{R}^{b \times hw \times hw}$. Even, we optimize the maximization of modularity through just ***one epoch*** with Adam optimizer, as we mentioned in the last paragraph of Sec 3.2. Furthermore, Appendix (B.1) ***"Maximizing Modularity"*** has already described how to efficiently compute modularity using trace property. For our experiment environment (CPU: Intel(R) Xeon(R) Gold 6230R, GPU: RTX 3090 $\times$ 4EA, and RAM: 256GB), in the end, we take about within 10 min, 30 min, 45 min, 1 hour, and 10 min to perform modularity maximization on Cityscapes, COCO-Stuff, COCO-81, COCO-171, and Pascal-VOC 2012. Based on these computation time in modularity maximization, therefore, calculating affinity matrix and modularity requires negligible computation.

---

**Q3. Does the codebook update while training? or fixed based on pretrained DINO features?**

**A3.** Concept clusterbook $M$ is trained only in Step 1 and fixed during Step 2. We would like to clarify its explicit mention in Sec 3.3.

---

**Q4. How to guarantee the the codebook spans different levels, how to decide the propoer granularity?**

**A4.** For the modularity maximization of pixel representation from DINO, the number of codebook ($k$) necessarily indicates the granularity level. For instance, let us assume that we have bunch of image samples of *person* with same background. In the extreme case ($k=2$), the codebook may include only two concepts, entire *person* and *background* concepts. On the contrary, with a larger $k$, the codebook would subdivide the *person* concept into subdivided components such as *head*, *torso*, *hands*, *legs*, and so forth. Here, determining the proper level of granularity is the most challenge in the context of ***"unsupervised"*** prediction. This is because somebody may want to group head and torso into upper body to divide person into upper body and lower body. Nobody knows the proper granularity until a criterion to grouping clusters is given (*i.e.,* label annotations).

Here, our insight to handle the unsupervised setup is on collecting a large number $k$ of implicit concepts in advance using an explicit mediator (clusterbook) to span a broad spectrum of concept prototypes (*i.e.,* granularity) in advance during Step 1, and we consolidate and separate the concepts into the coarse-level semantics by *controlling only two hyper-parameter positive $\phi^+$ and negative $\phi^-$ relaxation* based on the concept distances $\mathcal{D}_M$ during Step 2. In other words, if COCO-Stuff's ground truth is changed by someone who wants to adjust another granularity criteria to cluster, then we need to control the two hyper-parameter to fit the semantic level someone desires. We further remind that we have conducted concept retrieval analysis for the clusterbook in Appendix C.3 and ablation study for the number of $k$ in Figure 5.

Reviewer comment (Experiments):
STEGO has a similar mechanism of contrastive learning. Which module does the proposed method benefit more from — the codebook or the ST learning?

Author response:
**Q5. STEGO has similar mechanism of contrastive learning. Which module does the proposed method benefits more from? The codebook or the ST learning?**

**A5.** ***"Codebook!"*** We highlight again that our improvements are mainly on the explicit mediator design, "Concept ClusterBook". The definite evidence is empirical validation for the ablation study of the concept number $k$ in Fig. 5(d). If Concept ClusterBook $M$ does not highly impact to unsupervised semantic segmentation (USS) performance, then the dramatic degraded performance cannot happen with fewer number of concepts in Fig. 5(d). In other words, Concept Clusterbook is so important that USS performance highly depends on the number of concepts. 

Moreover, the main purpose of student-teacher structure is to acquire stably high performance despite with small batch because its ensemble capacity [R1] on Polyak-Ruppert averaging with an exponential decay [R2, R3] bootstraps the model performance effectively [R4]. That is, it is more effective when we cannot enlarge batch size due to task-specific dataloader burden or hardware resource limitation but should prepare a collection of various queue like [R5]. From this reason, we introduce a concept bank possibly including numerous concepts of out-batch features, instead of enlarging number of batch size (hardware constraint). In brief, Concept ClusterBook is highly dependent on high USS performance, and student-teacher learning strategy with Concept Bank by exponential averaging helps to bootstrap stable USS performance despite with few batches.

Reviewer comment (Presentation):
A minor issue is that Algorithm 1 should be placed near sec. 3.2 to prevent confusion.

Author response:
**Q6. A minor issue is that the ALGORITHM 1 should be placed near sec. 3.2 to prevent confusing**

**A6.** We appreciate your great comments to make our paper further understandable to readers. We will definitely modify the location of Algorithm 1 to make it near to Sec 3.2, and we will add technical details and insightful writings in Sec 3 based on Answer 1 and 3, once we take a potential next step (cam-ready) due to extra-one page for enough space. Further, we will add more discussion content about Answer 2, 4, and 5 in Sec 4.

Reviewer comment (Theory):
It is unclear how Eq. 4 is derived. Since this is probably the most important part of the algorithm, more explanations should be given.

Author response:
**Q1. It is unclear how Eq. 4 is derived. Since this is probably the most important part of the algorithm, more explanations should be given.**

**A1.** We would like to remind our writings ***"we proceed to enhance the likelihood $p(Y |t', m)$ for effectively clustering pixel-level semantics. ... However, we cannot directly compute this likelihood as in standard supervised learning, primarily because there are no available pixel annotations. Instead, we substitute the likelihood of unsupervised dense prediction to concept-wise self-supervised learning based on Noise-Contrastive Estimation (NCE) [R1]"***. in the paragraph of Concept-wise Self-supervised Learning at Sec 3.2.

In unsupervised setup, we adopt common contrastive learning for grouping pixel representation, which we have verified that this learning is closely related with the causal inference. In other words, we borrow the formulation of NCE to build the likelihood $p(Y |t', m)$ on our purpose: unsupervised dense prediction without any annotations, therefore its derivation is fundamentally based on empirical study of NCE [R1]. Here, $t'$ and $m$ are used to find positive $y^+$ and negative $y^-$ features for the given anchor features $y$. Please see further explanation in Answer 2.

---

**References**

* [R1] Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297–304. JMLR Workshop and Conference Proceedings, 2010.

Reviewer comment (Reproducibility):
There is confusion about how "find patch feature points in $T$ satisfying $\mathcal{D}_M[id_q,:]>\phi^+$" is implemented in practice. As far as understood, $\mathcal{D}_M$ only summarizes prototype-prototype similarities; it is unclear how to select features based on the aforementioned rule.

Author response:
**Q2. I am confused about how "find patch feature points in satisfying $\mathcal{D}_M[\text{id}_q, :]$" is implemented in practice. As far as I understand, only summarizes prototype-prototype similarities; then how to select features based on the aforementioned rule?**

**A2.** We first kindly remind that we have included comprehensive explanations of feature selections in Appendix B. Here, we add more intuitive explanation for the question. For example, we have a concept book $M$ with the number of four concepts. Then, we can compute the distance matrix $\mathcal{D}_{M} \in \mathbb{R}^{4\times 4}$.

Let us assume that the first row vector of the distance matrix is $\mathcal{D}_{M}[0,:] = [1, 0.02, 0.47, 0.18]$ which represents the concept-wise distance vector between the first concept (index=$0$) and all of the four concepts in $M$. 
When we set positive and negative relaxation to $\phi^+ = 0.3$ and $\phi^- = 0.1$, then the third concept (index=$2$) is considered as positive concept to the first concept (index=$0$) due to the value of $0.47>\phi^+$, and the second concept (index=$1$) gets to be negative concept to the first concept (index=$0$) due to the value of $0.02<\phi^-$. Next, once self-supervised DINO feature $T\in\mathbb{R}^{5(h)\times 5(w) \times 768(c)}$ is given and its spatial features $T[0, 0, :], T[2, 4, :], T[4, 1, :] \in\mathbb{R}^{768}$ are the closest to, respectively, the first concept (index=0), the second concept (index=1), and the third concept (index=2), then $T[0, 0, :]$ and $T[2, 4, :]$ are the negative feature to be far away from each other, and $T[0, 0, :]$ and $T[4, 1, :]$ are the positive feature to get clustered. Because DINO features are frozen so that we cannot learn DINO features, we instead deal with the segmentation head output features $y=Y[0, 0, :], y^+=Y[2, 4, :], y^-=Y[4, 1, :]$ to be separated or clustered,  by enhancing the likelihood of semantic groups: $p(y|t', m) = \frac{\exp(\text{cos}(y, y^+)/\tau)}{\exp(\text{cos}(y, y^+)/\tau) + \exp(\text{cos}(y, y^-)/\tau)}$.

Reviewer comment (Novelty):
The concept prototype bears high similarity with the work [ref1]. There is a question as to whether the sinkhorn-knopp algorithm used in [ref1] can be used here for prototype generation.

Author response:
**Q4. The concept prototype bears high similarity with the work [R2]. I am wondering whether the sinkhorn-knopp algorithm used in [R2] can be used here for prototype generation.**

**A4.** It is not feasible to use sinkhorn-knopp algorithm in our work. [R2] generates class-wise numerous prototypes using ***"labeled annotations"!*** based on *supervised training*, where Sinkhorn-Knopp algorithm is employed together with Cross-Entropy Loss (supervised loss) to discriminate which prototypes are matching to the class labels (*i.e.,* many-to-one matching). Here, the reason why Sinkhorn-Knopp is necessary in [R2] is not the purpose of generating prototypes themselves but of increasing stable matching between the generated prototypes and concepts. Instead, generating prototypes is rather related with Cross-Entropy Loss so that without Cross-Entropy Loss, Sinkhorn-Knopp will be not working due to the absence of pixel annotation's actual guidance to the clustering or separating. Instead, we would like to highlight that we have conducted ablation for other prototype generation methods in Table 4.

Reviewer comment (Evaluation):
From Fig. 4, it appears that the method works particularly well in object boundaries (very close to ground-truths for, e.g., persons). While this is impressive for unsupervised segmentation, there is curiosity about how the algorithm improves over other methods in boundary predictions.

Author response:
**Q5. From Fig. 4, it appears that the method works particularly well in object boundaries (very close to ground-truths for, e.g., persons). While this is impressive for unsupervised segmentation, I am curious how the algorithm improves over other methods in boundary predictions.**

**A5.** The primary goal for unsupervised semantic segmentation is grouping to coarse-level semantics for the pixel representation. We achieved SOTA results for USS task, and it is mainly due to ***mediator*** for explicitly quantized concepts based on the causal perspective. By comparing concept-wise distances for each explicit concepts, the boundary pixels can be considered more noticeable than inner pixel regions (high concept distance in boundary), which is machted with our purpose of course-level grouping. 

In addition, we also concurrently remind that we use common post-processing techniques such as Fully-Connected CRF [R3]. The sharpening is mainly attribute to such processing. It is worth to note that previous works: STEGO, HP, and ReCo+ equally have used CRF for a post-processing, of which original paper's Figure 1(e) shows significant boundary segmentation performance between "tree leafs". This CRF algorithm is commonly used in segmentation area such as DeepLab [R4] to improve qualitative results, thus we would like to note that the noticable quality in boundary is attributed to the CRF post processing. Please see the ablation results for the existence of CRF in Table 4. 

---

**References**

* [R3] Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems 24 (2011).

* [R4] Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs IEEE transactions on pattern analysis and machine intelligence 40.4 (2017): 834-848.

Reviewer comment (Reproducibility):
There is a lack of description of training details.

Author response:
**Q6. There lacks description of training details.**

**A6.** In Appendix B, we have already described a concrete implementation for training details of CAUSE about (B.1) Maximizing Modularity, (B.2) Transformer-based Segmentation Head, (B.3) Anchor Selection, (B.4) Positive \& Negative Concept Selection, (B.5) Concept Bank, and (B.6) Image Resolution and Augmentation.

Though, based on Answer 1 and 2, once we take potential next steps (cam-ready), we will add the explanation on one-extra page with a simple example to make readers more easily understandable to the training details of concept-wise self-supervised learning and selecting positive and negative concept in Sec 3.3.