Paper: CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images (ICLR-2024)

Reviewer comment (Experiments):
The number of datasets used for benchmarking is too small (2). [update: the authors have added the C17 dataset]

Author response:
We have now conducted an experiment using the CAMELYON17 [1] dataset, which consists of 1000 Whole Slide Images (WSI) of a similar type as the CAMELYON16 dataset. However, only half (500 WSI) of these images are labeled and accessible publicly. These images are expertly annotated by pathologic lymph node classification into pN-stage:
- pN0: No micro-metastases or macro-metastases or isolated tumour cells (ITCs) found.
- pN0(i+): Only ITCs found.
- pN1mi: Micro-metastases found, but no macro-metastases found.
- pN1: Metastases found in 1–3 lymph nodes, of which at least one is a macro-metastasis.
- pN2: Metastases found in 4–9 lymph nodes, of which at least one is a macro-metastasis.

Adopting the approach discussed in [2], we created a binary classification problem by treating pN0 as normal and unifying all classes that were not pN0 into a single class, cancerous. CAMELYON17 is a particularly challenging dataset, mainly because of the tumor heterogeneity it exhibits. There are WSIs, where the tumor is highly localized, and others, where only a handful of ITCs, such as those in the pN0(i+) class, are scattered across a sea of normal cells.  We specifically chose to conduct our evaluations on CAMELYON17 as, given the time of the rebuttal period, we were unable to train a new feature extractor on a different dataset. Thus, a reasonable alternative was to use our SimCLR feature extractor trained on cancer metastasis images in CAMELYON16. 

We used the SimCLR-trained feature extractor trained on CAMELYON16. Even though both CAMELYON16 and CAMELYON17 focus on lymph node metastasis detection, and transfer learning is, therefore, a reasonable choice, the two datasets contain different image sets and demonstrate distinct distributions. For example, it is well-known that there is an inevitable variability in H&E staining as a natural result of differing staining procedures and reagents among pathology labs and differences among digital WSI scanners that are being used. CAMELYON16 uses data collected from two centers, while CAMELYON17 uses data collected from five centers. 

Despite using an embedder that was not directly trained on CAMELYON17 images, we are pleased to report optimal results, highlighting the adaptability and robustness of our model across diverse datasets. We tested a 4-fold validation reporting the average test ACC and AUC as shown in the Table below:

|                     |CAMELYON17 |                               |
|---------------------|--------------------------------|--------------------------------|
| Method              | ACC($\uparrow$)                | AUC($\uparrow$)              |
| CLAM-SB             | 0.802            | 0.849           |
| CLAM-MB             | 0.803              | 0.858           |
| TransMIL            | 0.804              | 0.873            |
| DTFD-MIL            | 0.797              | 0.884            |
| GTP                 | 0.800          | 0.762            |
| DSMIL               | 0.815              | 0.863            |
| **CAMIL-L** | 0.828              | 0.881        |
| **CAMIL-G** | 0.818              | 0.875            |
| **CAMIL**   | 0.843   | 0.881 |

CAMIL outperforms all the baseline models in both ACC and AUC metrics, except for DTFD-MIL in AUC with a 0.003 difference. GTP  didn not perform well on this dataset, possibly due to its size, as was the case with CAMELYON16. We suspect that this drop in performance could be attributed to the MinCut pooling [3] operation, which could result in substantial information loss. DTFD-MIL performed well, leveraging the power of pseudo-bags, which could be beneficial when dealing with WSIs containing many tiles. As expected, TranMIL's performance is similar to CAMIL-G, since both models use the same Nystromformer. We presume that the superiority of CAMIL-G performance over TransMIL in accuracy may arise from the distinct aggregation strategies they employ, with CAMIL-G utilizing attention pooling before classifier input. Finally, the amalgamation of both elements in the CAMIL model yields the best performance overall.

[1] P. Bándi et al., "From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge," in IEEE Transactions on Medical Imaging, vol. 38, no. 2, pp. 550-560, Feb. 2019, doi: 10.1109/TMI.2018.2867350.

[2] Yufei CUI, Ziquan Liu, Xiangyu Liu, Xue Liu, Cong Wang, Tei-Wei Kuo, Chun Jason Xue, and Antoni B. Chan. Bayes-MIL: A new probabilistic perspective on attention-based multiple instance learning for whole slide images. In The Eleventh International Conference on Learning Representations, 2023.

[3] Bianchi FM, Grattarola D, and Alippi C. Spectral clustering with graph neural networks for graph pooling. In Proc. ICML, 2020, pp. 874–883.

Reviewer comment (Experiments):
The ablation study lacks an experiment showing the contribution of the pathology-specific self-supervised feature-extraction module. It would be easy to compare to a straight imagenet-pretrained ResNet18. [update: the authors have added an ablation study as requested]

Author response:
SimCLR feature extraction backbone is an integral part of our model, and it plays a pivotal role in generating concise and descriptive feature representations. Leveraging the pre-trained SimCLR weights lays the foundation for generating meaningful similarity scores and, therefore, is crucial for the optimal performance of the neighbor-constrained attention mask similar to many similarity-based histopathology approaches. Incorporating ImageNet weights directly into our model notably decreases its performance (as it does with most models [1, 2]), emphasizing the necessity of SimCLR within our model architecture. 

| Dataset    | ACC               | AUC               |
|------------|-------------------|-------------------|
| CAMELYON16 | 0.723	  | 0.743 |
| TCGA-NSCLC | 0.692 | 0.798 |


We believe that this decline in performance isn't model-specific but rather feature-specific, consistent with findings from other studies that utilize ResNet-18 pre-trained on ImageNet, such as those in the CAMELYON16 dataset. Works in [1] saw DS-MIL [3] achieve an AUC=0.478 ± 0.130 and ACC=0.478 ± 0.130 with the ResNet18 pre-trained on ImageNet only. Furthermore, works in [2] showed GTP performance on TCGA with the ResNet18 feature extractor pre-trained on ImageNet lead to ACC= 44 ± 0.280 and AUC: 60.4 ± 0.330. Typically, Resnet-50 with ImageNet weights is the backbone model used as the feature extractor.

However, we firmly believe this performance drop does not devalue our model's overall capabilities. Furthermore, we must stress that identical features were used across all baselines, ensuring fair model comparisons and enhancing evaluation integrity. Additionally, the availability of publicly available generalizable self-supervised histopathology models further simplifies the extraction of salient features for pathology analysis [4]. 

[1] Paul Tourniaire, Marius Ilie, Paul Hofman, Nicholas Ayache, Hervé Delingette, MS-CLAM: Mixed supervision for the classification and localization of tumors in Whole Slide Images, Medical Image Analysis, Volume 85, 2023, 102763, ISSN 1361-8415.

[2] Y. Zheng et al., "A Graph-Transformer for Whole Slide Image Classification," in IEEE Transactions on Medical Imaging, vol. 41, no. 11, pp. 3003-3015, Nov. 2022, doi: 10.1109/TMI.2022.3176598.

[3]  Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318–14328, 2021.

[4] Chen RJ, et al. A General-Purpose Self-Supervised Model for Computational Pathology. ArXiv [Preprint]. 2023 Aug 29:arXiv:2308.15474v1. PMCID: PMC10491320.

Reviewer comment (Evaluation):
The published TransMIL accuracy numbers for CAMELYON16 and TCGA-NSCLC are: (ACC=0.8837, ACC=0.8835). The paper cites: 0.905 and 0.905. I did not verify the other numbers, but please double-check. [update: this question has been answered - the papers uses different feature extractors SimCLR]

Author response:
> Q1: *"The published TransMIL accuracy numbers for CAMELYON16 and TCGA-NSCLC are: (ACC=0.8837, ACC=0.8835). The paper cites: 0.905 and 0.905. I did not verify the other numbers, but please double-check."*

A1: TransMIL used a ResNet-50 pre-trained on ImageNet as their backbone feature extractor. We used a ResNet-18 pre-trained on ImageNet and then fine tuned using SimCLR on CAMELYON16. We used the exact same features across the different MIL models in our evaluations. That is, for every model that was compared, we used features extracted from our SimCLR model on the same patches using the same folds. This will cause the difference in the metrics reported in TransMIL and our manuscript.

Reviewer comment (Theory):
Currently, the neighbor-constrained idea is operationalized by masking the attention matrix (from self-attention) using the neighborhood mask. However, the attention scores from self-attention typically represent how much a tile attends to other tiles, not necessarily how much the slide-level prediction attends to that specific tile. It is not clear how directly masking on this attention matrix relates to the intuition above.

Author response:
**2. Operationalisation of neighbor-constrained idea:**
Masking the attention matrix with a neighborhood mask aims to limit a tile's attention to its neighbors. This constraint guides the attention mechanism by controlling which neighboring tiles influence a specific tile's representation. Although this approach doesn't perfectly align with how slide-level predictions relate to individual tiles, it helps the model capture relevant local context within the broader context of slide-level predictions, facilitating the model's understanding of local and contextual information within the larger slide-level context. This can be seen by the improved performance over TransMIL regarding slide-level accuracy and localization.

Reviewer comment (Experiments):
In the compared baselines, are self-supervised learning also employed to enhance the feature extractors?

Author response:
> Q1: ***"In the compared baselines, are self-supervised learning also employed to enhance the feature extractors?"***

A1: We have used the exact same features for all models as we are aware that the feature set has a significant impact on the models' performance. That is, for all models, the SimCLR self-supervised learning feature extractor was used in part 2. in Figure 1 in the manuscript.

Reviewer comment (Writing):
In page 4, what is Q1 and K1, did you define it before? What are the m selected landmarks?

Author response:
> Q2: ***"In page4, what is Q1 and K1, did you define it before? What are the m selected landmarks?"***

A2: We apologize for the misunderstanding and thank the reviewer for spotting an error in our mathematical notation. We have now clearly defined $\boldsymbol{Q_1}$, $\boldsymbol{K_1}$, $\boldsymbol{V_1}$, and $m$ in the revised manuscript. To clarify here, in CAMIL, after features ($\boldsymbol{H}$) are extracted from the SimCLR feature extractor, they are first passed through a Transformer to capture the relationships between features and tiles and facilitate a more comprehensive feature aggregation. $\boldsymbol{Q_1}$, $\boldsymbol{K_1}$, $\boldsymbol{V_1}$ are the query, key, and value representations according to self-attention defined in [1]. We use the subscript $1$ as this is the first query, key, value representations in the CAMIL architecture. However, to address the challenge of memory overload due to the long-range dependencies in large WSIs, we adopt the Nystromformer architecture defined in [2]. This produces a "transformed" feature set $\boldsymbol{T}$= { $ \boldsymbol{t}_1, \ldots , \boldsymbol{t}_i, \ldots ,\boldsymbol{t}_N  $}, with each $\boldsymbol{t}_i \in \mathbb{R}^{n \times d}$, where, 

$$
\boldsymbol{t}_i = \text{softmax}(\frac{\boldsymbol{Q_1}(\boldsymbol{h_i}) \boldsymbol{\tilde{K}}_1^{T}(\boldsymbol{h_i})} {\sqrt{d_k}}) \left( \boldsymbol{A} \right)^+
\text{softmax}\left(\frac{\boldsymbol{\tilde{Q}}_1(\boldsymbol{h_i}) \boldsymbol{K}_1^{T}(\boldsymbol{h_i})} {\sqrt{d_k}}\right)\boldsymbol{V_1}(\boldsymbol{h_i}),
$$
where $\boldsymbol{\tilde{Q}}_1(\boldsymbol{h_i})$ and $\boldsymbol{\tilde{K}}_1(\boldsymbol{h_i})$ are the $m$ selected landmarks (see [2]) from the original $n$-dimensional sequence of $\boldsymbol{Q}_1$ and $\boldsymbol{K}_1$, $ \boldsymbol{A}^+=\text{softmax}\Bigg(\frac{\boldsymbol{\tilde{Q}}_1(\boldsymbol{h_i}) \boldsymbol{\tilde{K}}_1^{T}(\boldsymbol{h_i})} {\sqrt{d_k}}\Bigg)^+ $ is the approximate inverse of $\boldsymbol{A}$.  Softmax is applied along the rows of the matrix. $\boldsymbol{K_1}(\boldsymbol{h_i})$, $\boldsymbol{Q_1}(\boldsymbol{h_i})$, and $\boldsymbol{V_1}(\boldsymbol{h_i})$ are the first key, query, and value representations of $\boldsymbol{h_i}$ shown as $T(\boldsymbol{h})$ in step 3 in Figure 1 of the manuscript.

We have now updated our manuscript to show this clearly.

Reviewer comment (Writing):
Right below eq. 2, you talked about tile representations T, did you ever define it before? How is it different from H?

Author response:
> Q3: ***"Right below eq2, you talked about tile representations T, did you ever define it before? How is it different from H?"***

A3: We hope that the answer to Q2 above clarifies this. The tile representations $\boldsymbol{T}$ are the the output after $\boldsymbol{H}$ (the SimCLR extracted features) are passed through the Nystromformer $\boldsymbol{T(h)}$ in Figure 1 in the manuscript. $\boldsymbol{T}$ are then again transformed into new value, key, and query representations which are used to calculate the attention matrix. This attention matrix is the one that is masked by the neighborhood mask. Importantly, the neighborhood mask is calculated from the SimCLR representations $\boldsymbol{H}$. 

In our ablation studies we show how CAMIL-G (when the neighborhood attention module is removed from CAMIL) performs similarly to TransMIL [3], with the difference most likely being due to the feature aggregation as this is the only difference between CAMIL-G and TransMIL. We also show that CAMIL-L (removing the Nystromformer from CAMIL) outperforms CAMIL-G and the amalgamations of both the Nystromformer and neighborhood attention module lead to superior performance across all datasets.

[1]  Vaswani et al., Attention is all you need. Advances in Neural Information Processing Systems 30: Annual Conference on
Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998–6008, 2017.

[2] Xiong et al., Nystr ̈omformer: A nystr ̈om-based algorithm for approximating self-attention. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 14138–14148. AAAI Press, 2021. 

[3] Zhuchen Shao, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 2136–2147, 2021.

Reviewer comment (Evaluation):
Regarding the effectiveness of localization with this technique, the authors have produced some qualitative evidence in figure 3 and 4. However, since one of the main merits of this work is around better localization esp for problems with small foci of interest, can the authors a quantitative evaluation like in Table 1 in [1]?

Author response:
**Quantitative evaluation of localization:**
Numerous papers [1, 2] rely on the FROC curve for providing localizations, yet it wasn't evident to us from those studies which methodology to adopt to replicate the prior findings. Hence, we opted for a more direct method to assess our model's localization abilities. Following a methodology similar to [3], we used the dice score to qualify the model's ability to identify positive instances for the cancerous slides. For normal slides, we relied on computing specificity. This approach is straightforward, intuitive, and easily interpretable. We first computed the reference tile-accurate masks from the expert tumor delineations, considering a tile as a tumor if it contains at least 20\% annotated tumor. This allows a fairer comparison between predicted and reference masks since the models are not pixel-level accurate. To produce predicted masks, we used the scaled attention scores for each tile CAMIL, both CLAM models, TransMIL and DSMIL, the tile level logits for DTFD-MIL, and the tile-level GradCAM for GTP. For TransMIL, we used the attention values of class tokens in the self-attention matrix to the rest of the feature tokens, just as the authors describe. We applied a threshold of 0.5 to the model's output probabilities to generate the masks from the tile-level predictions. The results for the Dice score and Specificity are shown in the table below:

| Method            | Dice($\uparrow$)             | Specificity($\uparrow$)      |
|-------------------|------------------------------|------------------------------|
| CLAM-SB           | 0.459	         | 0.987            |
| CLAM-MB           | 0.406          | 0.573         |
| TransMIL          | 0.103           | 0.999 |
| DTFD-MIL          |0.525| 0.999 |
| GTP               | 0.418           | 0.851           |
| DSMIL             | 0.259          | 0.863           |        |                              |
| CAMIL | 0.515            | 0.980          |

CAMIL outperforms all except DTFD-MIL in terms of Dice score on cancerous slides. CAMIL shows a significant improvement over TransMIL in terms of tumour localization. However, TransMIL is more effective in terms of specificity on normal slides, meaning that it produces few false positives on normal slides. 

CAMIL outperforms DTFD-MIL on CAMELYON16 and TCGA-NSCLC in terms of ACC, F1 and AUC. Whole slide predictive accuracy, as well as cancerous localization, are necessary for these DL-models to be effective in the clinic. CAMIL represents a strong overall model that is only slightly surpassed in terms of F1 score on one dataset by one model and slightly surpassed in terms of localization by another model. 

We believe the decreased localization performance in CAMIL when compared to DTFD-MIL might be attributed to integrating the Nystromformer module in our model design, akin to its role in TransMIL. TransMIL, as indicated by the attention maps in our qualitative assessment and its slide-level performance, demonstrates the ability to grasp the general patterns within a WSI and distinguish between normal and cancerous slides. However, despite this, confirmed by its low Dice score, it falls short in effectively pinpointing specific cancerous evidence within slides. Integrating the Nystromformer into our model design might introduce a trade-off between slide-level accuracy and localization performance, resulting in improved slide-level accuracy with an expense of slightly decreased localization performance. CAMIL significantly improves cancerous localization over TransMIL, which can be attributed to our neighborhood attention module.


[1] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318–14328, 2021.


[2] Hongrun Zhang et al. DTFD-MIL: double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 18780–18790, 2022. doi: 10.1109/CVPR52688.2022.01824.


[3] Paul Tourniaire, Marius Ilie, Paul Hofman, Nicholas Ayache, Hervé Delingette, MS-CLAM: Mixed supervision for the classification and localization of tumors in Whole Slide Images, Medical Image Analysis, Volume 85, 2023, 102763, ISSN 1361-8415.

Reviewer comment (Experiments):
Currently, a comparison with DS-MIL [1] is missing. This would help to evaluate this work against a model that uses multi-scale features and motivate the need for local context even with a multi-scale featurizer.

Author response:
**DSMIL comparison**

We have included a comparison of DSMIL on all datasets using our feature set on our folds (single-scale features). Below is the table of results for DSMIL and CAMIL on all datasets. As requested by **Reviewer: 15GF*, we have added another dataset, CAMELYON17. Please see the table below:

| Model | Dataset    | ACC               | F1                | AUC               | Dice (patch-level) | Specificity (patch-level) |
|-------|------------|-------------------|-------------------|-------------------|-------------------|-------------------|
|**DSMIL**| CAMELYON16 | 0.874  | 0.848 | 0.949 | 0.259 	| 0.863
|           | TCGA-NSCLC | 0.853 | 0.864 | 0.954 |                |                      | 
| **CAMIL** | CAMELYON16 | 0.917 | 0.881 | 0.959| 	0.515 	|0.980|
|       | TCGA-NSCLC | 0.916 | 0.918 | 0.975|





On the same features that we use for comparisons for each dataset (single-scale features), CAMIL outperforms DSMIL in ACC, F1, AUC regarding slide-level classification for all datasets, and in both Dice and Specificity, regarding localization on CAMELYON17.

Reviewer comment (Presentation):
There is typo in table 1 heading: it should be 'TCGA-NSCLC'.

Author response:
We thank the reviewer for spotting the typos in our manuscript. We have now corrected those mentioned and other minor ones we found upon further inspection.

Reviewer comment (Writing):
There are some spelling mistakes in the work. Example - 'are impirative in successfull performance` in Conclusion section.

Author response:
We thank the reviewer for spotting the typos in our manuscript. We have now corrected those mentioned and other minor ones we found upon further inspection.

Reviewer comment (Experiments):
I would like to see performance on more challenging datasets for which the slide level labels are not visual features, such as mutation prediction or something similar.

Author response:
We appreciate the reviewers’ suggestion to test the model on more challenging datasets. While we agree that mutation prediction from whole slide images will be an exciting task to explore, we were unable to explore this in the time frame given. However, we have evaluated our model against all other models on CAMELYON17 [1] - another metastasis prediction dataset that has been more challenging than its previous counterpart, CAMELYON16 (as seen by lower classification metrics across models). We have provided the same comparison as the other datasets to CAMELYON17. Similar to the other datasets, CAMIL outperforms all other models in terms of ACC and AUC, except for DTFD-MIL in AUC with a 0.003 difference. Please see the table below:

|                     |CAMELYON17 |                               |
|---------------------|--------------------------------|--------------------------------|
| Method              | ACC($\uparrow$)                | AUC($\uparrow$)              |
| CLAM-SB             | 0.802            | 0.849           |
| CLAM-MB             | 0.803              | 0.858           |
| TransMIL            | 0.804              | 0.873            |
| DTFD-MIL            | 0.797              | 0.884            |
| GTP                 | 0.800          | 0.762            |
| DSMIL               | 0.815              | 0.863            |
| **CAMIL-L** | 0.828              | 0.881        |
| **CAMIL-G** | 0.818              | 0.875            |
| **CAMIL**   | 0.843   | 0.881 |

Reviewer comment (Experiments):
You use SimClr to train the feature extraction encoder. Have you test performance from other self-supervised pretrained encoders?

Author response:
We agree that the type of pre-trained encoder used in the pipeline affects model performance. It is worth mentioning that in all our reported results, we used the exact same features across the different MIL models. That is, for every model that was compared, we used features extracted from our SimCLR model on the same patches using the same folds. To evaluate how the SimCLR-based feature extraction encoder affects performance across models, we have conducted an ablation study using a ResNet18 model pre-trained on ImageNet, as suggested by **Reviewer 15GF**. Please see the response to that review for a full analysis.

Although we have not explored other self-supervised pre-trained autoencoders such as the mentioned ViT encoders, we will update our manuscript to discuss that this is an avenue for future work and cite those papers doing so, particularly works in [2].

Reviewer comment (Experiments):
Recent pre-prints suggest that larger ViT SSL trained encoders can benefit performance on less sophisticated aggregation functions.

Author response:
We agree that the type of pre-trained encoder used in the pipeline affects model performance. It is worth mentioning that in all our reported results, we used the exact same features across the different MIL models. That is, for every model that was compared, we used features extracted from our SimCLR model on the same patches using the same folds. To evaluate how the SimCLR-based feature extraction encoder affects performance across models, we have conducted an ablation study using a ResNet18 model pre-trained on ImageNet, as suggested by **Reviewer 15GF**. Please see the response to that review for a full analysis.

Although we have not explored other self-supervised pre-trained autoencoders such as the mentioned ViT encoders, we will update our manuscript to discuss that this is an avenue for future work and cite those papers doing so, particularly works in [2].