Dual-Curriculum Contrastive Multi-Instance Learning for Cancer Prognosis Analysis with Whole Slide Images
Abstract
The multi-instance learning (MIL) has advanced cancer prognosis analysis with whole slide images (WSIs).
However, current MIL methods for WSI analysis still confront unique challenges.
Previous methods typically generate instance representations via a pre-trained model or a model trained by the instances with bag-level annotations, which, however, may not generalize well to the downstream task due to the introduction of excessive label noises and the lack of ﬁne-grained information across multi-magniﬁcation WSIs.
Additionally, existing methods generally aggregate instance representations as bag ones for prognosis prediction and have no consideration of intra-bag redundancy and inter-bag discrimination.
To address these issues, we propose a dual-curriculum contrastive MIL method for cancer prognosis analysis with WSIs.
The proposed method consists of two curriculums, i.e., saliency-guided weakly-supervised instance encoding with cross-scale tiles and contrastive-enhanced soft-bag prognosis inference.
Extensive experiments on three public datasets demonstrate that our method outperforms state-of-the-art methods in this ﬁeld.
The code is available at https://github.com/YuZhang-SMU/CancerPrognosis-Analysis/tree/main/DC_MIL%20Code.
Introduction
Cancer prognosis analysis is of great signiﬁcance for risk-beneﬁt assessment and clinical decision [1].
Computational whole slide image (WSI), which entails the quantitative proﬁling of spatial patterns and tumor microenvironments in tissue slides has advanced the application of deep learning in cancer prognosis analysis [2, 3].
However, deep learning in WSIs is often hindered by the gigapixel size and the lack of pixel-level annotations [4].
Recently, multi-instance learning (MIL) methods have attracted increasing attention to performing inference in a weakly-supervised manner and have been successfully applied to WSI analysis [5, 6, 7].
In general, the MIL consists of two stages, i.e., instance encoding stage and instance aggregation stage [8, 9].
In the encoding stage, several methods have utilized the pre-trained model (e.g., ResNet on ImageNet) to extract instance representations in an unsupervised way (i.e., without the reference of outcome labels) [10, 11, 12].
However, recent studies have demonstrated that the pre-trained model may not generalize well to the downstream tasks as it is task-agnostic 1 and inclined to overﬁt the pretraining objective [13, 14, 15].
Alternatively, some work directly assigned each instance the bag-level annotation (i.e., the survival time of patient) for network training, which introduces task-speciﬁc 1 supervision and improves model’s generalisability [16, 17, 18].
Actually, the prognosis evaluation is primarily and comprehensively determined by certain representative regions, yet those regions might only occupy a small portion of WSI [17].
As a result, the strong supervision on all instances, especially on prognosis-irrelevant tiles, will introduce excessive label noises and limit model’s performance [19, 20].
For alleviating this issue, a feasible way is to supervise the instance encoding in a weak and easy-to-hard manner.
On the other hand, existing methods often generate mono-scale instance representations [11, 21] or typically concatenate multi-scale information from multi-magniﬁcation WSIs [22, 23], which may cause feature redundancy and ignore ﬁne-grained details across multi-magniﬁcation images.
In the aggregation stage, most previous studies aggregated instance representations within a bag for prognosis estimation by certain fusion strategies (e.g., pooling operation and attention mechanism) [24, 25].
However, it may overwhelm prognosis-relevant information, cause intra-bag redundancy, and reduce inter-bag discrimination, if many instances from irrelevant regions are enrolled [12, 26].
To address the aforementioned issues, we arm MIL with a dual-curriculum strategy and propose a dual-curriculum contrastive MIL method for cancer prognosis analysis with WSIs.
The proposed method consists of two easy-to-hard curriculums, i.e., saliency-guided weakly-supervised instance encoding with cross-scale tiles and contrastive-enhanced soft-bag prognosis inference.
The main contributions are summarized as follows:
1 ) We present a dual-curriculum contrastive MIL method which includes two easy-to-hard curriculums.
We ﬁrst conduct a preliminary task to learn instance representations by considering risk stratiﬁcation status (degraded from survival time) as annotation, followed by the prognosis inference with survival time as supervision.
We design the ﬁrst curriculum of saliency-guided weakly-supervised instance encoding with cross-scale tiles.
It is supervised by relatively weak annotations so as to reduce label noises and maintain prognosis-related guidance.
Additionally, to imitate the reviewing procedure of pathologists, we leverage the low-magniﬁcation saliency map to guide the encoding of high-magniﬁcation instances for exploring ﬁne-grained information across multi-magniﬁcation WSIs.
We develop the second curriculum of contrastive-enhanced soft-bag prognosis inference.
Instead of enrolling all instances, we adaptively identify and integrate representative instances within a bag (as the soft-bag) for prognosis inference and leverage the constrained self-attention strategy to obtain extra sparseness for soft-bag representations, which can help reduce intra-bag redundancy in both instance and feature levels.
Meanwhile, we improve the Cox loss with two-tier contrastive learning for enhancing intra-bag and inter-bag discrimination.
We evaluate the proposed method on three public cancer datasets, and extensive experiments demonstrate its superiority in cancer prognosis analysis with WSIs.
Related Work 2.1 WSI-based Cancer Prognosis Model
In recent years, many methods, especially convolutional neural networks (CNNs), have been developed for cancer prognosis evaluation with WSIs [27, 28, 29].
Due to the gigapixel size of WSI, some methods took small-size ROIs selectively sampled from WSIs as network’s input for training and inference, in which all ROIs were assigned with patient-level outcome labels [30, 31].
However, these methods generally require prior knowledge from pathologists, which is experience-dependent and may suffer from high inter-observer variation.
Therefore, current researches have proposed to crop the entire WSI into many tiles for model’s input so as to make full use of available information contained by WSI [10, 32, 33].
However, these methods often confront the lack of tile-level annotations and may result in a sub-optimal solution.
To this end, MIL methods have been introduced to perform cancer prognosis analysis in a weakly-supervised manner and have shown promising performance in this ﬁeld [5, 28].
2.2 Multi-instance Learning in WSI Analysis
The MIL methods in WSI analysis can be roughly divided into two categories according to the way instances are leveraged in the model [12, 17].
The one infers bag-level prediction from instance-level probabilities via average pooling, maximum pooling, or other ensemble strategies [34, 35].
By comparison, the other one, referred to as the bag embedding method, aggregates instance representations as bag ones, upon which a model is constructed for bag-level prediction [36, 37].
The former is simple and straightforward but empirically proven to be inferior to the latter in terms of generalization performance.
Therefore, the bag embedding method has attracted extensive attention in WSI analysis [17].
For example, Courtiol et al.
[6] developed a MIL method based on CNN, called MesoNet, to accurately predict the overall survival of mesothelioma patients with WSIs.
Yao et al.
[5] proposed a deep attention multi-instance survival model to efﬁciently learn features from WSIs and fuse them for prognosis prediction.
Our method essentially belongs to the bag embedding method.
Different from previous work, the proposed method consists of two easy-to-hard curriculums, comprehensively considering ﬁne-grained information across multi-magniﬁcation WSIs, intra-bag redundancy, and inter-bag discrimination.
Method 3.1 Problem Formulation
Denote the triplet { X n , T n , δ n } Nn =1 as the dataset consisting of N patients, where X n is the n -th patient with one or more WSIs, and T n and δ n are respectively its observed survival time and event indicator (viz. equals to 1 and 0 for uncensored and censored samples, respectively).
Prognosis analysis is an ordinal regression task that models time-to-event distribution, which is formulated as
[Equation]
[Equation]
where S and ˆ T n denote the prognosis inference function and the estimated survival time, respectively.
In practice, it is difﬁcult to directly feed the WSI into the model due to its gigapixel size.
Accordingly, many studies [10, 32, 33] have cropped the WSI into a large amount of tiles and generated a bag
X n = { x n,i } N n i =1 that contains N n instances (tiles) for the n -th patient.
Then, MIL is leveraged to construct the prognosis model as follows:
[Equation]
where E is an instance encoding function that extracts feature representation for each instance, and A is a permutation-invariant instance aggregation function that pools the instance representations into the bag ones.
However, previous studies [10, 11, 12, 16, 17, 18] typically generate instance representations via a pre-trained model or a model trained by the instances with bag-level annotations, which may not generalize well to the downstream task.
As a result, we decompose the model into two easy-to-hard curriculums, including (1) Curriculum I (C-I): instance encoding (via E ) during preliminary risk stratiﬁcation (via C , a risk stratiﬁcation function), and (2) Curriculum II (C-II): prognosis inference (via S ) after instance aggregation (via A ).
Formally, the optimization process can be formulated as
[Equation]
where † ( · ) outputs 1 (0) if true (false), and L I and L II denote the loss functions for C-I and C-II, respectively.
The risk stratiﬁcation status Y n is determined by { T n , δ n } with a three-year time threshold (denoted as T r ) as follows:
[Equation]
3.2 Curriculum
Saliency-guided Weakly-supervised Instance Encoding with Cross-scale Tiles
In this section, we introduce the ﬁrst curriculum of saliency-guided weakly-supervised instance encoding with cross-scale tiles, as shown in Figure 1.
Given the input instance with S magniﬁcations { x sn,i } Ss =1 ( S = 3 in this work), the proposed method contains S branches and each branch takes different-magniﬁcation tiles as input, which aims to explore multi-scale information from multimagniﬁcation images.
For the s -th branch, it mainly consists of a feature extractor F s , a feature aggregator G s and a classiﬁer C s .
And { F s } Ss =1 and { G s } Ss =1 form E , while { C s } Ss =1 constitute C in Eq.(3).
The feature extractor F s contains s residual modules with the parameters { θ 1 F s , · · · , θ s F s } , and the ﬁrst s − 1 modules share the same architecture with F s −
if s > 1 .
Figure 2 illustrates the architecture details 2 .
Different from previous studies [22, 23] that separately feed { x sn,i } Ss =1 into the network and ignore ﬁne-grained details across multi-magniﬁcation images, we encourage the network to focus on salient regions by utilizing the prior knowledge of the ( s − 1 )-th branch to highlight the input x sn,i , which can be formulated as
[Equation]
(5) where ⊗ denotes the element-wise multiplication operator, and m s − 1 n,i is the salient mask indicating high-risk regions.
For convenience, we introduce how to acquire m sn,i , which is easily generalized to m s − 1 n,i .
Speciﬁcally, m sn,i is computed by Class Activation Map [38] as follows:
[Equation]
where (cid:12) is the channel-wise multiplication operator, and ω s indicates the importance of channel of feature maps for risk stratiﬁcation prediction in the s -th branch.
To obtain ω s , we ﬁrst generate an instance embedding vector z sn,i by
[Equation]
where G s is a feature aggregator that performs channel-direction attention (as shown in Figure 2).
Incorporating comprehensive information from current and previous branches, we subsequently feed instance embeddings { z jn,i } sj =1 into the classiﬁer C s for predicting the high-risk probability p sn,i , which is computed by
[Equation]
where C s is implemented by a linear layer with the parameters { ω s , ω s } (as shown in Figure 2), and φ and T denote softmax activation function and transposition operation, respectively.
We introduce a hierarchical transfer strategy to leverage the prior from different-magniﬁcation images, accelerate network convergence, and maintain the speciﬁcity of current branch.
Speciﬁcally, as shown in Figure 1, the parameter θ s F s (for any s ) of the s -th module in F s is completely learnable and has not any constraints, while θ s − 1 F s (for s > 1 ) is also learnable but constrained to approximate θ s − [Equation] by the structural loss L Ω as follows: [Equation]
[Equation]
For the remaining parameters { θ 1 F s , · · · , θ s − 2 F s } (for s > 2 ) in F s , they are ﬁrst initialized with { θ 1 F s − 1 , · · · , θ s − 2 F s − 1 } , and then are frozen.
Furthermore, we propose a hybrid loss function L I that contains two parts, i.e., empirical loss L (cid:96) and structural loss L Ω , to train the network:
[Equation]
where β Ω is the weight coefﬁcient, and L (cid:96) is deﬁned as
[Equation]
(11) where y sn,i ∈ Y n denotes the risk stratiﬁcation status of the i -th instance in the n -th bag.
Finally, we can obtain the multi-scale instance representation z n,i for subsequent prognosis inference task by z n,i =
[ z 1 n,i , · · · , z Sn,i ] .
3.3 Curriculum II: Contrastive-enhanced Soft-bag Prognosis Inference
In this section, we develop the second curriculum of contrastive-enhanced soft-bag prognosis inference, as shown in Figure 1.
Given a set of instance representations from the n -th bag, i.e.,
Z n =
[ z n, 1 ; z n, 2 ; · · · ; z n,N n ] T ∈ R N n × C , where C denotes the feature dimension, the proposed method mainly consists of a soft-bag learning module B , a constrained self-attention module D , a contrastive-enhanced prognosis inference function S , where B and D form A in Eq.(3).
For the soft-bag learning module B (as shown in Figure 2), it ﬁrst learns new bag representation
E n ∈ R N n × D (where D denotes the feature dimension) from Z n via a linear layer.
Instead of enrolling all instances, we adaptively identify and integrate representative instances within a bag by introducing the function H :
[Equation]
(12) where ˆ E n ∈ R N B × D ( N B is the number of selected instances in each bag, and N B (cid:28) N n ) is the feature representation of those selected instances, and W n ∈ R N n × N B denotes the weight matrix that is generated by feeding E n into two linear layers (as shown in Figure 2).
And ˆ h n ∈ R N n × 1 is an indicator vector that adaptively selects N B representative instances and is computed by
[Equation]
The constrained self-attention module D and contrastive-enhanced prognosis inference function S are introduced below.
We leverage D to obtain extra sparseness for soft-bag representations.
Given the input ˆ E n , it ﬁrst linearly projects ˆ E n into three feature spaces (i.e., Q n ∈ R N B × ˆ D , K n ∈ R N B × ˆ D , and V n ∈
R N B × ˆ D , and ˆ D is the feature dimension) via three mapping matrices (i.e., W Q ∈ R D × ˆ D , W K ∈
R D × ˆ D , and W V ∈ R D
× ˆ D ), in which the mapping matrices are constrained by
[Equation]
Then, the sparse soft-bag representation B n ∈ R 1 × D B (where D B is the feature dimension) can be obtained by the correlation-based activation mechanism [39], as illustrated in Figure 2.
It is worth mentioning that the collaboration of B and D can signiﬁcantly help reduce intra-bag redundancy in both instance and feature levels so as to improve model’s generalization.
For S , it takes B n as input to infer the relative risk ˆ T n via a linear projection matrix W S ∈ R D B × 1 , as shown in Figure 2.
In order to boost the ability of soft-bag inference, we perform two-tier contrastive learning (TCL) to enhance intra-bag and inter-bag discrimination.
We assume that a bag B n is sampled from the distribution O
=
{O + , O − } , where the positive source O + denotes the distribution of high-hazard samples (i.e., those with survival time ≤ T r ), while the negative source O − presents the distribution of low-hazard samples (i.e., those with survival time > T r ).
We ﬁrst focus on inter-bag discrimination.
Let B n = { B 1 , · · · , B N Bn } / B n present the collection of bags (expect B n ) from the same source O +
(or O − ) as B n , where N B n denotes the number of such bags.
We can obtain the representation of B n (denoted as ˆ B n ) by merging the representations of all bags in B n via an attention function K (as illustrated in Figure 2).
Inspired by [40], inter-bag discrimination can be enhanced by maximally preserving the mutual information between B n and ˆ B n as follows:
[Equation]
According to Eq.(12), partial instances within a bag are adaptively discarded, and the rest remained.
However, all instances within the bag belong to homologous tissues sampled from the same patient, which means that it should show potential correlation among these instances.
Therefore, similar to
Eq.(15), we maximize the following mutual information to improve intra-bag discrimination:
[Equation]
where B n is obtained via K to merge the representations of those discarded instances E n that has the deﬁnition of E n = H ( E n , 1 − ˆ h n ) , as illustrated in Figure 2.
To this end, we propose to jointly maximize Eq.(15) and (16), i.e., max ( (cid:80) n ∈ [ N ] I ( B n , ˆ B n ) + I ( B n , B n )) , which is formulated by max ( (cid:88)
[Equation]
Instead of constructing a generative model p ( B n | ˆ B n ) p ( B n | B n )
[Equation]
we leverage a log-bilinear function L to model the density ratio which preserves the mutual information:
[Equation]
where W L and V L are trainable parameters.
Motivated by [41], we design the TCL loss based on the
InfoNCE function, which has the following deﬁnition:
[Equation]
where B denotes the collection of all bags.
Additionally, we utilize the negative log-partial likelihood of Cox proportional hazard regression model [42] to train the network, which is computed by
[Equation]
where W S is the linear projection matrix in S , and R ( T n )
= { i : T i ≥ T n } is the collection of all samples (including censored and uncensored ones) with survival time being longer than T n .
Finally, the loss function L II for the second curriculum can be deﬁned as
[Equation]
where β s is the weight coefﬁcient.
Experiments and Results
In this section, we brieﬂy describe datasets, preprocessing, and implementation details, followed by experimental results.
4.1 Datasets and Preprocessing
In this work, we evaluated the proposed method on three datasets from The Cancer Genome Atlas (TCGA) database, including colon adenocarcinoma (COAD)( N = 365 ), hepatocellular carcinoma (LIHC)( N = 334 ), and bladder urothelial carcinoma (BLCA)( N = 396 ).
Each dataset contained
WSIs stained with hematoxylin and eosin (H&E) and clinical information (i.e., survival time and event status).
After segmenting tissue areas via the Otsus method, a set of tiles under different magniﬁcations ( 20 × , 10 × , 5 × ) were non-overlappingly sampled from each segmented tissue area, and the window sizes were set to 512 × 512 , 256 × 256 , and 128 × 128 , respectively.
4.2 Implementation Details
In our experiments, we adopted the 5-fold cross-validation strategy to comprehensively evaluate our proposed method and the ratio of training and validation sets was set to 4:1.
Speciﬁcally, we divided the entire dataset into ﬁve folds, among which four folds for model training while the remaining one for model evaluation.
Notably, the cross-validation strategy was conducted on patient level to prevent data leakage, which means that the WSI of each patient only appeared in one of these subsets.
During the training stage, we selected the best model in terms of two speciﬁc evaluation metrics for different curriculums.
That is, the model with the highest accuracy was selected for Curriculum I, while the model with the highest C-index for Curriculum II.
The concordance index (CI), the receiver operating characteristic (ROC) curve, the area under ROC curve (AUC), and Kaplan-Meier (KM) curve were used to assess model’s performance.
We chose three-year time as threshold for risk stratiﬁcation status and AUC calculation, which lies on two reasons as follows: 1) Three-year survival time is an important indicator for clinically evaluating the prognosis of patients, especially for three cancer types studied in our work [43, 44, 45]. 2)
For the used datasets, using three-year survival time as threshold can make the distribution of positive and negative samples relatively balanced, which is beneﬁcial to network training.
We implemented all competing methods using the Pytorch1.9 library on Python3.6.
All intensive calculations were ofﬂoaded to a 12 GB NVIDIA Pascal Titan X GPU.
Curriculum I. We adopted Adam optimizer with the weight decay of 0.2 for network training.
For different magniﬁcations s ∈ { 1 , 2 , 3 } , the learning rate, batch size and epoch number were set to { 10 − 5 , 3 × 10 − 6 , 3 × 10 − 6 } , { 32 , 16 , 16 } and { 50 , 50 , 50 } , respectively.
The weight coefﬁcient β Ω was set to 10 − 5 .
Curriculum II.
We adopted SGD optimizer with the momentum of 0.9 for network training.
The learning rate, batch size and epoch number were set to 10 − 4 , 16 and 1000, respectively.
The soft-bag size N B was set to 3 and the weight coefﬁcient β s was set to 10 − 3 . [Table] [Table] DC_MIL(our)
C
OAD p = 0. p = 0. p = 0. p = 0.
p = 0.
p = 0.
p = 0.002 p = 0.
p = 0.
p = 0.
p =
4.96Ep = 4.07E-6 p = 0.415 p =
0.039 p = 0.
p =
3.99E-7 p = 0.
p = 0.
L I H
C
B L C A 4.3 Comparison with State-of-the-Art Methods We compared our method with other weakly-supervised methods in cancer prognosis analysis, including clustering approaches [46], graph networks [10, 47], and multi-instance learning methods [5, 6].
Table 1 presents the 5-fold cross-validation results (in terms of CI and AUC) of all competing methods.
The KM curves as well as p -values, are provided in Figure 3.
We can observe that the proposed method outperforms others in head-to-head comparisons, achieving an overall CI of 0. and AUC of 0.740.
It may beneﬁt from several potential advantages in DC_MIL: 1) Unlike some methods that leverage the dimension reduction technique [46] or pre-trained model [5, 6, 10, 47], it encodes instances in a weakly-supervised manner so as to reduce label noises and maintain prognosis-related guidance. 2)
It utilizes the low-magniﬁcation saliency map to guide the encoding of high-magniﬁcation instances for exploring ﬁne-grained information across multi-magniﬁcation WSIs.
3) It introduces a soft-bag learning method and a constrained self-attention strategy to help reduce intra-bag redundancy in both instance and feature levels, respectively.
4) It equips the Cox loss with two-tier contrastive learning for enhancing intra-bag and inter-bag discrimination.
Additionally, we also ﬁnd that WSISA [46], MesoNet [6], and DeepAttnMISL [5] outperform DeepGraphSurv [47] and Patch-GCN [10], which may owe to the selection of discriminative tiles.
More results can refer to Appendix D.
4.4 Ablation Study
Ablation on Curriculum I.
We validated the efﬁcacy of each crucial component in Curriculum
I. From Table 2, we can observe several key points: 1) Compared with pre-trained ResNet- (regardless with MM or not), the proposed model shows signiﬁcant performance improvement, which demonstrates the effectiveness of Curriculum I. 2)
The model with MM outperforms the one without MM and gets CI gain of 0.039 and AUC gain of 0.061, which beneﬁts from the utilization
[Table]
of multi-magniﬁcation information.
The model without SG or FT suffers from performance degradation (average CI drops of 0.028 and 0.011 and AUC drops of 0.039 and 0.028).
It indicates that ﬁne-grained information across multi-scale tiles and hierarchical transfer strategy can help signiﬁcantly improve the performance of model.
More results can refer to Appendix D.
Ablation on Curriculum II.
We also investigated the efﬁcacy of each key component in Curriculum
II.
From Table 2, we have the following observations: 1)
The baseline method performed prognosis prediction with the Cox model, which achieves average CI of 0.642 and AUC of 0.669.
Compared to the baseline method, the model with soft-bag strategy achieves average CI gain of 0.024 and
AUC gain of 0.031, which mainly proﬁts by adaptively identifying and integrating the representative instances within a bag.
The models with CSA, TCL, and both of them show better performance with average CI gains of 0.006, 0.024, and 0.032 and AUC gains of 0.011, 0.02, and 0.04, respectively, which beneﬁts from the reduction of intra-bag redundancy and the enhancement of intra-bag and inter-bag discrimination.
More results can refer to Appendix D.
4.5 Parameter Analysis
Inﬂuence of the Soft-bag Size N B .
To explore the inﬂuence of the soft-bag size N B , we conducted a set of experiments by varying N B within the set of { 1 , 2 , 3 , 4 , 5 } .
Note that N B = 1 corresponds to the standard MIL method.
From Table 3, we can observe that the model shows the best performance when N B equals to 3 (for COAD and LIHC) or 4 (for BLAC).
It indicates that the top 3 ∼ 4 instances are adequate to represent the bag.
Besides, the model with N B > 1 works better than the standard
MIL method, indicating the effectiveness of the soft-bag strategy for prognosis inference.
Inﬂuence of the Magniﬁcation Number S .
We conducted experiments to investigate the inﬂuence of the magniﬁcation number S by varying S in the range of { 1 , 2 , 3 , 4 } .
Note that S = 1 corresponds to the mono-magniﬁcation input and S = 4 corresponds to the multi-magniﬁcation input with { 20 × , 10 × , 5 × , 2 . 5 ×} .
As shown in Table 3, the model’s performance is improved as S increases.
And the model can achieve the best performance on all datasets and reach a stable state after S = 3.
Inﬂuence of the Weight Coefﬁcients β Ω and β s .
We turned to the inﬂuence of the weight coefﬁcients β Ω and β s by varying them within the sets of { 2 e − 4 , 5 e − 5 , 1 e − 5 , 2 e − 6 } and { 2 e − 2 , 5 e − 3 , 1 e − 3 , 2 e − 4 } , respectively.
As we can observe from Table 4, the performance of the model ﬂuctuates slightly when both β Ω and β s vary, which veriﬁes the robustness of the proposed method to the weight coefﬁcients.
4.6 Visualization
This section exhibits the salient maps output by the proposed method for some samples.
As illustrated in Figure 4, the model mainly focuses on the ﬁne-grained regions (under 20 × WSIs) that are wrapped by the salient regions under 10 × WSIs, indicating the proposed method can effectively guide the [Table] [Table] R I S
K
L o w encoding of high-magniﬁcation instances so as to encourage the network to focus on more ﬁne-grained information.
We have provided more intuitive results and analyses in Appendix D.
Conclusion
In this paper, we present a dual-curriculum contrastive MIL method for cancer prognosis analysis with WSIs, which contains two easy-to-hard curriculums, i.e., saliency-guided weakly-supervised instance encoding with cross-scale tiles and contrastive-enhanced soft-bag prognosis inference.
The proposed network is easy to train in each curriculum and can be applied to gigapixel WSI analysis with good visualization.
Besides, the proposed method outperforms state-of-the-art methods in cancer prognosis analysis over three public datasets.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 61971213 and Grant 61671230, in part by the Basic and Applied Basic Research Foundation of
Guangdong Province under Grant 2019A1515010417, and in part by the Guangdong Provincial Key
Laboratory of Medical Image Processing under Grant No.2020B1212060039.
We would like to express our gratitude to The Cancer Genome Atlas (TCGA) database for the open-source data and to anonymous reviewers for their insightful comments.