End-to-end Multiple Instance Learning for Whole-Slide Cytopathology of Urothelial Carcinoma
Editor: TBA
Abstract
As a non-invasive approach, cytopathology of urine sediment is a highly promising approach to diagnosing urothelial carcinoma.
However, computational assessment of the cytopathological status of a sample raises the challenge of identifying few cancerous cells among thousands of cells in a microscopic whole-slide image.
To address this challenge, we propose an end-to-end trainable multiple instance learning approach that combines the attention mechanism and hard negative mining to classify hematoxylin and eosin stained patient-level whole-slide images of urine sediment cells.
The singular cells are extracted by a simple foreground detection algorithm.
With feature embeddings computed for each image patch in a bag by a convolutional neural network, the attention mechanism serves as the pooling operator, enabling a bag-level prediction while still giving an interpretable score for each image patch.
This enables the identiﬁcation of key instances and potential regions of interest that trigger a patient-level decision.
Our results show that the proposed system can diﬀerentiate between normal and cancerous urothelial cells, thus enabling the non-invasive diagnosis of urothelial carcinoma in patients using urine sediment analysis.
1. Introduction
Bladder cancer is the 11 th most commonly diagnosed cancer in the world, with approximately 75% of patients suﬀering from non-muscle invasive bladder cancer (NMIBC).
As bladder cancer is likely to recur, patients often undergo follow-up treatment which is one of the reasons it leads to the highest cost of care per cancer patient (Antoni et al., 2017; Lee et al., 2012).
Cystoscopy and urine cytology still form the backbone of diagnosis and follow-up tests of bladder cancer.
Despite the now predominant use of ﬂexible endoscopes, cystoscopy remains an invasive diagnostic procedure that is perceived as uncomfortable by patients.
The usage of non-invasive alternative diagnostic methods remains challenging as urine cytology for instance is highly dependent on the expertise of the examiner and has limited sensitivity (Sullivan et al., 2010).
Several urine-based tests have been developed, but none of them have been recommended in the European Association of Urology (EAU) guidelines for diagnosis and treatment of bladder cancer, and there is consensus that markers can not replace urethrocystoscopy.
However, non-invasive methods promise potential support in tumor surveillance through guiding the use of more invasive techniques (Babjuk et al., 2013).
The commonly accepted working hypothesis of urine based urothelial cancer cytopathology is that cancerous urothelial cells are contained in the urine sediment (Kirkali et al., 2005).
Yet, the relatively few cancerous urothelial cells are largely outnumbered not just by normal urothelial cells, but also by cells of other types such as squamous cells or leukocytes.
Thus, the application of deep learning approaches inevitably faces similar problems as gigapixel whole-slide images (WSIs) in histopathology, where systematic annotation is feasible only at the whole-sample level, while annotations with cellular or even pixel precision are prohibitively expensive.
Yet, WSI based approaches promise improved diagnostics and better treatment strategies compared to the subjective bias introduced among observers and alleviate laborious annotation eﬀorts by trained experts.
One potential approach for this setting was prominently introduced by Campanella et al.
(2019) in the application of multiple instance learning (MIL) to WSI analysis in the context of histopathology.
Here, each WSI is weakly annotated with a coarse-grained patient-level label into a binary classiﬁcation setting of healthy vs. cancerous.
The WSI is then regarded as a bag of extracted or tiled image instances whose individual sub-labels are unknown.
A bag is classiﬁed as either positive if it contains at least one positive element, or as negative if it contains none.
In our MIL setting, we classify patient urine sediment samples into two disease statuses without any given labeling except for the clinically attributed patient status of urocystitis (negative) or cancerous (positive).
Compared to MIL in histopathology, cells in cytopathological WSIs are randomly distributed and thus lack dependencies on their spatial surrounding.
To summarize, the main contributions of our paper are:
• the ﬁrst approach that works fully end-to-end on whole-slide images of urine sediment; • the combination of gated attention pooling (Ilse et al., 2018) and a modiﬁed version hard negative mining (Li et al., 2019); and • the use of large, adaptively extracted patches as a crucial step for WSI-based learning.
2. Methods
Given a patient WSI of H&E stained urine sediment cells (as a bag X i ) with its extracted instances X ij ∈
X
i , j = 1 , ..., N i we try to identify the class label C i ∈ { 0 , 1 } and potentially the key instances, with c ij = 1 of singular urothelial cells in this bag that triggered this decision.
Therefore we train an end-to-end deep learning system based on multiple instance learning principles with a trainable pooling operator in the form of the attention mechanism.
To improve performance, instances falsely classiﬁed as positive are collected to form additional data samples via hard negative mining.
The complete, integrative approach can be examined in Fig.
2.1 Preprocessing (Extraction of image patches)
There exist many suitable oﬀ-the-shelf algorithms for single cell detection to extract suitable image patches from a WSI, such as using the Mask R-CNN (He et al., 2017) architecture in a similar experimental case by Sadaﬁ et al.
(2020) or the generalist segmentation approach
Cellpose (Stringer et al., 2020).
However, using these methods eﬀectively often requires additional training or transfer learning and some amount of professionally annotated data, especially when dealing with the medical imaging domain.
Instead, we propose a simple image binarization based approach, based on Otsu’s method (Otsu, 1979) and morphological closing to segment objects in the foreground, ﬁnd the contours of potential objects and determine their area and centroids by using spatial image moments (Hu, 1962).
By empirically setting a lower and upper boundary on the desired area size, suitable image patches of cells can then be extracted from the collection of centroids per WSI.
2.2 Multiple instance learning
In the classic, binary supervised learning setting a model is trained to predict the value of a target categorical variable c ∈ { 0 , 1 } for a given sample or instance x ∈ R D .
However, in the case of MIL, instead of a single instance, there exists a so-called bag of instances
X = { x 1 , ..., x N } , where the total number and ordering of instances does not matter and with only a singular label C associated with the bag.
According to the original problem formulation by Dietterich et al.
(1997) we can assume that individual labels exist for each of the instances in a bag, and while these are not directly accessible we can formulate the MIL problem setting as:
[Equation]
To implement MIL there is usually a distinction between two diﬀerent approaches, instance- and embedding-based.
In either way, the model Θ( X ) is composed of, a transformation f ψ of individual instances, a permutation invariant pooling function σ such as mean, max or noisy-or that aggregates all elements in a bag (Wang et al., 2019; Maron and
Lozano-P´erez, 1998) and lastly a transformation g φ of combined instances (Zaheer et al., 2017) to classify the whole bag such that:
[Equation]
Both transformations can be modeled using neural networks with parameters ψ and φ .
In the instance-based approach however transformation g is the identity function.
Importantly, both approaches have disadvantages as the embedding-based approach lacks interpretability, while the instance-based approach is hard to train due to error propagation (Liu et al., 2012).
2.3 Attention-based pooling
The attention mechanism is widely known and used in a plethora of deep learning tasks.
Originally introduced by Bahdanau et al.
(2014) it is now a key component in many stateof-the-art applications from natural language processing (Vaswani et al., 2017) to semantic image segmentation (Chen et al., 2016).
Typically, it is assumed that all instances are sequentially dependant, however in the case of MIL it was introduced by Ilse et al.
(2018) that the attention mechanism can be used as a permutation-invariant pooling operator, substituting the normally used mean or max aggregation operators.
This attention-based pooling calculates a weighted average over the previously extracted feature embeddings, where the weights are learned by a neural network.
In our work we employ a slightly modiﬁed version, which was also introduced in the same paper of Ilse et al.
(2018), called gated attention, as results have shown increased performance in similar histopathological problem settings.
Here, an additional, learnable non-linearity in the form of the sigmoid function sigm( · ) is introduced as a gating mechanism, to counter the potential approximately linear behaviour of the tanh( · ) function for x ∈ [ − 1 , 1] (Dauphin et al., 2017).
Let H =
{ h 1 , ..., h N } be a bag of N instance embeddings as a result of transformation f ψ , then the attention-based MIL pooling is deﬁned as:
[Equation]
where
[Equation]
in which w , V and U are parameters that are learned during training, in contrast to the previously mentioned pooling operators that are not trainable.
The resulting attention scores a i have numerous advantages, as they provide a scalar, interpretable value for each instance in a bag and act as a similarity measure to compare instances against each other.
Additionally the attention mechanism serves as a gradient update ﬁlter during training (Ilse et al., 2018), as instances with higher correlated attention will contribute more to learning the model.
This methodology extends the embedding-based
MIL approach and integrates desirable advantages of the instance-based approach, enabling the discovery of key instances.
2.4 Hard negative mining
One further possible use of the learned attention scores is that they can be used to select hard negative instances, which enables us to extend the model with a hard negative mining (HNM) loop (Li et al., 2019).
HNM can be used to repeatedly bootstrap negative samples, which are selected from the samples in the training dataset that are falsely classiﬁed as positive by the diagnostic model (Dalal and Triggs, 2005).
This simple but eﬀective technique is now commonly used in many applications in the medical domain to improve performance on diﬃcult negative instances such as in breast cancer WSI classiﬁcation (Bejnordi et al., 2017).
We follow the approach proposed by Li et al.
(2019) to incorporate a HNM system into our experiments.
Given the training bags X and learned attention scores a we select all l -th bags that have C l =
but were predicted as being positive after training.
From these false positive bags F L we then extract hard negative instances M by attention score ﬁltering:
[Equation]
where ¯ a l is the mean and σ l is the standard deviation of the attention scores of the l -th bag.
The collected hard negative examples can then be used to generate new hard negative bags which are mixed into the training dataset for a second training run.
The generation process is based on a feature extraction stage and a sampling stage.
For feature extraction, each hard negative instance in M is passed through a VGG16 deep convolutional neural network, pretrained on the ImageNet dataset (Simonyan and Zisserman,
2014), without its last softmax classiﬁcation layer, resulting in vectors of size 4096.
These feature vectors are then clustered with the k -means algorithm (Lloyd, 1982) into k clusters.
To enable the MIL model to better learn features comprehensively, a varied selection of instances into the new bags is used.
Thus, in the following sampling stage, the h -th new bag
B h is generated by randomly selecting instances from the clusters with a discrete uniform probability distribution.
The size of each new bag is limited, based on s min and s max , which are the minimum and maximum number of elements found in the original training bags, having a normal distribution based random size, with mean µ and standard deviation σ of all original training bag sizes.
3. Experiments Dataset
Cells are stained with hematoxylin and eosin (H&E) staining, before obtaining all whole-slide images with an Olympus BX61VS slide scanner, using the Olympus UPlanSApo 20x/0.75 NA objective lens.
For details on urine sampling and preparation of urine sediment cells, we refer to a previous publication (Krauß et al., 2018).
The dataset consists of 40 individual patients, with only a binary disease status (cancer vs normal) given as their annotation.
The distribution of classes is imbalanced with 27 patients exhibiting cancer and 13 patients of normal controls with urocystitis, an inﬂammation of the bladder.
Each WSI is approximately 20000x20000 pixels in size, with a resolution of 115 nm/pixel, containing hundreds and thousands of individual cells of diﬀerent cell types present in urine sediment.
Implementation Details
We follow previous works (Ilse et al., 2018; Li et al., 2019;
Sadaﬁ et al., 2020) and train the model in a similar fashion, using the Adam optimizer (Kingma and Ba, 2014) with default hyperparameter settings of β 1 = 0 .
9, β 2 = 0 .
999 and a learning rate of 0.001 for 150 epochs, where the model state at the epoch with the lowest training loss is chosen for validation.
For experiments with HNM a second training run with 100 epochs is conducted on the artiﬁcially enlarged training dataset.
In experiments without
HNM the previously saved model state is trained for another 100 epochs on the original dataset to ensure fair comparison.
The model is composed of a deep convolutional neural network feature extraction stages that operates on bags of instances with arbitrary size, the attention-based MIL pooling and a ﬁnal classiﬁcation multilayer perceptron.
The computed embedding H is multiplied with the just-in-time generated attention matrix a and ﬁnally classiﬁed on the bag-level.
The same training parameters were used across all experimental runs.
Previous publications dealing with MIL and attention often use datasets that lend themselves to very small patches, of up to 32x32 pixels.
Our extracted patches of urothelial candidate cells are of size 150x150 pixels with an average bag size of 300 instances.
As such the approach is implemented using model parallelism with a batch size of 1, putting the ﬁrst convolutional layer of the MIL model on its own GPU and then subsequently increasing the amount, using 4 NVIDIA V100 GPUs in total.
Further details as well as the code can be found publicly available at https://github.com/butkej/MIL4Cyto.
Baselines
To compare the performance of our proposed MIL system with attention and hard negative mining data augmentation loop, we choose diﬀerent baseline approaches in the
[Table]
form of the two most often encountered pooling operators mean and max, as well attentionbased pooling without further dataset augmentation.
All baselines are embedding-based
MIL neural networks.
Additionally we test against the best performing conﬁguration of Li et al.
[Table]
which uses a marginally altered attention mechanism with adaptive weighting to explicitly enlarge diﬀerences between positive and negative instances.
Metrics
We report the accuracy, F 1 score, area under the ROC Curve
[Table]
and false positive rate averaged across 5 runs to compare the diﬀerent approaches
[Table]
3.1 Results and Discussion
We perform 3-fold stratiﬁed cross validation, in which three independent models are trained and their results are averaged.
The data split is performed patient-wise on the full WSIs and resulting bags, in the same way for each run and method to enable fair comparison.
In Tablewe summarize the performance of ﬁve diﬀerent approaches on our dataset.
Additionally,
Fig.
2 shows the exemplary results of attention scores for diﬀerent types of cellular objects encountered in positive and negative bags.
Similar to its original introduction, the gated attention mechanism performs slightly better than normal attention.
Naive mean and max pooling perform the worst, often only predicting the majority class, thus also having a high FPR.
Using the attention-based pooling already performs remarkably better, as urothelial cells which are responsible for the bag label are attributed with high attention.
The additional usage of HNM then further reduces the FPR by constructing new hard negative training examples from normal urothelial cells that were attributed a higher than average attention score compared to other instances in their bag.
Other cell types and image artifacts that occur in extracted image patches are correctly attributed with low attention as they do not inform the bag label decision.
However, using the setting of Li et al.
(2019) results in signiﬁcantly weaker performance.
Including softsign activation + adaptive weighting is seemingly detrimental in our problem setting.
Instead our approach focuses on a modiﬁed selection of pseudo hard negative instances during HNM which achieves better results.
Finding the key instances that trigger the bag label is also highly important in the context of interpretability, which is an essential requirement of the currently evolving regulatory standards for good machine learning practice (Abels et al., 2019).
4. Conclusion
Our results demonstrate reliable classiﬁcation of patient-level disease status as well as the diﬀerentiation between normal and cancerous urothelial cells present in inﬂammations of the bladder or NMIBC.
The explicit identiﬁcation of relevant cells along with a score of attentiveness provides a degree of interpretability that could be a key for cytopathology as a clinical decision support.
This could be strengthened further by incorporating uncertainty estimates, e.g. by utilizing recent Bayesian dropout ideas (Gal and Ghahramani, 2016).
In terms of clinical relevance, our approach can be extended to facilitate multi-class diﬀerentiation, e.g. for grading or staging of tumors, possibly in combination with a multi-label setting regarding diﬀerent cell types on the instance level (Feng and Zhou, 2017).
We have introduced a variant of multiple instance learning that improves the performance of related recent approaches by a signiﬁcant margin.
Compared to other MIL cytopathology approaches, our contribution avoids the reliance on precise cell identiﬁcation, and uses a rather rough background separation and obtains bags of instances by centroid detection.
As our results clearly indicate, the combination of attention-based multiple instance learning with hard negative mining fully compensates the coarse grained instance identiﬁcation, leading to a signiﬁcant edge over the recently proposed MIL+attention as our baseline method.
Acknowledgments
This work was supported by the Ministry for Culture and Science (MKW) of North RhineWestphalia (Germany) through grant 111.08.03.05-133974 and the Center for Protein Diagnostics (PRODI).
Appendix A. Supplemental Material A.1 Urine sampling
Urine samples were collected from patients diagnosed with bladder cancer and also from patients with pathologically conﬁrmed urocystitis but without bladder cancer at the Department of Urology of the Ruhr University Bochum, Marien Hospital Herne, Germany.
Institutional review board approval (IRB 3674-10) and written informed consent from all patients were obtained.
A.2 Methods - Baselines
For the methods of MIL + mean as well as MIL + max the same approach as in our proposed contribution was performed.
Due to the missing attention scores no hard negative mining was used.
The MIL is based on the embedding approach and uses the same MLP classiﬁer after the pooling function σ .
For the mean pooling we thus have
[Equation]
and for the max pooling equivalently
[Equation]
Without the elementwise multiplication term the gated attention mechanism, becomes the standard attention mechanism, which is used in the MIL + attention method:
[Equation]
in which w and V are parameters that are learned during training.
A.3 Experiments - Implementation details
For all implementations and experiments Pytorch 1.6.0 was used as the main framework with additional usage of scikit-learn 0.23.2 and their respective dependencies.
Preprocessing was performed with OpenCV 4.4.0.
Further details as well as the code can be found publicly available at https://github.com/butkej/MIL4Cyto.