Seeded iterative clustering for histology region identification
Anonymous Author(s)
AffiliationAddress email
Abstract
Annotations are necessary to develop computer vision algorithms for histopathology, but dense annotations at a high resolution are often time-consuming to make.
Deep learning models for segmentation are a way to alleviate the process, butrequire large amounts of training data, training times and computing power.
Toaddress these issues, we present seeded iterative clustering to produce a coarsesegmentation densely and at the whole slide level.
The algorithm uses precomputedrepresentations as the clustering space and a limited amount of sparse interactiveannotations as seeds to iteratively classify image patches.
We obtain a fast andeffective way of generating dense annotations for whole slide images and a framework that allows the comparison of neural network latent representations in thecontext of transfer learning.
Introduction
Deep learning segmentation models are already being used for easing the burden of cancer histopathology whole slide image (WSI) annotation [22].
However, supervised training requires annotatedimages, often long training times and computational resources such as graphical processing units(GPU).
To overcome these bottlenecks, approaches such as active learning [20] are gaining popularity,as they reduce the time needed to perform the segmentation while retaining high performance viaexpert supervision.
One strategy is to have pathologists annotate a small part of the image, either byclicking or scribbling around the regions of interest.
Then, features are extracted from those sparselyannotated regions and the dense segmentation masks are obtained by retraining segmentation models.
This strategy, however, requires both interactive and GPU support at run time.
For this proof-of-concept, we instead propose seeded interactive clustering (SIC) to decouple theproblem by (1) precomputing the latent network representations, which does not require a GPU at runtime so it can be done remotely in a computing cluster without interaction and, once the embeddingsare obtained, (2) perform interactive clustering locally that does not involve any further training andis relatively cheaper computationally.
1.1 Previous work
Different authors have approached this problem by creating interactive tools based on extracting classical features [1, 11] or on deep learning approaches [12, 14, 16].
Some of these approaches are intended for cell detection on smaller images and therefore are harder to scale to bigger WSI.
The approaches that are meant for digital pathology usually require a GPU while performing the interactive annotations, which may not always be available
The re-purposing of convolutional neural networks (CNN) as feature extractors was proposed veryearly after the first models trained on large datasets were made avaialable [8, 18, 21, 23].
Since then,using model architectures such as ResNet-50 [10] trained on ImageNet has become common practicefor extracting features and performing transfer learning, even if some authors argue that these modelsmay not capture all the relevant histology features [4].
Recent work also proposes using self-supervised learning approaches for obtaining the embeddings [5,6], under the assumption that the feature space would be more representative of the tissue differences.
However comparing different embeddings is not a trivial task and is influenced by many factors [13,17].
Finally, image-set clustering is a field where all these concepts, including the choice of architectureand layer for feature extraction, or the clustering algorithms, has been extensively studied [9].
Method
2.1 Seeded iterative clustering
The proposed pipeline is presented in Figure 1 and has two parts: one remote and one local.
Thefirst remote part of the process starts by dividing the image into smaller patches.
The size of the patches, the stride between them and the resolution of choice defines the granularity of the final tissueclustering.
In this case, we decided to extract 256 × 256 patches with 50% overlap at 20 × resolution.
Next, each of the patches are embedded using the pretrained model of choice by extracting the lastlayer latent representation.
Finally, we store the matrix of number of patches × embedding sizetogether with the center coordinates of each patch to represent the complete WSI.
The second part of the pipeline can be run locally.
The seeded iterative clustering (SIC) approach,presented in Algorithm 1, divides the previously generated patch embedding space in a binary fashion,guided by small seeds -or sparse annotations- selected by the user in representative regions.
Thenseeded clustering [2] is performed iteratively until a score relative to the sparse annotations does notimprove.
We used F1-score, but any classification metric could be used to optimize the algorithm.
The seeded KMeans algorithm only differs from the original KMeans in the first assignment of clustercenters, which are not initialized randomly but based on the seeds.
An additional step of conventional
KMeans is added as it had shown to improve the performance when the tumor region of the WSI wasnot big enough.
Algorithm
X ← embeddings ▷ Embeddings of patches y ← seeds ▷ Sparse annotations score prev ←▷ Initialize score labels ← SeededKMeans ( n clusters = 2) .fit
( X, y )
▷ Initialize labels while score ≤ score prev do ▷ Run until defined score goes down
X ← embeddings [ labels == 1] ▷ Select data only in the postive class y ← seeds [ labels =
= 1] if { 0 , 1 } ⊂ y then ▷
If there are still annotations of both classes labels ← SeededKMeans ( n clusters = 2) .fit
( X, y )
▷ Re-cluster positive cluster elseif { 1 } ⊂ y then
▷ If there are only negative annotations labels ← KMeans ( n clusters = 2) .fit
( X ) ▷ Perform conventional clustering end if end if score ← metric ( y, label )
▷ Compare only sparse annotations end while
2.2 Feature space visualization
Inspired by the spatial-omics field, an intuitive way of visualizing the embeddings is by reducingthe dimensionality of the space to three dimensions using Uniform Manifold Approximation and
Projection for Dimension Reduction (UMAP) [15].
This 3D space can be mapped to the RGB spaceand visualized directly on top of the images as colored dots at the center of each patch in the image,as previously presented by Chelebian et al.
[3] and shown in Figure 1 (bottom-right).
The code fordoing this can be found at github.com/eduardchelebian/histology-umap.
In the UMAP space, it is possible to manually select regions that are clustered together and see if theycorrespond to biologically relevant regions on the image.
This approach however, is highly dependenton the dimensionality reduction algorithm and should be used only for qualitative visualizationpurposes.
Experimental results3.1 Dataset
For this proof-of-concept we used the DigestPath challenge cancer segmentation dataset [7].
It is anopen database including 250 images of tissue from 93 positive WSI (i.e. with pixel-level annotations)slides in JPEG format which were used for the experimental results.
The additional 410 imagesfrom 231 WSI negative slides were only used together with the positive slides when self-supervised pretraining was performed.
All WSI are hematoxylin-eosin (H&E) stained and scanned at 20 ×resolution.
3.2 Experiments
In order to explore how different embedding spaces would affect the clustering performance, we chosefour different feature extraction frameworks: (1) ResNet-18 pretrained on ImageNet (ResNet18),(2) ResNet-50 pretrained on ImageNet (ResNet50), (3) ResNet-18 pretrained on the dataset using
SimCLR self-supervision
[5] (SimCLR) and (4) ResNet-18 pretrained on 57 histopathology datasetsusing SimCLR self-supervision [6] (HistoSSL).
To mimic the complete workflow where the pathologist would select small benign and cancerousregions, we chose to experiment by randomly selecting 1, 3 and 5 patches that were located insidethe benign and cancerous masks, respectively.
Knowing that this would probably underestimate theresults, as the seeds may not be representative of what a pathologist would choose, we report the
F1-score for the patch classification task of the best run per slide in a 5-fold experiment, to simulatethe selection an expert would make.
The ground-truth coordinates are generated by selecting thepatches that are at least 50% in a cancer region.
3.3 Results
[Figure]
The results are summarized in Figure 2.
The highest per-slide F1-score for the ImageNet pretrainedmodels are very similar regardless of the architecure, 0 .
57 ± 0 .
22 for ResNet18 and 0 .
54 ± 0 .
21for ResNet50, both lower than the self-supervised approaches.
There seems to be some benefit forpretraining (SimCLR) on the dataset at hand ( 0 .
74 ± 0 . 23 ), but this may not be always possible, so it isvery promising that a publicly available model (HistoSSL) also achieves higher results ( 0 .
68 ± 0 . 22 ).
The method itself shows high classification variability, although the worse performing slides areusually the ones that have low positive patch ratio, that is, when the lesion is very small compared tothe whole tissue, such as Figure 3 (upper right).
There is no clear difference from having more seeds,except when it comes to the standard deviation.
As they are only used for initialization it seemsthat if the feature space is descriptive enough, adequately selected patches are sufficient for a goodclustering result.
Discussion
We presented seeded iterative clustering (SIC) as a way of leveraging latent representations of neuralnetworks to speed-up the time-consuming process of manual annotation in histopathology.
From an annotation tool point of view, the local clustering module takes seconds, even on a normal computer.
The embedding time depends on the amount of patches or coordinates one wants.
However, contraryto previous work, this can be run remotely without the pathologist interaction and does not need to berecalculated at any point.
Additionally, performing the clustering per-slide is more robust to domainshift artifacts.
From a technical point of view, it allowed to compare the representations from different networks.
When performing transfer learning, there is a separate effect of the feature reuse and learning low-levelstatistics from the data [17].
Retraining, then, may not be ideal to explore the quality of the features.
This is precisely why we find that ImageNet pretrained ResNet off-the-shelf are not comparable tousing self-supervised pretraining with SimCLR.
Work in progress includes validating this approach on other datasets and developing it into a fullinteractive tool called in which pathologists select a handful of patches and the method proposesclusters for the rest.
Additionally, the method needs to be compared against other approaches inorder to confirm its feasibility.
Finally, extending this approach in a multi-class problem (e.g. cancer,benign and stroma regions, instead of binary), would provide further insights into how the featurespace generated by the different models as well as increase the usability of the tool.
Broader Impact
[Figure] The uncoupling of the feature extraction and clustering modules could allow for the remote generationof embeddings that can then be shared with the community.
This would allow for groups with lesscomputational resources to work with the data and circumvent the potential privacy issues for sharingmedical images, sharing instead the latent representation matrices per coordinate.