T OWARDS BETTER UNDERSTANDING AND BETTER GENERALIZATION OF LOW - SHOT CLASSIFICATION IN HISTOLOGY IMAGES WITH CONTRASTIVE LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Low-shot learning is an established topic in natural images for years, but few work is attended to histology images, which is of high clinical value since well-labeled datasets and rare abnormal samples are expensive to collect.
Here, we pioneer the study of low-shot learning for histology images by setting up three cross-domain tasks that reﬂect real clinics problems.
To enable label-efﬁcient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build a few-shot system.
CL learns useful representations without manual labels, while LA transfers semantic variations of base dataset in an unsupervised way.
These two components fully exploit unlabeled training data and can scale gracefully to other label-hungry problems.
In experiments, we ﬁnd: i) models learned by CL generalize fairly better than supervised learning for histology images and ii) LA brings consistent gains over baselines.
Prior studies of self-supervised representation learning mainly focus on ImageNet-like images, which only present a dominant object in their centers.
Recent attention has been paid to images with multi-objects and multi-textures (
Chen Li, 2020).
Histology images are a natural choice for such study.
We show the superiority of
CL over supervised learning in terms of generalization for such data and provide our empirical understanding for this observation.
The ﬁndings in this work could contribute to understanding how model generalizes in the context of both self-supervised learning and histological image analysis.
I NTRODUCTION
Histological images provide crucial phenotypical and diagnostic information for disease assessment and prognosis (Srinidhi et al., 2020).
Their examination, which nevertheless requires intensive workload and extensive expertise, is an indispensable component in practical clinics.
Thus, computeraided histological image classiﬁcation systems are highly demanded but expensive to build due to the scarcity of well-annotated data.
Besides, histological images diversify in many aspects, including acquisition protocols, body sites and tissue types.
Such heavy domain shifts and variations pose more challenges and aggravate the data-hungry issues.
How to train robust models with limited access to annotated samples becomes the key of general diagnosis systems.
The aforementioned issues have motivated us to sort for low-shot learning , e.g. few-shot learning (FSL) and generalized few-shot learning (GFSL), which aim to learn new concepts from only limited labeled data.
Recent works have demonstrated the success of few-shot systems in natural images, but they are much unexplored in medical images, especially in histological ones.
Here, we pioneer the study of GFSL and FSL for histology images by setting up 3 cross-domain tasks where there exist near-, middle- and out-domain shifts from base class to novel class.
In addition, we also investigate the impact of homogeneous and heterogeneous shot selection, i.e. few-shot samples come from the same whole slide image (WSI) or different ones (more on this later), to few-shot system.
To enable label-efﬁcient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build a few-shot system.
Concretely, CL learns a meaningful encoder during pre-training stage without relying on label information, while LA inherits knowledge from “unlabeled” base dataset by transferring semantic variations in latent space.
These two components fully exploit a base dataset by using its legacy: the learned model weights, and the captured latent variations.
[Figure]
Also to our surprise, we show that state-of-the-art CL models generalize fairly better than the supervised counterparts for histological images.
Previously studies of CL mainly focus on iconic natural images where only a dominant object occupies image centers, while images like histological ones, where multiple small objects (e.g. cells, nucleus) and various textures (e.g. muscle, mucus) are densely presented, are largely unexplored.
We also aim to ﬁll the gap of studying CL for non-iconic, multi-object and multi-texture images, and empirically explain why the large generalization gap between CL models and supervised ones exists for such images.
To summarize, our key ﬁndings and contributions are:
• We pioneer the study of low-shot classiﬁcation problem for histological images.
Investigations w.r.t.
domain-speciﬁc problems,
i
e. domain shifts across tissue types and body sites, heterogeneous and homogeneous few-shot samples selection, are conducted.
• We propose a simple label-efﬁcient method for low-shot learning, which incorporates contrastive learning and latent augmentation to fully exploit training data in an unsupervised way.
Extensive experiments conﬁrm their consistent gains and improved generalizability.
• We show that, at slight odds with ﬁndings in iconic natural images, contrastive learning learned models generalize fairly better than supervised counterparts for non-iconic, multiobject and multi-texture histology images.
We analyze and provide our empirical explanation for this observation, which we believe could contribute to understanding how model generalizes to
novel samples in the context of contrastive representation learning and histological image analysis.
P RELIMINARIES AND PROBLEM FORMULATION
Whole-Slide Image (WSI).
Whole-slide images are digital scans of pathology tissue slides collected by biopsy or surgery.
Given micron-size pixels and centimeter-size slides, a WSI is usually of Gigapixel size and thus have to be divided into hundreds or thousands of small “patches” for computational analysis.
A patch reﬂects the basic unit in patch-level classiﬁcation problem.
As part of tissue processing protocols, a slide needs to be stained before imaging for diagnosis.
The tissues context and staining quality could vary across WSIs, so the extracted patches’ styles are confronted to their source WSIs, leading to inter-WSI domain shift.
Besides, unlike iconic natural images which only present a dominant object in their centers, histological patches contain multiple small objects and multiple texture-like tissues (see Figure 1).
To diagnose, a pathologist needs to inspect both the local cellular details and the overall global patterns, making the process different from the major recognition systems which only need to focus on dominant objects.
Few-shot Learning (FSL).
Few-shot classiﬁcation is to learn from a large “base” dataset, and afterwards generalize to unseen classes with only limited labeled data.
Formally, a base dataset is deﬁned as D base = { ( x i , y i ) } N base i = ⊂
X base × Y base , where X base is a sample set and Y base is their label space.
Novel dataset D novel =
{ ( x i , y i ) } N novel i = has a disjoint label space, i.e. Y base ∩ Y novel =
Ø where Y novel is the novel label space.
A few-shot learner is trained on D base and evaluated on a series of meta-tasks sampled from D novel .
Such task is deﬁned as T =
{ ( S i , Q i ) } Ii =1 , where
S = { ( x i , y i ) } NKi =1 ∼ D novel is a small training set, called support set, and Q = { x i } NQi =1 ∼ X novel is another small test set, called query set, and I is the number of tasks.
This formulation is termed as N -way K -shot ( Q -query) task, since N classes are sampled from Y novel , each with K labeled samples for training and Q unlabeled samples for testing.
Usually, K is less than Q , e.g. K = 1 , 5 , and Q = 15 .
The evaluation stage is often referred as meta-testing stage.
Generalized few-shot learning (GFSL).
Unlike in FSL, GFSL samples meta-tasks from a joint dataset D joint = D base ∪ D novel , with a joint label space
Y joint =
Y base ∪ Y novel .
Now, support sets and query sets contain both seen base classes and unseen novel classes.
M ETHODS
Consider a few-shot classiﬁer as f = f θ ◦ f φ 1 , where f φ is a embedding function, i.e. a feature extractor, that maps the high-dimensional input image data x into a low-dimensional latent space R d , and f θ is a classiﬁer trained on the support set S and predicts the results for query set Q .
φ and θ are their corresponding parameters.
Our method consists of two phases — a) pre-training f φ on base datasets and b) training f θ on support sets with latent augmentation during meta-testing stage.
Figure 2 shows the overview of our methods.
We elaborate them in the followings.
3. P RE - TRAINING
Current paradigms in FSL for training f φ lie in two folds: i) meta-training, also known as episodic training, where base datasets are divided into various episodic N -way K -shot meta-tasks that simulate meta-learning; and ii) standard training, which do standard fully supervised classiﬁcation pretraining without splitting data.
The former one emphasizes the idea of meta-learning, i.e. “learning to learn” for fast adaption and generalization (Schmidhuber, 1987;
Finn et al., 2017), while the latter one attributes the success of FSL to feature reuse (Raghu et al., 2019) and good representations (Chen et al., 2019; Tian et al., 2020).
We follow the latter for simplicity and we believe better learned encoders lead to stronger generalizability.
Figure 2-(a) shows the pre-training step.
Fully-supervised pre
-training (FSP).
Given a base dataset D base = { ( x i , y i ) } N base i =1 , we jointly train the feature extractor f φ and a proxy classiﬁer f ψ using standard cross-entropy loss:
[Equation]
The proxy classiﬁer f ψ is then deprecated for downstream evaluation, while the feature extractor f φ is kept and ﬁxed for all tasks 2 .
We term the embedding functions learned by FSP as f FSP φ .
Contrastive-learning pre
-training (CLP).
Self-supervised learning methods alleviate the need for data annotations.
Here we focus on a contrastive learning method – MoCo-v3 (Chen et al., 2021b), which currently holds state-of-the-art performance.
It consists of three components: a feature extractor (backbone) f φ , a projection head f g and a prediction head f q .
Given an unlabeled base training dataset D ubase = { x i } N base i =1 , the model learns to minimize the contrastive loss function w.r.t.
unlabeled batch data:
[Equation]
where L CLP is a contrastive loss function; x , x ′ are two views of the same images obtained by applying random data augmentation t ; ˜ φ and ˜ g denote the momentum updated copies of φ and g .
In short, contrastive learning aims to maximize the similarity between positive pairs (two augmented views of a same image), while minimize the similarity between negative pairs (two different images).
We leave more detailed descriptions of MoCo-v3 and L CLP to Appendix B. Once CLP is done, two auxiliary heads, f g and f q , are deprecated, while f φ is kept and ﬁxed , termed as f CLP φ .
3. L ATENT A UGMENTATION
The pre-trained feature extractor f φ only transfers parts of available knowledge in base datasets by reusing the learned weights.
The more transferable knowledge is inherent in data representations .
It is reasonable to assume that base classes and novel classes share similar modes of variations (Wang et al., 2018) since they are all pathology-related.
Such inductive biases allow us to transfer variations from seen tissues or styles to unseen ones.
Here we propose to transfer the representation variations in a simple unsupervised way.
Below, we ﬁrst introduce latent augmentation and then discuss our motivations and intuitions about it.
DictionaryConstruction a Pre-training ˜
z
f ϕ z
LA f θ
Low-shot Classification b Learn with LA
Unlabeled WSI patches z
Contrastive Learning f ϕ or other proxy tasks f
* f ϕ f θ
Prediction c Inference d Latent Augmentation (LA)
Legacy Dictionary ℬ z
K-Means
N-way K-shot patches
WSI patches to predict
[Figure]
Legacy dictionary and Latent augmentation (LA).
Our goal is to fully exploit training data.
This is done not only by reusing pre-trained model weights f φ , but also by transferring possible semantic shifts of clustered representations.
Given an unlabeled base dataset, we perform K-Means on the representations z
= f φ ( x ) to obtain C clusters (Figure 2-(a), red arrows).
The legacy dictionary is constructed as B = { ( c i , Σ i ) } Ci =1 , where c i is the i -th cluster prototype (i.e. average representation) and Σ i denotes its intra-cluster covariance matrix.
Roughly, B captures how the pre-trained f
φ
thinks samples from base dataset will vary in latent space conditioned on cluster i , e.g. using a multivariate Gaussian N ( c i , Σ i ) .
Given the legacy dictionary B , during meta-testing stage, LA uses original representations z to query the most likely variations from B , followed by additive augmentation ˜
z
=
z + δ (Figure 2-(c)).
This is done by sampling δ ∼ N ( 0 , Σ i ∗ )
where
[Equation]
/ ’ z ’’ c i ’ .
The classiﬁer f θ is then trained on both the original representations z and the augmented representations
˜ z
(Figure 2-(b)).
3.2. I NTUITIONS AND MOTIVATIONS ON LATENT AUGMENTATION
Why transferring variations works.
LA aims at transferring the knowledge of variations.
Such knowledge brings semantic diversity from base class to novel class.
For example, tumorous cells are mutated from normal cells; when given limited tumorous samples, LA can replicate how normal cells alter under this mutation using the captured variations in legacy dictionaries.
This closely resembles how a pathologist builds his/her knowledge on unseen phenotype from seen phenotype.
From the view of under-representation learning (Yin et al., 2019), replicating latent variations encourages under-represented distributions to be closer to regular ones.
From the view of low-data learning, distribution of few samples is not well calibrated (Yang et al., 2021), so using established distributions in base class to calibrate untrustworthy novel class can help.
Besides, LA can be seen as a consistency regularization technique.
Enforcing the classiﬁers’ predictions to be consistent across different perturbation is known to help in low-data regime (Bachman et al., 2014; Berthelot et al., 2019; Sohn et al., 2020).
In fact, LA is a stronger alternative against data augmentation (DA), as we later show in § 4.3 that LA outperforms DA by a large margin and can cover the role played by DA.
Why linear additive augmentation is meaningful.
Deep representation space is shown to be highly linearized (Bengio et al., 2013; Upchurch et al., 2017), demonstrating the rationality behind linearly interpolating or extrapolating features, i.e. using “add” operation to generate new features.
Why legacy dictionary construction is warranted in both FSP and CLP.
FSP uses classiﬁcation task as a proxy task to learn useful encoders f φ .
During optimization, features are incentivized to maximize their dot-product similarity with class weights in f ψ , thus forming meaningful metric space.
In the regard of CLP, the contrastive loss, a form of metric-based loss, brings similar features closer while spreading dissimilar representations farther, which also results in an informative metric space.
Thus, feature distance in the representation space of both FSP and CLP is meaningful, justifying the use of unsupervised clustering method to construct legacy dictionary.
E XPERIMENTS 4. S ETUP
To better study low-shot classiﬁcation problem for histological image, we use three public histology datasets to construct three tasks with different degrees of domain shift, elaborated as below.
Datasets.
Since tissue type and appearance vary across body sites, we use three datasets from different body sites.
They are: NCT-CRC-HE-100K dataset collected from colon site (Kather et al., 2018), LC25000 dataset collected from lung and colon sites (Borkowski et al., 2019), and PAIP dataset collected from liver site (Kim et al., 2021); we term them as NCT, LC-25K and PAIP respectively.
NCT consists of 9 classes with 100k non-overlapping patches in total, each of size 224 × 224 .
LC-25K has 5 classes with 5,000 patches in each class; each patch is of size 768 × 768 .
PAIP is composed of 50 WSIs, each of size about 45k × 45k with 3 mask annotated classes.
For LC-25K, all patches are resized to 224 × 224 .
For PAIP, large images are cropped into patches of size 224 ×with the same pixel resolution as NCT.
Only patches containining foreground tissues are kept (75k patches) and their labels are assigned by majority voting.
More details are in Appendix A.
We deﬁne three tasks to better reﬂect real-world scenarios: i) Near-domain task (to study GFSL).
NCT bears slight domain shift since samples are collected by various organizations with different staining intensity
We randomly split NCT by 80%/20% to construct a training set (80k images) and a test set (20k images).
Then, we do leave-one-classout-as-novel-class to the training set to construct 9 base datasets and use the test set as D joint for evaluation
This results in 9 sub-tasks, each of which has one class regarded as novel class and samples belong to it are excluded from pre
-training
datasets
See Appendix A.1 for more details.
ii) Mixture-domain task (to study FSL).
LC-25K collects images from lung and colon sites.
Two of ﬁve classes in LC-25K are colon-related, which suffers moderate domain shift due to difference in imaging protocols and pixel resolutions; we see them as middle-domain novel classes.
The rest three lung-related classes are regarded as out-domain novel classes.
We use the training set of NCT as D base , and LC-25K as D novel .
See Appendix A.2 for more details.
iii) Out-domain task (to study FSL).
Liver has different tissue organization and cell types compared with colon.
Similar to mixture-domain task, we use the NCT training set as D base and PAIP as target dataset D novel .
To study heterogeneous and homogeneous shot selection problem, we use WSI ID information to split PAIP into a support WSI set (15 WSIs with 22.5k images) and a query WSI set (35 WSIs with 52.5k images).
During evaluation, support samples and query samples are drawn from the support WSI set and the query WSI set respectively.
Heterogeneous strategy selects few-shot samples from different support WSIs, while homogeneous strategy selects them from a single randomly chosen support WSI.
See Appendix A.3 for more details.
Evaluation.
If not otherwise speciﬁed, for near-domain task, we evaluate methods in 1000 ×(9 sub-tasks) random meta-tasks; for mixture- and out-domain task, we evaluate methods inrandomly sampled meta-tasks.
All meta-tasks use 15 samples per class as the query set.
We report the average F1-score and 95% conﬁdence interval.
To handle the unequal numbers of base classes and novel classes, we follow convention in GZSL (Xian et al., 2018) and GFSL (Shi et al., 2020) to report their average harmonic mean.
More details about evaluation metrics are in Appendix A.
Implementations.
I. Pre-training.
We use ResNet-18 as the embedding function f φ , and follow previous arts in FSL (Tian et al., 2020; Chen et al., 2019) to use l 2 -normalized features for clustering and downstream meta-tasks.
II.
Latent Augmentation.
We use faiss (Johnson et al., 2019) to perform K-means with a ﬁx seed.
The number of prototypes in legacy dictionary is 16 ( C = 16 , discussed in ablation § 4.3).
In each meta-task, each sample is augmented 100 times (including the original one) by LA.
We defer more implementation details of pre-training and LA to Appendix C.
Compared methods.
Recent works (Chen et al., 2019; Tian et al., 2020), including a concurrent work for histology image (Shakeri et al., 2021), show that sophisticated episodic training (metatraining) is no better than standard pre-training.
Hence, we summarizes methods using standard pre-training as: 1) NearestCentroid.
It computes class centroids from support sets and classiﬁes query samples to their nearest centroids.
Related works using such strategy includes Wang et al.
(2019a), Snell et al.
(2017), and Chen et al.
(2020c)
, to list a few; 2) LinearClassiﬁer.
It trains a new
[Table]
fully-connected layer with different loss functions (Chen et al., 2019;
Lee et al., 2019) w.r.t.
support samples, or directly uses linear models in scikit-learn (Pedregosa et al., 2011), e
[Table]
g. LogisticRegression (Yang et al., 2021; Tian et al., 2020).
For the ease of implementation and consistency, we use
NearestCentroid, and two l 2 -regularized linear classiﬁers — LogisticRegression and RidgeClassiﬁer, all from scikit-learn APIs (Pedregosa et al., 2011).
4. M AIN R ESULTS
Main results in near-domain task and mixture-domain task are presented in Table 1.
We discuss the overall ﬁndings here and leave the discussion of design choices to ablation study in § 4.3, and the choice of linear classiﬁers (i.e. LogisticRegression/RidgeClassiﬁer) to § 4.4.
Fully-supervised f FSP φ v.s. Self-supervised f CLP φ .
Results in Table 1 show that CLP generalizes fairly better to novel classes than FSP.
Comparing the best vanilla entries (w/o.
LA) using two types of pre-training methods, we observe an average advantage in HarmMean of 4%, 5% and 8% in 1-/5- /10-shot settings by CLP in near-domain task and 10%, 19%, 16% in mixture-domain task.
Besides,
CLP representations beneﬁt more from the increase of number of shots than FSP’s in both tasks, e.g.
+17% vs. +11% and +12% vs. +10% when 1-shot → 5-shot for linear classiﬁers in near-domain, and mixture-domain tasks respectively.
Despite the inevitable advantage of FSP in base classes under full supervision, CLP demonstrates stronger generalizability compared to FSP.
Furthermore,
Table 2 also conﬁrms the superiority of CLP over FSP in out-domain task where large domain shift exists.
Such generalization gap between FSP and CLP in pathology images is at slight odds with observations in natural images, where they show similar generalizability.
We study and discuss it at § 4.4.
We also provide the linear evaluation results of all the 20 pre-trained models in Appendix C. for a reference to see how each model perform in NCT dataset.
Latent augmentation brings consistent improvement.
Regardless of pre-training methods, LA brings consistent gains over baseline linear classiﬁers, conﬁrming its effectiveness.
With legacy dictionaries, limited few-shot samples are able to proliferate in a reasonable way by transferring latent variations.
Such boost maintains its signiﬁcance from near-domain task to mixture-domain task (Table 1) but becomes humble in out-domain task (Table 2).
This is in our expectation since the three
A v e r a g e F 1 - s c
o r e s ( % )[-
-]
[Figure] [Figure] Number [-of prototypes  8 16 32 64 
-][Figure]
HM
[Figure]
Base-LA
Novel-LA HM
Base
Novel
A v e r a g e H a r m o n i c m e a n o f
F 1 - s c o r e ( % )
[Figure]
LA
LA × DA
DA[-2 2 4 2 6 2 8 2 10 2 12 2 14 
 [Figure] [Table]-]
[Figure]
[Table]
classes (non-tumor, viable-tumor and other) deﬁned in PAIP are extremely coarse-grained 4 : it may include couples of real ﬁne-grained classes.
Few samples are unable to well represent their entangled semantics.
Hence, this observation does not repudiate the effectiveness of latent augmentation but re-ensures its tenability.
4. A BLATIONS
To ablate design choices, we exclude two cancer-related classes, i.e. cancer-associated stroma (STR) and colorectal adenocarcinoma epithelium (TUM), from NCT to be novel classes, and use the rest as base classes.
All ablations are conducted on CLP backbones with RidgeClassiﬁer if not speciﬁed.
Number of prototypes in legacy dictionary.
We do this ablation in 5-shot setting.
Figure 3 (a) shows how the number of prototypes, i.e. C , affects recognition performance.
We observe the similar tendency between base class and novel class, where their harmonic mean peak at C = 16 ; we subsequently choose C = 16 for all experiments.
Besides, the performance of base classes and novel classes shows opposite trends from C = 4 to C = 16 .
The trade-off exits here that as the granularity of clusters increases ( C ↑ ), the intra-cluster variance decreases, which results in better grouping accuracy but brings less semantic variation.
The novel classes beneﬁt from larger variation while the base classes beneﬁt from more accurately estimated variation since they have been exposed in training.
Nevertheless, LA demonstrates its robustness by consistent improvement over baselines (solid lines vs. dashed lines of same color in Figure 3-(a)).
DA vs. LA, and number of augmentation times.
Consistency regularization helps in low-data regime as discussed in § 3.2.1.
Here we compare LA with data augmentation (DA), and their combination.
Results are shown in Figure 3-(b).
It can be seen that LA outperforms DA by a large margin.
The boost brought by DA saturates easily and keeps dropping thereafter, while LA keeps improving with all tested cases.
Besides, DA can slightly polish LA (LA × DA v.s. LA).
We conclude that LA has already covered the role played by DA in an implicit way since the most of gains are brought by LA.
It is worthy to emphasize the computation budget involved in LA (addition in R d space) is signiﬁcantly lower than LA (image augmentation in R 3 HW space and encoder forwards).
Therefore, we run all experiments only with unsupervised LA, as one of our core contributions go with it, and its combination with DA is beyond the scope of this work.
Heterogeneous v.s. Homogeneous patch selection.
In real clinics, WSIs’ appearance varies across patients.
Patches from a single WSI are confronted to its stain degree (impacts color) and f CLP ϕ f FSP ϕ k= k= k= k= k= k=
Novel Class
Low
Middle
High
Abs.
Sim.
Rel.
Sim.
Abs.
Sim.
Rel.
Sim.
Similarity:
Clusters: 2 3 4 5  body site (impacts pattern and context).
We therefore investigate the hetero-/homo-geneous patch selection strategies as deﬁned in out-domain task ( § 4.1).
Table 2 shows the results.
We observe: i) heterogeneous selection is favored over homogeneous one, as it shows higher baselines; and ii)
LA brings more gains for heterogeneous selection over baselines.
Heterogeneous few-shot patches provide reliable and more diversiﬁed “anchor” samples than homogeneous ones, thus can beneﬁt more from bootstrapping the legacy dictionary.
4. M ORE DISCUSSION
Disparity between f CLP φ and f FSP φ inﬂuences the choice of base learner.
In Table 1, we ﬁnd i) stronger baselines for CLP and FSP vary, and ii) simple NearestCentroid can sometime outperform the vanilla l 2 -regularized 5 linear classiﬁers for FSP.
Here we brieﬂy discuss our understandings.
Representations produced by CLP can have different distributions compared to FSP, as also noticed by He et al.
With limited training samples, different classiﬁers can have their own biases in building decision boundary, leading to different generalizability.
Besides, in our implementation, no regularization techniques are used during FSP, e.g. weight decay (Krogh & Hertz, 1992), DropBlock (Lee et al., 2019; Tian et al., 2020) or “distill” regularization (Tian et al., 2020).
The linear classiﬁers, though with l 2 penalty, may still be overﬁtted in such non-regularized representation space when only limited samples are provided.
Subsequently, the simplest NearestCentroid model can yield better results than these overﬁtted linear models, as it has the least complexity.
Why do CLP models generalize better than FSP ones in pathology?
To study why the large generalization gap exists, we follow Chen & Li (2020) to see how features are aggregated in space.
Speciﬁcally, we run K-means on features from different layers (i.e. stage 2, 3, 4 of ResNet) with different cluster numbers (Figure 4).
The CLP model extracts low-level features that are edge-related, and afterwards successively agglomerates adjacent similar structures (dashed boxes in Figure 4).
In contrast, the FSP model can differentiate nuclei in low- and middle-levels but fails to encode tissue-structure-related features in deeper layer.
These ﬁndings differ from those in ImageNet-like images (Chen & Li, 2020), where FSP and CLP show no difference across layers 6 .
We also visualize the similarity between global features and local ones.
The FSP model maintains global-local consistency, i.e. high similarity between global&local features, in low-/middle-levels, while the CLP model holds it in high-level (solid boxes in Figure 4).
Unlike iconic natural images which only present a dominant object in centers, histology patches contain multiple small objects as well as texture-like tissues.
The lack of dominant objectiveness makes FSP tend to overﬁt severely in high level, but CLP can capture the overall patterns from the crudely scattered tissues.
We believe this is the main reason why CLP models generalize better than FSP models, in pathology.
We provide detailed visualization procedures and more examples in Appendix D.3.
R ELATED W ORK
Few-shot learning (FSL) and generalized few-shot learning (GFSL).
FSL has been tackled from different perspectives, e.g. metric-based and optimization-based (Finn et al., 2017; Rusu et al., 2018; Munkhdalai & Yu, 2017).
Here we focus on metric-based ones.
They aim to learn taskdependent metric space by “meta-training”, where base datasets are divided into many meta-tasks during training, and predictions are made by comparing distance between support samples and query samples (Gidaris & Komodakis, 2018;
Snell et al., 2017; Vinyals et al., 2016).
Later methods is attracted to transfer learning , i.e. pre-training & ﬁne-tuning , which learns shared metric space by standard fully-supervised pre-training (Tian et al., 2020; Chen et al., 2019; Wang et al., 2018).
GFSL is usually studied in the context of low-shot learning (Wang et al., 2018; Hariharan & Girshick, 2017), with adopted meta-learning few-shot methods (Gidaris & Komodakis, 2018; Shi et al., 2020).
This paper follows “pre-training & ﬁne-tuning” scheme for both FSL and GFSL, but we propose to incorporate self-supervised pre-training to enable label-efﬁcient learning, and show that it can yield stronger generalization than supervised pre-training.
FSL in medical images.
FSL in medical images is at its early stage, especially for histological images.
Mahajan et al.
(2020) investigate FSL methods in skin-disease classiﬁcation, while Chen et al.
(2021a) tackle COVID-
CT image classiﬁcation using contrastive pre-training and prototypical network ﬁne-tuning.
In the regard of histology image, Medela et al.
(2019) use a triplet loss (Schroff et al., 2015) to pre-train an encoder with a followed ﬁne-tuned SVM classiﬁer for low-shot domain adaptation.
Concurrent to our work, Shakeri et al.
(2021) simultaneously propose a benchmark for few-shot classiﬁcation of histological images.
Our work has similar but different settings, with more investigations conducted, e.g. GFSL task, and hetero-/homo-geneous few shots selection.
Self-supervised learning.
Self-supervised learning aims to learn good representations without golden labels.
Recent state-of-the-art variants can be categorized as contrastive-based learning (Chen et al., 2021b; 2020a;
He et al., 2020), cluster-based learning (Caron et al., 2018; 2020) and expectation prediction based learning (Grill et al., 2020; Chen & He, 2021).
However, this line of works focus on pre-training on ImageNet-like images, and recent attention has been attracted to images with multi-objects and multi-texture presented (Chen & Li, 2020).
We see histology image as a natural choice for such study, and show that contrastive learning can agglomerate structural “part-whole” information and maintain “global-local consistency”, which make it generalize better for such data than supervised counterparts (see § 4.4 and Table C.1 in Appendix C.3).
Representation variation augmentation.
The idea of exploiting representation variations dates back to a decade ago (Heller et al., 2009; Salakhutdinov et al., 2012) or more earlier.
Recent variants further develop this idea.
For example, Hariharan & Girshick (2017) use a generator to generate “hallucinated” novel features from variation of given base samples.
This method is later extended by not relying given base samples (Wang et al., 2018).
Wang et al.
(2019b) use class variance to perform semantic augmentation for classiﬁcation and segmentation (Wang et al., 2021), while Yin et al.
and Liu et al.
(2020) utilize intra-class variance of head classes to augment tail classes for “long-tail” face recognition problem.
Yang et al.
(2021) use the distribution information, i.e. mean and variance, of base classes to calibrate novels’ distribution.
This work follows the line of
Yang et al.
; Wang et al.
Liu et al.
(2020), but in contrast to them, we obtain and transfer variations without relying on any label information, allowing our method to scale gracefully to other label-hungry problems.
C ONCLUSION
This work has studied, as an early attempt, the low-shot classiﬁcation problem for histological images.
We incorporate contrastive learning and latent augmentation to fully exploit training data in an unsupervised way, which means our method can gracefully scale to other large label-hungry problems.
More importantly, we show that state-of-the-art contrastive learning pre-training method can generalize fairly better than supervised pre-training for histological images, which is in slight contrast to observations in iconic natural images.
We analyze the underlying reasons and provide our empirical understandings.
We hope our work could contribute to the study of representation learning and generalization for both self-supervised learning community as well as histology image analysis community.
A D ATASET A. NCT
Data.
NCT-CRC-HE-100K dataset (Kather et al., 2018) contains 100 , 000 non-overlapping patches extracted from hematoxylin and eosin (H&E) stained human colorectal cancer and normal tissues.
Each image is of size 224 × 224 at 0.5 MPP ( 20 × magniﬁcation).
Tissue classes and their label index are: 0) background (BACK), 1) adipose (ADI), 2) debris (DEB), 3) lymphocytes (LYM), 4) mucus (MUC), 5) smooth muscle (MUS), 6) normal colon mucosa (NORM), 7) cancerassociated stroma (STR), 8) colorectal adenocarcinoma epithelium (TUM).
Figure A.2 shows the class distribution and Figure A.1 shows 4 examples per class from NCT.
These images are from National Center for Tumor Diseases (Heidelberg, Germany) and University Medical Center Mannheim (Mannheim, Germany).
Their acquisition protocols differ across organizations, which lead to intersource domain shift.
Processing.
We use the “NCT-CRC-HE-100K-NONORM” dataset, which does not apply color normalization to images.
We randomly split NCT dataset (100k images) into a training set (80k images) and a test set (20k images).
The class distribution is the same for the training set and test set, i.e. data are sampled w.r.t.
each class.
Since each image in NCT is of size 224 × 224 , we do not further resize it.
Near-domain task.
NCT dataset contains tissue patches from multiple sources, which bears slight domain shift since the staining intensity varies.
To better study the generalizability in FSL problem, we do leave-one-class-out-as-novel-class to the training set of NCT.
This procedure closely resembles the leave-one-out cross-validation.
Since pathology datasets usually have far less number of classes compared to natural image dataset (e.g. 9 classes in NCT v.s. 100 classes in mini ImageNet and 608 classes in tiered ImageNet), we believe such leave-one-class-out-as-novel-class can better reﬂect the generalization error bound when number of classes is small.
After the leave-one-classout-as-novel-class split, 9 sub-base datasets are constructed, each of which has one class regarded as the novel class and samples belong to it are excluded from pre-training.
For each sub-base dataset, a CLP model and a FSP model will be trained on it.
Therefore, a total of 18 models will be obtained (9 CLP models and 9 FSP models).
All models are evaluated on the NCT test set.
It contains both seen classes and unseen novel classes, so as to simulate generalizes few-shot classiﬁcation.
Evaluation metric.
Typical FSL methods use accuracy as metrics.
However, to study GFSL problem, the performance of novel classes and base classes should be appropriately considered.
Accuracy w.r.t.
speciﬁc classes cannot be computed in a joint label space.
Hence, we choose F1-score as our metric.
Besides, the numbers of novel classes and base classes are unequal in each sub-task, so we follow convention in GFSL to compute their harmonic mean.
The ﬁnal metrics are
[Equation]
where F 1 ( j ) denotes the F1-score of j -th class.
M base ( i ) (Equation A.1) computes the average
F1-score of base classes in i -th sub-task.
Equation A.2 and Equation A.3 compute the F1-score of novel class and the harmonic mean of M base and M novel in i -th sub-task respectively.
Equation A. shows the ﬁnal metrics we report in this work.
A. LC
Data.
The LC25000 (LC-25K) dataset contains 25,000 color images from 5 classes.
They are: 0) colon adenocarcinoma (Colon ACA), 1) benign colonic tissue (Colon benign), 2) lung adenocarcinoma (Lung ACA), 3) benign lung tissue (Lung benign), and 4) lung squamous cell carcinoma
[Figure]
Processing.
We resize all images from 768 × 768 to 224 × 224 .
No other process is taken.
Mixture-domain task.
LC-25K dataset includes tissues from colon and lung sites.
Two of ﬁve classes in LC-25K are colonic tissues, which suffers moderate domain shift due to difference in imaging protocol and pixel resolutions.
Thus, we see them as middle-domain novel classes.
The rest three lung-related classes are regarded as out-domain novel classes since they are from a different oragan.
To study few-shot classiﬁcation, we use the entire training set (80k images) of NCT as
D base , and use LC-25K as D novel for evaluation.
Evaluation metric.
Similar to near-domain task evaluation in Appendix A.1, we compute
Middle = 12 ( F 1 (0) + F 1 (1)) , Out = 13 ( F 1 (2) + F 1 (3) F 1 (4)) , HarmMean = 2 1 / Middle+1 / Out .
Here F 1 ( i ) denotes the F1-score of i -th class.
Class 0 and class 1 are colon-related, termed as middle-domain novel classes, and the rest three classes are lung-related, termed as out-domain novel classes.
A. PAIP
Data.
The PAIP 2019 (Kim et al., 2021) is composed of 50 H&E stained WSIs at 20 × magniﬁcation.
Each WSI is approximately of size 45 , 000 × 45 , 000 with XML annotation.
It contains three
[Figure]
classes, which are: 0) non-tumor liver tissue, 1) viable tumor, and 2) other tissues in whole tumor area but not viable tumor.
The class 2 may include intratumoral hemorrhage, necrosis or non-tumor tissue in whole tumor area.
Figure A.4 shows an example WSI of PAIP19.
This dataset provides “case number” (WSI identiﬁer) information.
Processing.
We use mask information to extract 500 randomly selected non-overlapping patches at 20 × magniﬁcation for each class from each WSI, leading to a dataset with 75 , 000 samples.
We deﬁne patches from 15 randomly selected WSIs as the support WSI set and patches from the rest
WSIs as query WSI set.
[Figure]
B C ONTRASTIVE LEARNING P RE - TRAINING : M O C O - V
MoCo-v3 (Chen et al., 2021b) aims for an empirical study for vision transformer pre-training, but has a ResNet variant.
We adopt
Overview.
MoCo-v3 follows its ancestors MoCo v1/2 (He et al., 2020; Chen et al., 2020b) with straightforward modiﬁcation.
As shown in Figure B.1, each batch is augmented twice under stochastic augmentation t to obtain two views, denoted as x 1 , x 2 ∼ t ( x ) , where x represents the raw im- ages.
They are then encoded by query encoder and key encoder respectively.
The query encoder is composed of three components: a backbone f φ , a projector (projection head) f g and a predictor (prediction head) f q , formulated as f q ◦ f g ◦ f φ .
The key encoder, also known as momentum encoder, consists of momentum copies of backbone and projector, formulated as f ˜ g ◦ f ˜ φ .
[Figure]
[Equation]
[Figure]
Momentum update.
In iteration k , the momentum update rule is:
[Equation]
where m is the momentum.
Loss function.
The contrastive loss function is a form of InfoNCE (Oord et al., 2018):
[Equation]
Here, v + denotes the positive sample, i.e. the other augmented view of u , and v − denotes the negative samples, i.e. other representations in the batch.
τ is a hyper-parameter, known as temperature (Wu et al., 2018).
The ﬁnal loss is a symmetric sum: L CLP =
[ L ctr ( z 1 , ˜ z 2 ) + L ctr ( z 2 , ˜ z 1 )] / 2 , where the subscript indicates view source, z = l 2 [ f q ◦ f g ◦ f φ ( x )] , ˜ z
= l! f ˜ g ◦ f ˜ φ ( x ) $ , and l 2 [ · ] means l 2 normalization.
C I MPLEMENTATION D ETAILS C. F ULLY - SUPERVISED P RE - TRAINING
Optimization.
We use SGD optimizer with lr = 0 . 5 , momentum = 0 .
9 , and no weight decay is used, i.e. weight decay=0.
The batch size is 512.
We train for 100 epochs with “step decay” learning schedule.
The lr is multiplied by 0.1 at 30, 60 and 90 epochs respectively.
Data augmentation.
We use RandomResizedCrop, RandomHorizontalFlip , followed by normalization (subtract mean and divide std ) using ImageNet (Deng et al., 2009) statistics, i.e. mean = (0 . 485 , 0 .
456 , 0 . 406) and std = (0 . 229 , 0 .
During testing, we only resize the image to 224 × 224 , followed by ImageNet normalization.
Following previous work (Chen et al., 2019; Tian et al., 2020), we further l 2 -normalize features.
C. C ONTRASTIVE - LEARNING P RE - TRAINING
The detailed description of MoCo-v3 is in Appendix B.
[Table]
Architecture.
The architecture of MoCo-v3 is ResNet18-Projector-Predictor .
We use the codebase of OpenSelfSup 7 .
For projector, we use 3-layer NonLinearNeckSimCLR (Chen et al., 2020a), with the dimension transitions of 512 → 1024 → 1024 → 256 .
For predictor, we use 2-layer NonLinearNeckSimCLR , with the dimension transitions of 256 → 1024 → 256 .
The base momentum m to update key encoder is 0.996 and is linearly increased to 1 as training iteration goes.
The temperature τ for contrastive loss is set to 1.
Optimization.
We follow Chen et al.
(2021b) to use LARS optimizer with initial learning rate of 0.3, weight decay of 1 . 5 e − 6 , momentum = 0 . 9 , and use CosineAnnearling learning schedule.
We train all models with batch size of 256 for 200 epochs.
Data augmentation.
Following previous work (Grill et al., 2020; Chen et al., 2021b), we use strong data augmentation:
RandomResizedCrop , RandomHorizontalFlip , and
ColorJitter with (brightness=0.4, contrast=0.4, saturation=0.4, hue=0.4) and probability of 0.8, RandomGrayscale with probability of 0.2, GaussianBlur with probability of 0.5, and
Solarization with probability of 0.2.
C. L INEAR EVALUATION FOR PRE - TRAINING
In § 4.1, we manually construct 9 sub-tasks in near-domain task, thus resulting 9 pre-trained models for each pre-training method (FSP and CLP).
Besides, for mixture-domain and out-domain tasks, we use models pre-trained on the entire NCT training dataset with no class excluded.
Here we report their performance w.r.t.
linear classiﬁer.
Speciﬁcally, we use the pre-trained models to extract features from NCT training set and test set.
Then, we train LogisticRegression models on whole training features, and report the accuracy for whole test features.
We report the results of Base-Base and Joint-Joint.
Base-Base means the novel class is excluded when we train and test the linear classiﬁers, while Joint-Joint means the linear classiﬁers are trained on all training features (80k samples) and then evaluated on all test features (20k samples).
Table C.1 shows the results.
With full supervision, FSP models can achieve better results in base label space (Base-Base).
However, when evaluated in joint label space (Joint-Joint), they are worse than CLP models.
Besides, in low-shot setting, CLP models underperform FSP models in base label space ( c.f. the “Base” column in Table 1 near-domain task), but when the number of training samples
[Table] [Table]
increases (no longer few-shot setting), CLP models can achieve similar results as FSP models (Table
C.1 “Base-Base” column).
D D ETAILED R ESULTS D. 10- SHOT RESULTS
Table D.1 shows the results of 10-shot settings in near-domain and mixture-domain task, in complementary to results in Table 1.
D. A BLATION
For ablation study, we exclude cancer-related classes, i.e. cancer-associated stroma (STR) and colorectal adenocarcinoma epithelium (TUM), from NCT training set, and use them as novel classes.
An additional CLP model is trained on this base dataset for ablation.
Table D.2 shows the numerical results of Figure 3-(a), which studies the effect of the number of prototypes in the legacy dictionary.
Figure D.1 shows the full results of comparison between data augmentation (DA), latent augmentation (LA), and their combination (LA × DA).
It can be seen that, DA brings slight gain when the number of augmentation times is small, and brings negative gain when the number of augmentation times increases.
In contrast, LA and LA × DA can consistently improve the baselines for base class and novel class, resulting in best performance in harmonic mean.
The data augmentations used in this experiment are RandomResizedCrop(scale=(0.8, 1.0)) ,
RandomHorizontalFlip , and
ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.2) with probability 0.8.
[Figure]
D. V ISUALIZATION D.3. V ISUALIZATION PROCEDURE
Models.
All images are visualized with models that have never seen their classes during pretraining.
For example, if the class “NORM” in NCT is regarded as novel class, samples belonging to it are excluded from this sub- pre-training dataset.
Models trained on this sub-dataset are used to visualize “NORM” class.
Similarity map.
To better understand why the generalization gap exists between CLP and FSP models, we visualize how CLP models and FSP models are attended to different local features for non-iconic, multi-object and multi-texture pathology images.
To this end, we resize images in the NCT test set to 448 × 448 , and use the pre-trained CLP models and FSP models to extract l 2 -normalized features from stage-2, stage-3, and stage-4 of ResNet-18.
To obtain similarity heat maps, we compute the cosine similarity between global representations (after pooling) and local representations (before pooling).
For absolute similarity (“Abs.
Sim.”), we directly rescale the similarity values by multiplying 255.
For normalized relative similarity (“Rel. Sim.”), we ﬁrst rescale the similarity values to [0 , 1] by subtracting the minimum and dividing the maximum, and then multiply them by 255.
Local feature agglomeration.
We follow Chen & Li (2020) to see how local features are agglomerated across layers.
To this end, we run K-means in scikit-learn Pedregosa et al.
(2011) on the l 2 normalized local features (before pooling) from stage-2, stage-3, and stage-4 of ResNet-18 with different numbers of clusters, i.e. 2, 4, 6.
D.3. M ORE EXAMPLES
Figure D.2 and Figure D.3 show more example images in the NCT dataset.
Figure D.4 and Figure
D.5 show the results of the LC-25K dataset and the PAIP dataset respectively.
We visualize the absolute and relative cosine similarity between the global average pooled feature and the local features before pooling (“Abs.
Sim.”
and “Rel.
Sim.” columns accordingly).
To inspect correlation between local features, we conduct k-means clustering with different k values (columns with “k=*”).
“Low”, “middle” and “high” represent using features from stage-2, 3, and 4 from ResNet-18.
[Figure]
[Figure]
[Figure]
[Figure]