T OWARDS BETTER UNDERSTANDING AND BETTER GENERALIZATION OF FEW - SHOT CLASSIFICATION IN HISTOLOGY IMAGES WITH CONTRASTIVE LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot learning is an established topic in natural images for years, but few work is attended to histology images, which is of high clinical value since well-labeled datasets and rare abnormal samples are expensive to collect.
Here, we facilitate the study of few-shot learning in histology images by setting up three cross-domain tasks that simulate real clinics problems.
To enable label-efﬁcient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build a few-shot system.
CL learns useful representations without manual labels, while LA transfers semantic variations of base dataset in an unsupervised way.
These two components fully exploit unlabeled training data and can scale gracefully to other label-hungry problems.
In experiments, we ﬁnd: i) models learned by CL generalize better than supervised learning for histology images in unseen classes, and ii) LA brings consistent gains over baselines.
Prior studies of self-supervised learning mainly focus on ImageNet-like images, which only present a dominant object in their centers.
Recent attention has been paid to images with multi-objects and multi-textures (Chen & Li, 2020).
Histology images are a natural choice for such study.
We show the superiority of CL over supervised learning in terms of generalization for such data, and provide our empirical understanding for this observation.
The ﬁndings in this work could contribute to understanding how model generalizes in the context of both representation learning and histological image analysis. Code will be available.
I NTRODUCTION
Histological images provide crucial phenotypical and diagnostic information for disease assessment and prognosis (Srinidhi et al., 2020).
Thus, computer-aided histological image classiﬁcation systems are highly demanded but expensive to build due to the scarcity of well-annotated data.
Besides, histological images diversify in many aspects, including acquisition protocols, body sites and tissue types.
Such heavy domain shifts and variations pose more challenges and aggravate the data-hungry issues.
How to train robust models with limited access to annotated samples becomes the key of general diagnosis systems.
The aforementioned issues have motivated us to sort for few-shot learning (FSL), which aims to learn new concepts from limited labeled data.
Recent works have demonstrated the success of few-shot systems in natural images, but they are much unexplored in medical images, especially in histological ones.
Here, we facilitate the study of FSL and generalized FSL (GFSL) for histology images by setting up 3 cross-domain tasks where there exist near-, middle- and out-domain shifts from base class to novel class.
In addition, we also investigate the impact of homogeneous and heterogeneous shot selection, i.e. few-shot samples come from the same whole slide image (WSI) or different ones (more on this later), to a few-shot system.
To enable label-efﬁcient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build a few-shot system.
Concretely, CL learns a meaningful encoder during the pre-training stage without relying on label information, while LA inherits knowledge from “unlabeled” base dataset by transferring semantic variations in latent space.
These two components fully exploit a base dataset by using its legacy: the learned model weights, and the captured latent variations.
[Figure]
Also to our surprise, we show the generalization gap between state-of-the-art CL models and the supervised models is larger in histological images than that in natural images.
Previously studies of CL mainly focus on iconic natural images where only a dominant object occupies image centers, while images like histological ones, where multiple small objects (e.g. cells, nucleus) and various textures (e.g. muscle, mucus) are densely presented, are much unexplored.
We also aim to ﬁll the gap of studying CL for non-iconic, multi-object and multi-texture histological images, and empirically explain why the large generalization gap between CL models and supervised ones exists for them.
To summarize, our key ﬁndings and contributions are:
• We study the few-shot classiﬁcation problem in histological images.
Extensive investigations with respect to domain-speciﬁc problems are conducted.
• We propose a simple label-efﬁcient method for few-shot learning, which incorporates contrastive learning and latent augmentation to fully exploit training data in an unsupervised way.
Extensive experiments conﬁrm their consistent gains and improved generalizability.
• We show that, at slight odds with ﬁndings in iconic natural images, CL learned models generalize better than the supervised counterparts for histology images by a large margin.
We analyze and provide our empirical explanations for this observation, which we believe could contribute to understanding how model generalizes to novel samples in the context of representation learning and histology image analysis.
P RELIMINARIES AND PROBLEM FORMULATION
Whole-Slide Image (WSI).
Whole-slide images are digital scans of histology tissue slides collected by biopsy or surgery.
Given micron-size pixels and centimeter-size slides, a WSI is usually of gigapixel size and thus have to be divided into hundreds or thousands of small “patches” for computational analysis.
A patch represents the basic unit in patch-level classiﬁcation problem.
As tissues context and staining quality could vary across WSIs, the extracted patches’ styles are conﬁned to their source WSIs, leading to inter-WSI domain shift.
Besides, unlike iconic natural images which only present a dominant object in their centers, histological patches contain multiple small objects and multiple texture-like tissues (see Figure 1), making the process different from the major recognition systems which only need to focus on dominant objects.
Few-shot Learning (FSL).
Few-shot classiﬁcation is to learn from a large “base” dataset, and afterwards generalize to unseen classes with only limited labeled data.
Formally, a base dataset is deﬁned as D base = { ( x i , y i ) } N base i = ⊂
X base × Y base , where X base is a sample set and Y base is their label space.
Novel dataset D novel =
{ ( x i , y i ) } N novel i = has a disjoint label space, i.e. Y base ∩Y novel =
Ø where Y novel is the novel label space.
A few-shot learner is trained on D base and evaluated on a series of meta-tasks sampled from D novel .
Such task is deﬁned as T =
{ ( S i , Q i ) } Ii =1 , where
S = { ( x i , y i ) } NKi =1 ∼ D novel is a small training set, called support set, and Q = { x i } NQi =1 ∼ X novel is another small test set, called query set, and I is the number of tasks.
This formulation is termed as N -way K -shot ( Q -query) task, since N classes are sampled from Y novel , each with K labeled samples for training and Q unlabeled samples for testing.
Usually, K is less than Q , e.g. K = 1 , 5 , and Q = 15 .
The evaluation stage is often referred to as meta-testing stage.
Generalized few-shot learning (GFSL).
Unlike in FSL, GFSL samples meta-tasks from a joint dataset D joint = D base ∪ D novel , with a joint label space
Y joint =
Y base ∪ Y novel .
Now, support sets and query sets contain both seen base classes and unseen novel classes. DictionaryConstruction a Pre-training ˜ z
f ϕ z LA f θ Low-shot Classification b Learn with LA Unlabeled WSI patches z
Contrastive Learning f ϕ or other proxy tasks f * f ϕ f θ Prediction c Inference d Latent Augmentation (LA) Base Dictionary ℬ z
K-Means N-way K-shot patches WSI patches to predict [Figure]
M ETHODS
Consider a few-shot classiﬁer as f = f θ ◦ f φ , where f φ is a embedding function, i.e. a feature extractor that maps the high-dimensional input image x ∈ R 3 HW into a low-dimensional latent space R d , and f θ is a classiﬁer trained on the support set S and predicts the results for query set Q .
φ and θ are their corresponding parameters.
Our method consists of two phases — a) pre-training f φ on base datasets and b) training f θ on support sets with latent augmentation during meta-testing stage.
Figure 2 shows the overview of our methods.
We elaborate them in the followings.
3. P RE - TRAINING
Current paradigms in FSL for training f φ lie in two folds: i) meta-training, also known as episodic training, where base datasets are divided into various episodic N -way K -shot meta-tasks that simulate meta-learning; and ii) standard training, which does standard fully supervised classiﬁcation pre-training without splitting data.
The former one emphasizes the idea of meta-learning for fast adaption (Schmidhuber, 1987;
Finn et al., 2017), while the latter one attributes the success of FSL to feature reuse (Raghu et al., 2019) or good representations (Chen et al., 2019; Tian et al., 2020).
We follow the latter one and we believe better learned encoders lead to stronger generalizability.
Fully-supervised pre
-training (FSP).
Given a base dataset, we jointly train a feature extractor f φ and a proxy classiﬁer f ψ using the standard cross-entropy loss.
Once pre-trained, only f φ is kept and ﬁxed for downstream tasks.
We term the embedding functions learned by FSP as f FSPφ .
Contrastive-learning pre
-training (CLP).
Self-supervised learning methods alleviate the need for data annotation.
Here we focus on a contrastive learning method – MoCo-v3 (Chen et al., 2021b), which currently holds the state-of-the-art performance.
It consists of three components: a feature extractor (backbone) f φ , a projection head f g and a prediction head f q .
Given an unlabeled base training dataset D ubase = { x i } N base i =1 , the model learns to minimize the contrastive loss function w.r.t.
unlabeled batch data:
[Equation]
where L CLP is a contrastive loss function; x , x (cid:48) are two views of the same images obtained by applying random data augmentation t ; ˜ φ and ˜ g denote the momentum updated copies of φ and g .
In short, contrastive learning aims to maximize the similarity between positive pairs (two augmented views of a same image), while minimizing the similarity between negative pairs (two different images).
We leave more detailed descriptions of MoCo-v3 and L CLP to Appendix B. Once CLP is done, two auxiliary heads, f g and f q , are removed, while f φ is kept and ﬁxed , termed as f CLPφ .
3. L ATENT A UGMENTATION
The pre-trained feature extractor f φ only transfers parts of available knowledge in base datasets by reusing the learned weights.
The more transferable knowledge is inherent in data representations .
It is reasonable to assume that base classes and novel classes share similar modes of variations (Wang et al., 2018) since they are all histology-related.
Such inductive biases allow us to transfer variations from seen tissues or styles to unseen ones.
Here we propose to transfer the representation variations in a simple unsupervised way.
Below, we ﬁrst introduce latent augmentation and then discuss our motivations and intuitions about it.
Base dictionary and Latent augmentation (LA).
Our goal is to fully exploit training data.
This is done not only by reusing pre-trained model weights f φ , but also by transferring possible semantic shifts of clustered representations.
Given an unlabeled base dataset, we perform K-Means on the representations z
= f φ ( x ) to obtain C clusters (Figure 2-(a), red arrows).
The base dictionary is constructed as B = { ( c i , Σ i ) } Ci =1 , where c i is the i -th cluster prototype ( i.e. mean representation) and Σ i denotes its intra-cluster covariance matrix.
Roughly, B captures how the pre-trained f
φ
thinks samples from base dataset will vary in latent space conditioned on cluster i , e.g. using a multivariate Gaussian N ( c i , Σ i ) .
Given the base dictionary B , during meta-testing stage, LA uses original representations z to query the most likely variations from B , followed by additive augmentation ˜
z
=
z + δ (Figure 2-(b,d)).
This is done by sampling δ ∼ N ( 0 , Σ i ∗ ) where i ∗ is selected by ﬁnding the maximum cosine similarity between z and c i .
The classiﬁer f θ is then trained on both the original representations z and the augmented representations
˜ z
(Figure 2-(c)).
3.2. I NTUITIONS AND MOTIVATIONS ON LATENT AUGMENTATION
Why transferring variations works.
LA aims at transferring the knowledge of variations.
Such knowledge brings semantic diversity from base classes to novel classes.
For example, tumorous cells are mutated from normal cells; when given limited tumorous samples, LA may replicate how normal cells alter under this mutation using the captured variations in base dictionaries.
This closely resembles how a pathologist builds his/her knowledge on unseen phenotypes from seen phenotypes.
From the view of under-representative learning (Yin et al., 2019), replicating latent variations encourages under-represented distributions to be closer to regular ones.
From the view of low-data learning, distribution of few samples is not well calibrated (Yang et al., 2021), so using established distributions in base class to calibrate untrustworthy novel class may help.
Besides, LA can be seen as a consistency regularization technique.
Enforcing the classiﬁers’ predictions to be consistent across different perturbation is known to help in low-data regime (Bachman et al., 2014; Berthelot et al., 2019; Sohn et al., 2020).
In fact, LA is a stronger alternative against data augmentation (DA), as we later show in §4.3 that LA outperforms DA by a large margin and can cover the role played by DA.
Why linear additive augmentation is meaningful.
When well trained, deep networks are hypothesized to be good at linearizing deep features (Bengio et al., 2013; Upchurch et al., 2017). This gives the rationality behind linearly inter/extrapolating features, i.e. using “add” operation to generate new features.
Recently, Cheung & Yeung (2020) study the universal label-preserving additive augmentations in latent space that can be used in different data modalities, showing the effectiveness of simple linearly transforming latents.
Why base dictionary construction is warranted in both FSP and CLP.
FSP uses classiﬁcation task as a proxy task to learn useful encoders f φ .
During optimization, features are incentivized to maximize their dot-product similarity with class weights in f ψ , thus forming meaningful metric space.
In the regard of CLP, the contrastive loss, a form of metric-based loss, brings similar features closer while spreading dissimilar representations farther, which also results in an informative metric space.
Thus, feature distance in the representation space of both FSP and CLP is meaningful, justifying the use of unsupervised clustering method to construct base dictionary.
E XPERIMENTS 4. S ETUP
Datasets.
Since tissues vary across body sites, we use three public histology datasets from different body sites to construct three tasks with different degrees of domain shift.
They are: NCT-CRCHE-100K collected from colon site (Kather et al., 2018), LC25000 collected from lung and colon sites (Borkowski et al., 2019), and PAIP19 collected from liver site (Kim et al., 2021); we term them as NCT, LC-25K and PAIP respectively.
NCT consists of 9 classes with 100k non-overlapping patches in total, each of size 224 × 224 .
LC-25K has 5 classes with 5,000 patches in each class; each patch is of size 768 × 768 .
PAIP is composed of 50 WSIs, each of size about 45k × 45k with 3 mask annotated classes.
For LC-25K, all patches are resized to 224 × 224 .
For PAIP, the foreground tissues are cropped into 75k patches of size 224 × 224 with the same pixel resolution as NCT and labels are assigned by majority voting.
More details are in Appendix A. When novel and base classes are from different organs, we view the novel classes as out-domain classes. When they are from the same organ, we consider them as near-domain classes only when the data is collected from the same source. Otherwise, they are considered as middle-domain classes due to difference in imaging protocols.
In our experiment, we deﬁne three tasks to study different domain generalization scenarios:
i) Near-domain task (to study GFSL).
We randomly split NCT by 80%/20% to construct a training set (80k images) and a test set (20k images).
Then, we do leave-one-class-out-as-novelclass to the training set to construct 9 base datasets and use the test set as D joint for evaluation.
This results in 9 sub-tasks, each of which has one class regarded as novel class and samples belonging to it are excluded from pre-training datasets.
ii) Mixture-domain task (to study FSL).
We use the entire training set (80k images) of NCT as D base , and use LC-25K as D novel for evaluation.
Two of ﬁve classes in LC-25K are colon-related (middle-domain novel classes) and the rest three are lung-related (out-domain novel classes).
iii) Out-domain task (to study FSL).
Similar to mixture-domain task, we use the NCT training set as D base and PAIP as the novel dataset D novel . Liver tissues from PAIP are different from colon tissues in NCT. We thus regard them as are out-domain novel classes.
To study heterogeneous and homogeneous shot selection problem, we use WSI ID information to split PAIP into a support WSI set (15 WSIs with 22.5k images) and a query WSI set (35 WSIs with 52.5k images).
During evaluation, support samples and query samples are drawn from the support WSI set and the query WSI set respectively.
Heterogeneous strategy selects few-shot samples from different support WSIs, while homogeneous strategy selects them from a single randomly chosen support WSI.
Evaluation.
If not otherwise speciﬁed, for near-domain task, we evaluate methods in 1000 ×(9 sub-tasks) random meta-tasks; for mixture- and out-domain task, we evaluate methods inrandomly sampled meta-tasks.
All meta-tasks use 15 samples per class as the query set.
We report the average F1-score and 95% conﬁdence interval.
To handle the unequal numbers of base classes and novel classes, we follow convention in GZSL (Xian et al., 2018) and GFSL (Shi et al., 2020) to report their average harmonic mean.
More details about evaluation metrics are in Appendix A.
Implementations.
I. Pre-training.
We use ResNet-18 as the embedding function f φ , and follow previous arts in FSL (Tian et al., 2020; Chen et al., 2019) to use l 2 -normalized features for clustering and downstream meta-tasks.
II.
Latent Augmentation.
We use faiss (Johnson et al., 2019), a library for clustering, to perform K-means with a ﬁx seed for reproducibility.
The number of prototypes in the base dictionary is 16 ( C = 16 , discussed in ablation §4.3).
In each meta-task, each sample is augmented 100 times (including the original one) by LA.
We defer more implementation details of pre-training and LA to Appendix C.
Compared methods.
Recent works (Chen et al., 2019; Tian et al., 2020), including a concurrent work for histology image (Shakeri et al., 2021), show that sophisticated episodic training (metatraining) is no better than standard pre-training.
Hence, we summarize methods using standard pre-training as: 1) NearestCentroid.
It computes class centroids from support sets and classiﬁes query samples to their nearest centroids.
Related works using such strategy includes Wang et al.
(2019a), Snell et al.
(2017), and Chen et al.
(2020c)
, to list a few; 2) LinearClassiﬁer.
It trains a new fully-connected layer with different loss functions (Chen et al., 2019; Lee et al., 2019) w.r.t.
support samples, or directly uses linear models in scikit-learn (Pedregosa et al., 2011), e.g. LogisticRegression (Yang et al., 2021; Tian et al., 2020).
For the ease of implementation and consistency, we use
NearestCentroid, and two l 2 -regularized linear classiﬁers — LogisticRegression and RidgeClassiﬁer, all from scikit-learn APIs (Pedregosa et al., 2011).
4. M AIN R ESULTS
Main results in near-domain task and mixture-domain task are presented in Table 1.
We discuss the overall ﬁndings here and leave the discussion of design choices to ablation study in §4.3, and the choice of linear classiﬁers ( i.e. LogisticRegression/RidgeClassiﬁer) to §4.4.
Fully-supervised f FSPφ v.s. Self-supervised f CLPφ .
Results in Table 1 show that CLP generalizes better to novel classes than FSP by a large margin.
Comparing the best vanilla entries (w/o.
LA) using two types of pre-training methods, we observe an average advantage in HarmMean of 4%, 5% and 8% in 1-/5-/10-shot settings by CLP in near-domain task and 10%, 19%, 16% in mixture-domain [Table] task.
Besides
[Table] CLP representations beneﬁt more from the increase of number of shots than FSP’s in both tasks, e [Table]
g. +17% vs. +11% and +12% vs. +10% when 1-shot → 5-shot for linear classiﬁers in near-domain, and mixture-domain tasks respectively.
Despite the inevitable advantage of FSP in base classes under full supervision, CLP demonstrates stronger generalizability to novel classes.
Furthermore, Table 3 also conﬁrms the superiority of CLP over FSP in out-domain task where a larger domain shift exists.
Such generalization gap between FSP and CLP in histology images is at slight odds with observations in natural images, where they show similar generalizability.
We study and discuss it at §4.4.
We also provide the linear evaluation results of all the 20 pre-trained models in Appendix C.3 for a reference to see how each model perform in NCT dataset.
Latent augmentation brings consistent improvement.
Regardless of pre-training methods, LA brings consistent gains over baseline linear classiﬁers, conﬁrming its effectiveness.
With base dictionaries, limited few-shot samples are able to proliferate in a reasonable way by transferring latent variations.
Such boost maintains its signiﬁcance from near-domain task to mixture-domain task (Table 1) but turns smaller in out-domain task (Table 3).
This is in our expectation since the three classes (non-tumor, viable-tumor and other) deﬁned in PAIP are extremely coarse-grained: it may include couples of real ﬁne-grained classes ( c.f. Figure A.4 in Appendix).
Few samples are unable to well represent their entangled semantics.
Hence, this observation does not repudiate the effectiveness of latent augmentation but re-ensures its tenability. 4. A BLATIONS To ablate design choices, we exclude two cancer-related classes, i.e. cancer-associated stroma (STR) and colorectal adenocarcinoma epithelium (TUM), from NCT to be novel classes, and use the rest as base classes. If not otherwise speciﬁed, all ablations are conducted on CLP models with RidgeClassiﬁer for 300 meta-tasks in 5-shot setting.
Number of prototypes in base dictionary.
Figure 3 (a) shows how performance varies with the number of prototypes C .
We observe the similar tendency between base class and novel class, where their harmonic means peak at C = 16 ; we subsequently choose C = 16 for all experiments.
Besides, the performance of base classes and novel classes shows opposite trends from C = 4 to C = 16 .
A v e r a g e F 1 - s c
o r e s ( % ) {+
+}
Number {+of prototypes  8 16 32 64 
+}
HM-LA
Base-LA
Novel-LA HM
Base
Novel
A v e r a g e H a r m o n i c m e a n o f
F 1 - s c o r e ( % )
Number of augmentation times
LA
LA × DA
DA {+2 2 4 2 6 2 8 2 10 2 12 2 14 
+}
[Figure]
See text for more details.
[Table]
[Figure]
[Table]
The trade-off exists here that as the granularity of clusters increases ( C ↑ ), the intra-cluster variance decreases, which results in better grouping accuracy but brings less semantic variation.
The novel classes beneﬁt from larger variation while the base classes beneﬁt from more accurately estimated variation since they have been exposed in training.
Nevertheless, LA demonstrates its robustness by consistent improvement over baselines ( in Fig.
DA vs. LA, and number of augmentation times.
Here we compare LA with data augmentation (DA), and their combination.
Details of DA used here can be found in Appendix D.2.
Results are shown in Figure 3-(b).
It can be seen that LA outperforms DA by a large margin.
The boost brought by DA saturates easily and keeps dropping thereafter, while LA keeps improving with all tested cases.
Besides, DA can marginally improve LA (LA × DA v.s. LA).
We conclude that LA has already covered the role played by DA in an implicit way since the most of gains are brought by LA.
It is worthy to emphasize the computation budget involved in LA (addition in R d space) is signiﬁcantly lower than DA (image augmentation in R 3 HW space and encoder forwards).
Therefore, we run all experiments only with unsupervised LA, as one of our core contributions go with it, and its combination with DA is beyond the scope of this work. Covariance type.
Here we explore more types of covariances that LA can use.
Speciﬁcally, we also include: 1) “Tied”, where all clusters share a covariance matrix estimated from the entire base dataset, 2) “Diag”, where each cluster has its own diagonal covariance matrix, i.e. diagonal elements are a variance vector and non-diagonal elements are zeros, 3) “Spherical”, where each cluster has its own single scalar-variance shared by all feature dimensions.
Table 2 shows the results.
LA with all types of covariances improves performance.
This emphasizes the importance of diversifying few samples with variation. Using full covariance estimation achieves the best performance. We further show how different covariance types perform with smaller or larger cluster sizes in Appendix D.3. Heterogeneous v.s. Homogeneous patch selection.
In real clinics, WSIs’ appearance varies across patients.
Patches from a single WSI usually have less variations than those from different ones. We therefore investigate the hetero-/homo-geneous patch selection strategies as deﬁned in out-domain task (§4.1). Table 3 shows the results. We observe: i) heterogeneous selection is favored over homogeneous one, as it shows higher baselines; and ii) LA brings more gains for heterogeneous selection.
Heterogeneous patches provide reliable and diverse “anchor” samples than homogeneous ones, thus can beneﬁt more from bootstrapping the base dictionary.
Additional ablation studies. We conduct two more ablation studies: (a) clustering with different random seeds in K-Means, (b) studying the effect of l 2 -normalization to covariance. We show that LA’s improvement is stable under different random seeds used in K-Means (Appendix D.4), and covariances estimated before and after l 2 -normalization are highly correlated, and further l 2normalizing augmented samples marginally degenerates LA’s performance (Appendix D.5). f CLPϕ f FSPϕ k= k= k= k= k= k=
Novel Class
Low
Middle
High
Abs.
Sim.
Rel.
Sim.
Abs.
Sim.
Rel.
Sim.
Similarity:
Clusters: 2 3 4 {+
+}
Base
[Figure]
4. M ORE DISCUSSION
Disparity between f CLPφ and f FSPφ inﬂuences the choice of base learner.
In Table 1, we ﬁnd i) stronger baselines for CLP and FSP vary, and ii) simple NearestCentroid can sometimes outperform the vanilla l 2 -regularized 1 linear classiﬁers for FSP.
Here we brieﬂy discuss our understandings.
Representations produced by CLP can have different distributions compared to FSP, as also noticed by He et al.
With limited training samples, different classiﬁers can have their own biases in building decision boundary, leading to different generalizability.
Besides, no regularization techniques are used during FSP (Chen et al., 2020a), e.g. weight decay (Krogh & Hertz, 1992), DropBlock (Lee et al., 2019; Tian et al., 2020) or “distill” regularization (Tian et al., 2020).
The linear classiﬁers, though with l 2 penalty, may still be overﬁtted in such representation space when only limited samples are provided.
Subsequently, the simplest NearestCentroid model can yield better results than these overﬁtted linear models, as it has the least complexity.
Why do CLP models generalize better than FSP ones in histology images?
To study why the large generalization gap exists, in Figure 4, we follow Chen & Li (2020) to see how features aggregate in space.
Speciﬁcally, we visualize the cosine similarity between a feature map (a set of local representations) and its global averaging (global representation), and run K-Means on the feature maps from different layers ( i.e. stage 2, 3, 4 of ResNet) with different cluster numbers.
We observe: the FSP model maintains high global-local similarity in low-/middle-levels, while the
CLP model holds it in high-level (solid boxes). Besides, the CLP model extracts low-level features that are edge-related, and afterwards successively agglomerates adjacent similar structures (dashed boxes).
In contrast, the FSP model can differentiate nuclei in low-&middle-levels but fails to encode structure-related features in a deeper layer.
We further visualize some samples from base classes (bottom of Figure 4 and D.6), and ﬁnd that such disparity between FSP and CLP does not only exist in a previously unseen class but also in seen classes. In the bottom row of Figure 4, FSP only pays attention to the most discriminative parts, leaving the rest “redundant parts” disorder (dashed box).
However, the discriminative parts are likely to alter when a new class is presented.
FSP’s inability to fully encode meaningful information may lead to its failure in generalizing to new classes. Meanwhile, CLP encodes most of tissuestructure-related features that may be useful for novel class recognition, possibly resulting in better generalizability. However, FSP models and CLP models are shown to perform similarly, instead of differently, under the same visualization procedure in ImgNet dataset 2 . What might cause this disparity?
ImgNet has more diverse classes (1000 classes) and samples ( ∼ 1.28M images) than those in histology datasets. FSP models in ImgNet need to recognize the discriminative parts of allclasses.
In such case, the redundant information in one class might contribute to the recognition of another class. Therefore FSP may eventually encode most of available information for new classes that are related to ImgeNet classes. However, histology datasets usually lack enough diverse annotated classes that help to build a know-everything FSP model. A topic we leave for future work is to explore whether CLP always generalizes better than FSP when pre-training on a base dataset with limited number of annotated classes and if the generalization gap would increase as the label diversity decreases.
We point out that the visualization results and the large generalization gap shown in our work still remains as empirical observations. Our discussion is about what the reasons behind them can be. We hope our work would be helpful for representation learning, histology image analysis, and beyond.
R ELATED W ORK
Few-shot learning (FSL).
FSL has been tackled from different perspectives, e.g. metric-based and optimization-based (Finn et al., 2017; Rusu et al., 2018; Munkhdalai & Yu, 2017).
Being one branch of metric-based methods, this paper follows a “pre-training & ﬁne-tuning” scheme, where previous works typically learn a shared metric space by standard fully-supervised pre-training (Tian et al., 2020; Chen et al., 2019; Wang et al., 2018).
In contrast, we propose to incorporate selfsupervised pre-training to enable label-efﬁcient learning, and show that it can yield stronger generalization than supervised pre-training.
FSL in medical images.
FSL in medical images is at its early stage, especially for histology images.
Mahajan et al.
(2020) investigate FSL methods in skin-disease classiﬁcation, while Chen et al.
(2021a) tackle COVID-
CT image classiﬁcation using contrastive pre-training and prototypical network ﬁne-tuning.
In the regard of histology image, Medela et al.
(2019) use a triplet loss (Schroff et al., 2015) to pre-train an encoder with a followed ﬁne-tuned SVM classiﬁer for low-shot domain adaptation. Sikaroudi et al.
and Teh & Taylor (2020) study learning with less data in histology images.
Concurrent to our work, Shakeri et al.
) simultaneously propose a benchmark for few-shot classiﬁcation of histological images.
Our work has similar but different settings, with more investigations conducted, e.g. GFSL task, and hetero-/homo-geneous few shots selection.
Self-supervised learning.
Self-supervised learning aims to learn good representations without true labels.
Recent state-of-the-art variants can be categorized as contrastive-based learning (Chen et al., 2021b; 2020a; He et al.,
, cluster-based learning (Caron et al., 2018; and expectation prediction based learning (Grill et al., Chen & He,
However, this line of works focus on pre-training on ImageNet-like images, and recent attention has been attracted to images with multi-objects and multi-texture presented (Chen & Li, 2020).
We see histology image as a natural choice for such study, and show that contrastive learning can agglomerate structural “partwhole” information and maintain “global-local consistency”, which make it generalize better for such data than supervised counterparts (see §4.4 and Table C.1 in Appendix C.3).
Representation variation augmentation.
The idea of exploiting representation variations dates back to a decade ago (Heller et al., 2009; Salakhutdinov et al., 2012) or more earlier.
Recent variants further develop this idea.
For example, Hariharan & Girshick (2017) use a generator to generate “hallucinated” novel features from variation of given base samples.
This method is later extended by not relying on given base samples (Wang et al., 2018).
Wang et al.
(2019b) use class variance to perform semantic augmentation for classiﬁcation and segmentation (Wang et al., 2021), while Yin et al.
and Liu et al.
(2020) utilize intra-class variance of head classes to augment tail classes for “long-tail” face recognition problem.
Yang et al.
(2021) use the distribution information, i.e. mean and variance, of base classes to calibrate novels’ distribution.
Cheung & Yeung (2020) propose a novel and systematical framework to apply automated augmentation with more considerations in label-preserving transformations.
This work follows the line of Yang et al.
; Wang et al.
(2021); Liu et al.
(2020), but in contrast to them, we obtain and transfer variations without relying on any label information, allowing our method to scale gracefully to other label-hungry problems.
C ONCLUSION
This work has studied, as an early attempt, the few-shot learning problem for histology images.
We incorporate contrastive learning and latent augmentation to fully exploit training data in an unsupervised way, which means our method can gracefully scale to other large label-hungry problems.
More importantly, we show that the generalization gap between the state-of-the-art contrastive learning pre-training method and supervised pre-training in histological images is larger than that in ImageNet experiments.
We analyze the underlying reasons and provide our empirical understandings.
We hope our work could contribute to the study of representation learning and generalization for both self-supervised learning community as well as histology image analysis community.
A D ATASET A. NCT
Data.
NCT-CRC-HE-100K dataset (Kather et al., 2018) contains 100 , 000 non-overlapping patches extracted from hematoxylin and eosin (H&E) stained human colorectal cancer and normal tissues.
Each image is of size 224 × 224 at 0.5 MPP ( 20 × magniﬁcation).
Tissue classes and their label index are: 0) background (BACK), 1) adipose (ADI), 2) debris (DEB), 3) lymphocytes (LYM), 4) mucus (MUC), 5) smooth muscle (MUS), 6) normal colon mucosa (NORM), 7) cancerassociated stroma (STR), 8) colorectal adenocarcinoma epithelium (TUM).
Figure A.2 shows the class distribution and Figure A.1 shows 4 examples per class from NCT.
These images are from National Center for Tumor Diseases (Heidelberg, Germany) and University Medical Center Mannheim (Mannheim, Germany).
Their acquisition protocols differ across organizations, which lead to intersource domain shift.
Processing.
We use the “NCT-CRC-HE-100K-NONORM” dataset, which does not apply color normalization to images.
We randomly split NCT dataset (100k images) into a training set (80k images) and a test set (20k images).
The class distribution is the same for the training set and test set, i.e. data are sampled w.r.t.
each class.
Since each image in NCT is of size 224 × 224 , we do not further resize it.
Near-domain task.
NCT dataset contains tissue patches from multiple sources, which bears slight domain shift since the staining intensity varies.
To better study the generalizability in FSL problem, we do leave-one-class-out-as-novel-class to the training set of NCT.
This procedure closely resembles the leave-one-out cross-validation.
Since histology datasets usually have far less number of classes compared to natural image dataset (e.g. 9 classes in NCT v.s. 100 classes in mini ImageNet and 608 classes in tiered ImageNet), we believe such leave-one-class-out-as-novel-class can better simulate the generalization error bound when number of classes is small.
After the leave-one-classout-as-novel-class split, 9 sub-base datasets are constructed, each of which has one class regarded as the novel class and samples belonging to it are excluded from pre-training.
For each sub-base dataset, a CLP model and a FSP model will be trained on it.
Therefore, a total of 18 models will be obtained (9 CLP models and 9 FSP models).
All models are evaluated on the NCT test set.
It contains both seen classes and unseen novel classes, so as to simulate generalizes few-shot classiﬁcation.
Evaluation metric.
Typical FSL methods use accuracy as metrics.
However, to study GFSL problem, the performance of novel classes and base classes should be appropriately considered.
Accuracy w.r.t.
speciﬁc classes cannot be computed in a joint label space.
Hence, we choose F1-score as our metric.
Besides, the numbers of novel classes and base classes are unequal in each sub-task, so we follow convention in GFSL to compute their harmonic mean.
The ﬁnal metrics are
[Equation]
where F 1 ( j ) denotes the F1-score of j -th class.
M base ( i ) (Equation A.1) computes the average
F1-score of base classes in i -th sub-task.
Equation A.2 and Equation A.3 compute the F1-score of novel class and the harmonic mean of M base and M novel in i -th sub-task respectively.
Equation A. shows the ﬁnal metrics we report in this work. [Figure] [Figure] c o un t
A. LC
Data.
The LC25000 (LC-25K) dataset contains 25,000 color images from 5 classes.
They are: 0) colon adenocarcinoma (Colon ACA), 1) benign colonic tissue (Colon benign), 2) lung adenocarcinoma (Lung ACA), 3) benign lung tissue (Lung benign), and 4) lung squamous cell carcinoma (Lung SCC).
The class distribution is balanced, i.e. each class has 5000 images. All images are of size 768 × 768 . LC25K is constructed by augmenting 1250 images (250 images for each class). The augmentations are: left and right rotations (within 25 degrees, p=1.0) and vertical and horizontal ﬂips (p=0.5), where “p” represents the probability.
Processing.
We resize all images from 768 × 768 to 224 × 224 .
No other process is taken.
Mixture-domain task.
LC-25K dataset includes tissues from colon and lung sites.
Two of ﬁve classes in LC-25K are colonic tissues, which suffers moderate domain shift due to difference in imaging protocol and pixel resolutions.
Thus, we see them as middle-domain novel classes.
The rest three lung-related classes are regarded as out-domain novel classes since they are from a different organ.
To study few-shot classiﬁcation, we use the entire training set (80k images) of NCT as D base , and use LC-25K as D novel for evaluation.
Evaluation metric.
Similar to near-domain task evaluation in Appendix A.1, we compute
Middle = 12 ( F 1 (0) + F 1 (1)) , Out = 13 ( F 1 (2) + F 1 (3) + F 1 (4)) , HarmMean = 2 1 / Middle+1 / Out .
Here F 1 ( i ) denotes the F1-score of i -th class.
Class 0 and class 1 are colon-related, termed as middle-domain novel classes, and the rest three classes are lung-related, termed as out-domain novel classes. [Figure]
A. PAIP
Data.
The PAIP 2019 (Kim et al., 2021) is composed of 50 H&E stained WSIs at 20 × magniﬁcation.
Each WSI is approximately of size 45 , 000 × 45 , 000 with XML annotation.
It contains three classes, which are: 0) non-tumor liver tissue, 1) viable tumor, and 2) other tissues in whole tumor area but not viable tumor.
The class 2 may include intratumoral hemorrhage, necrosis or non-tumor tissue in whole tumor area.
Figure A.4 shows an example WSI of PAIP19.
This dataset provides “case number” (WSI identiﬁer) information.
Processing.
We use mask information to extract 500 randomly selected non-overlapping patches at 20 × magniﬁcation for each class from each WSI, leading to a dataset with 75 , 000 samples.
We deﬁne patches from 15 randomly selected WSIs as the support WSI set and patches from the rest
WSIs as query WSI set.
[Figure]
, fatty inﬁltration, etc.
B C ONTRASTIVE LEARNING P RE - TRAINING : M O C O - V
MoCo-v3 (Chen et al., 2021b) aims for an empirical study for vision transformer pre-training, but has a ResNet variant. We adopt the ResNet variant.
Overview.
MoCo-v3 follows its ancestors MoCo v1/2 (He et al., 2020; Chen et al., 2020b) with straightforward modiﬁcation.
As shown in Figure B.1, each batch is augmented twice under stochastic augmentation t to obtain two views, denoted as x 1 , x 2 ∼ t ( x ) , where x represents the raw images.
They are then encoded by query encoder and key encoder respectively.
The query encoder is composed of three components: a backbone f φ , a projector (projection head) f g and a predictor (prediction head) f q , formulated as f q ◦ f g ◦ f φ .
The key encoder, also known as momentum encoder, consists of momentum copies of backbone and projector, formulated as f ˜ g ◦ f ˜ φ .
[Figure]
Momentum update.
In iteration k , the momentum update rule is:
[Equation]
where m is the momentum.
Loss function.
The contrastive loss function is a form of InfoNCE (Oord et al., 2018):
[Equation]
, (A.6) Here, v + denotes the positive sample, i.e. the other augmented view of u , and v − denotes the negative samples, i.e. other representations in the batch.
τ is a hyper-parameter, known as temperature (Wu et al., 2018).
The ﬁnal loss is a symmetric sum: L CLP =
[ L ctr ( z 1 , ˜ z 2 ) + L ctr ( z 2 , ˜ z 1 )] / 2 , where the subscript indicates view source, z = l 2 [ f q ◦ f g ◦ f φ ( x )] , ˜ z = l 2 (cid:104) f
˜ g ◦ f ˜ φ ( x ) (cid:105) , and l 2 [ · ] means l 2 normalization.
C I MPLEMENTATION D ETAILS C. F ULLY - SUPERVISED P RE - TRAINING
Optimization.
We use SGD optimizer with lr = 0 . 5 , momentum = 0 .
9 , and no weight decay is used, i.e. weight decay=0.
The batch size is 512.
We train for 100 epochs with “step decay” learning schedule.
The lr is multiplied by 0.1 at 30, 60 and 90 epochs respectively.
Data augmentation.
We use RandomResizedCrop, RandomHorizontalFlip , followed by normalization (subtract mean and divide std ) using ImageNet (Deng et al., 2009) statistics, i.e. mean = (0 . 485 , 0 .
456 , 0 . 406) and std = (0 . 229 , 0 .
During testing, we only resize the image to 224 × 224 , followed by ImageNet normalization.
Following previous work (Chen et al., 2019; Tian et al., 2020), we further l 2 -normalize features.
C. C ONTRASTIVE - LEARNING P RE - TRAINING
The detailed description of MoCo-v3 is in Appendix B.
[Table]
Architecture.
The architecture of MoCo-v3 is ResNet18-Projector-Predictor .
We use the codebase of OpenSelfSup 3 .
For projector, we use 3-layer NonLinearNeckSimCLR (Chen et al., 2020a), with the dimension transitions of 512 → 1024 → 1024 → 256 .
For predictor, we use 2-layer NonLinearNeckSimCLR , with the dimension transitions of 256 → 1024 → 256 .
The base momentum m to update key encoder is 0.996 and is linearly increased to 1 as training iteration goes.
The temperature τ for contrastive loss is set to 1.
Optimization.
We follow Chen et al.
(2021b) to use LARS optimizer with initial learning rate of 0.3, weight decay of 1 . 5 e − 6 , momentum = 0 . 9 , and use CosineAnnearling learning schedule.
We train all models with batch size of 256 for 200 epochs.
Data augmentation.
Following previous work (Grill et al., 2020; Chen et al., 2021b), we use strong data augmentation:
RandomResizedCrop , RandomHorizontalFlip , and
ColorJitter with (brightness=0.4, contrast=0.4, saturation=0.4, hue=0.4) and probability of 0.8, RandomGrayscale with probability of 0.2, GaussianBlur with probability of 0.5, and
Solarization with probability of 0.2.
C. L INEAR EVALUATION FOR PRE - TRAINING
In §4.1, we manually construct 9 sub-tasks in near-domain task, thus resulting 9 pre-trained models for each pre-training method (FSP and CLP).
Besides, for mixture-domain and out-domain tasks, we use models pre-trained on the entire NCT training dataset with no class excluded.
Here we report their performance w.r.t.
linear classiﬁer.
Speciﬁcally, we use the pre-trained models to extract features from NCT training set and test set.
Then, we train LogisticRegression models on whole training features, and report the accuracy for whole test features.
We report the results of Base-Base and Joint-Joint.
Base-Base means the novel class is excluded when we train and test the linear classiﬁers, while Joint-Joint means the linear classiﬁers are trained on all training features (80k samples) and then evaluated on all test features (20k samples).
Table C.1 shows the results.
With full supervision, FSP models can achieve better results in base label space (Base-Base).
However, when evaluated in joint label space (Joint-Joint), they are worse than CLP models.
Besides, in few-shot setting, CLP models underperform FSP models in base label space ( c.f. the “Base” column in Table 1 near-domain task), but when the number of training samples
[Table] [Table]
increases (no longer few-shot setting), CLP models can achieve similar results as FSP models (Table
C.1 “Base-Base” column).
D D ETAILED R ESULTS AND M ORE ABLATIONS D. 10- SHOT RESULTS
Table D.1 shows the results of 10-shot settings in near-domain and mixture-domain task, in complementary to results in Table 1.
D. A BLATION ON D ATA A UGMENTATION
For ablation study, we exclude cancer-related classes, i.e. cancer-associated stroma (STR) and colorectal adenocarcinoma epithelium (TUM), from NCT training set, and use them as novel classes.
An additional CLP model is trained on this base dataset for ablation.
Table D.2 shows the numerical results of Figure 3-(a), which studies the effect of the number of prototypes in the base dictionary.
Figure D.1 shows the full results of comparison between data augmentation (DA), latent augmentation (LA), and their combination (LA × DA).
It can be seen that, DA brings slight gain when the number of augmentation times is small, and brings negative gain when the number of augmentation times increases.
In contrast, LA and LA × DA can consistently improve the baselines for base class and novel class, resulting in best performance in harmonic mean.
The data augmentations used in this experiment are RandomResizedCrop(scale=(0.8, 1.0)) with probability 1.0,
RandomHorizontalFlip with probability 0.5, and
ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.2) with probability 0.8.
These augmentations are applied sequentially and jointly on original images.
[Figure]
D. A BLATION ON C OVARIANCE T YPES
By default, LA estimates the “full” covariance matrix for each cluster.
Following notations in GaussianMixture model API in scikit-learn library (Pedregosa et al., 2011), we further compute different types of covariance: 1) “tied”, where all clusters share the same general covariance matrix estimated from the entire base dataset; 2) “diag”, where each cluster has its own diagonal covariance matrix; 3) “spherical”, where each cluster has its own single scalar variance.
Speciﬁcally, “diag” covariance matrix only computes the variance of each channel (each dimension) without considering the correlation between channels, and “spherical” covariance further averages diagonal elements in each “diag” covariance matrix, resulting in a single scalar variance.
Results are shown in Table 2.
Table 2 reports results of our default setting, where 16 clusters are used.
Here we further explore more options of cluster numbers (leading to larger or smaller clusters).
Figure D.2 show the results.
LA with full covariance matrix does degenerate performance as the cluster size decreases (larger number of prototypes).
In contrast, LAs with isotropic covariance matrices (“spherical”/“diag”) perform more stably when cluster size alters.
However, they underperform “full” covariance in all cluster sizes.
Besides, similar performance degeneration is also observed when using isotropic covariance, but becomes slight.
Note, the “Tied” covariance is estimated from the whole dataset.
Therefore the results should only be affected by randomness in latent augmentation (yellow line in
Figure D.2). [Figure] D. A BLATION ON K-M EANS RANDOM SEEDS
LA relies on K-Means clustering, which might be effected by random initialization.
Here we explore how LA performs under different random seeds.
Still, we use the K-Means API from faiss Johnson et al.
(2019), a library for clustering.
Table D.3 reports the results.
LA is robust to the choice of random seeds.
Among different seeds, the numerical differences of performance are within 95% conﬁdence interval.
We choose the random seed 66 for reproducibility in main paper.
[Table]
D. A BLATION ON C OVARIANCE & L2 NORMALIZATION .
In our experiments
[Table]
we compute the covariance matrices after l 2 -normalization.
To inspect whether the covariance is still meaningful after normalization.
We compute the covariance matrices of NCT test set features before and after l 2 -normalization.
Then, we plot the covariance matrix as heatmaps.
Since the length of a feature vector is 512, and inspecting a 512 × 512 heatmap might be difﬁcult, we ﬁrst compute the whole covariance matrix, then randomly select 128 channels to visualize; the upper triangle elements of this 128 × 128 matrix are masked out.
Figure D.3, D.4, D.5 are results of three runs.
We observe that the covariance before and after l 2 -normalization are highly correlated, where the Pearson correlation coefﬁcients are 0.9908, 0.9900, 0.9910 between before/after l 2 -normalization covariance in Figure D.3, D.4, and D.5 respectively.
Due to this high correlation, the covariance after l 2 -normalization may still be informative to some extend.
Besides, a recent work on self-supervised learning, BarlowTwins (Zbontar et al., 2021), also compute covariance after l 2 -normalization in their ablation study (Section 4, “Loss Function Ablations”).
After l 2 -normalization, all original samples are on the surface of a unit-hypersphere (a d -1 dimensional manifold, i.e. S d − 1 ), but the augmented samples generated by LA are not necessarily on the surface.
We therefore further normalize the augmented samples to see whether this impacts performance.
In the same setting as ablation study (300 meta-tasks, 5-sot), we observe the HMean marginally drops from 75.30 ± 1.01 to 74.69 ± 1.01.
Although the l 2 -normalized features are on the surface of a unit-hypersphere, the linear classiﬁers, typically, are still ﬁtted in the original R d space and no special consideration is taken.
Generating augmented samples that are not on the surface of unit-hypersphere may still help.
For more analysis about the niceness of unit-hypersphere, we refer the reader to the discussion section in Wang & Isola (2020).
D. V ISUALIZATION D.6. V ISUALIZATION PROCEDURE
Models.
All images are visualized with models that have never seen their classes during pretraining.
For example, if the class “NORM” in NCT is regarded as novel class, samples belonging to it are excluded from this sub- pre-training dataset.
Models trained on this sub-dataset are used to visualize “NORM” class.
Similarity map.
To better understand why the generalization gap exists between CLP and FSP models, we visualize how CLP models and FSP models are attended to different local features for non-iconic, multi-object and multi-texture histology images.
To this end, we resize images in the NCT test set to 448 × 448 , and use the pre-trained CLP models and FSP models to extract l 2 -normalized features from stage-2, stage-3, and stage-4 of ResNet-18.
To obtain similarity heat maps, we compute the cosine similarity between global representations (after pooling) and local representations (before pooling).
For absolute similarity (“Abs.
Sim.”), we directly rescale the similarity values by multiplying 255.
For normalized relative similarity (“Rel. Sim.”), we ﬁrst rescale the similarity values to [0 , 1] by subtracting the minimum and dividing the maximum, and then multiply them by 255.
Local feature agglomeration.
We follow Chen & Li (2020) to see how local features are agglomerated across layers.
To this end, we run K-means in scikit-learn Pedregosa et al.
(2011) on the l 2 normalized local features (before pooling) from stage-2, stage-3, and stage-4 of ResNet-18 with different numbers of clusters, i.e. 2, 4, 6.
D.6. M ORE EXAMPLES
Figure D.6 show more visualizations of a base class that has been exposed during both supervised pre-training and self-supervised pre-training.
Figure D.7 and Figure D.8 show more example images in the NCT dataset.
Figure D.9 and Figure
D.10 show the results of the LC-25K dataset and the PAIP dataset respectively.
We visualize the absolute and relative cosine similarity between the global average pooled feature and the local features [Figure] before pooling (“Abs. Sim.”
and “Rel.
Sim.” columns accordingly).
To inspect correlation between local features, we conduct k-means clustering with different k values (columns with “k=*”).
“Low”, “middle” and “high” represent using features from stage-2, 3, and 4 from ResNet-18. [Figure] Low
[Figure]
Low
Middle
High [Figure] [Figure] [Figure]
CLPϕ f FSPϕ k=
[Figure]
k= k= k= k= k=
Low
Middle
High
Abs.
Sim.
Rel.
Sim.
Abs.
Sim.
Rel.
Sim.
Low
Middle
High
Low
Middle
High
Low
Middle
High
Low
Middle
High a b c d e
[Figure]