T HE AUGMENTED IMAGE PRIOR : D ISTILLINGCLASSES BY EXTRAPOLATING FROM A SINGLE IMAGE
Anonymous authors
Paper under double-blind review
A BSTRACT
What can neural networks learn about the visual world when provided with only a single image as input?
While any image obviously cannot contain the multitudes of all existing objects, scenes and lighting conditions – within the space of all 256 3 · 224 · 224 possible 224 -sized square images, it might still provide a strong prior for natural images.
To analyze this “augmented image prior” hypothesis, we develop a simple framework for training neural networks from scratch using a single image and augmentations using knowledge distillation from a supervised pretrained teacher.
With this, we ﬁnd the answer to the above question to be: ‘surprisingly, a lot’.
In quantitative terms, we ﬁnd accuracies of 94% / 74% on
CIFAR-10/100, 69 % on ImageNet, and by extending this method to video and audio, 51% on Kinetics-400 and 84 % on SpeechCommands.
In extensive analyses spanning 13 datasets, we disentangle the effect of augmentations, choice of data and network architectures and also provide qualitative evaluations that include lucid “panda neurons” in networks that have never even seen one.
I NTRODUCTION
Deep learning has both relied and improved signiﬁcantly with the increase in dataset sizes.
In turn, there are many works that show the beneﬁts of dataset scale in terms of data points and modalities used.
Within computer vision, these models trained on ever larger datasets, such as Instagram1B (Mahajan et al., 2018) or JFT-3B (Dosovitskiy et al., 2021), have been shown to successfully distinguish between semantic categories at high accuracies.
In stark contrast to this, there is little research on understanding neural networks trained on very small datasets.
Why would this be of any interest?
While smaller dataset sizes allow for better understanding and control of what the model is being trained with, we are most interested in its ability to provide insights into fundamental aspects of learning: For example, it is an open question as to what exactly is required for arriving at semantic visual representations from random weights, and also of how well neural networks can extrapolate from their training distribution.
While for visual models it has been established that few or no real images are required for arriving at basic features, like edges and color-contrasts (Asano et al., 2020;
Kataoka et al., 2020; Bruna &
Mallat, 2013; Olshausen & Field, 1996), we go far beyond these and instead ask what the minimal data requirements are for neural networks to learn semantic categories , such as those of ImageNet.
This approach is also motivated by studies that investigate the early visual development in infants, which have shown how little visual diversity babies are exposed to in the ﬁrst few months whilst developing generalizeable visual systems (Orhan et al., 2020; Bambach et al., 2018).
In this paper, we study this question in its purest form, by analyzing whether neural networks can learn to extrapolate from a single datum.
However, addressing this question na¨ıvely runs into the difﬁculties of i) current deep learning methods, such as SGD or BatchNorm being tailored to large datasets and not working with a single datum and ii) extrapolating to semantic categories requiring information about the space of natural images beyond the single datum.
In this paper, we address these issues by developing a simple framework that recombines augmentations and knowledge distillation .
First, augmentations can be used to generate a large number of variations from a single image.
This can effectively address issue i) and allow for evaluating the research question on standard architectures and datasets.
This use of data augmentations to generate variety is drastically different to their usual use-case in which transformations are generated to implicitly encode desirable invariances during training.
Second, to tackle the difﬁculty of providing information about semantic categories in the single datum setting, we opt to use the outputs of a supervisedly trained model in a knowledge distillation (KD) fashion.
While KD (Hinton et al., 2015) is originally proposed for improving small models’ performance by leveraging what larger models have learned, we re-purpose this as a simple way to provide a supervisory signal about semantic classes into the training process.
We combine the above two ideas and provide both student and teacher only with augmented versions of a single datum, and train the student to match the teacher’s imagined class-predictions of classes – almost all of which are not contained in the single datum, see Fig.
While practical applications do result from our method – for example we provide results on single image dataset based model compression in the Appendix – our goal in this paper is analyzing the fundamental question of how well neural networks trained from a single datum can extrapolate to semantic classes, like those of
CIFAR, SpeechCommands, ImageNet or even Kinetics.
What we ﬁnd is that despite the fact that the resulting model has only seen a single datum plus augmentations, surprisingly high quantitative performances are achieved: e.g . 74 % on CIFAR-100, 84% on SpeechCommands and 69 % top-1, single-crop accuracy on ImageNet-12.
We further make the novel observations that our method beneﬁts from high-capacity student and low-capacity teacher models, and that the source datum’s characteristics matter – random noise or less dense images yield much lower performances than dense pictures like the one shown in Figure 1.
In summary, in this paper we make these four main contributions:
A minimal framework for training neural networks with a single datum using distillation.
Extensive ablations of the proposed method, such as the dependency on the source image, augmentations and network architectures.
Large scale empirical evidence of neural networks’ ability to extrapolate on > 12 vision and audio datasets.
Qualitative insights on what and how neural networks trained with a single image learn.
R ELATED W ORK
The work presented builds on top of insights from the topics of knowledge distillation and singleand no-image training of visual representations and yields insights into neural networks’ ability to extrapolate.
Distillation.
In the original formulation of knowledge distillation (KD), the goal is to train a typically lower capacity (“student”) model from a pretrained (“teacher”) model in order to surpass the performance of solely training with a label-supervised objective.
KD has also been explored extensively to train more performant and, or compressed student models from the soft-target predictions of teacher models (Ba & Caruana, 2014; Hinton et al., 2015;
Gou et al., 2021; Furlanello et al., 2018;
Czarnecki et al., 2017).
Similarly, other approaches have also been developed to improve transfer from teacher to student, including sharing intermediate layers’ features (Romero et al., 2014), spatial
Zagoruyko
Komodakis the networks (Tung & Mori, 2019), contrastive distillation (Tian et al., 2020a) or few-shot distillation (Li et al., 2020).
KD has also been shown to be an effective approach for learning from noisy labels (Li et al., 2017).
More recently, Beyer et al.
(2022) conducted a comprehensive empirical investigation to identify important design choices for successful distillation from large-scale vision models.
In particular, they show that long training schedules, paired with consistent augmentations (including MixUp ) for both student and teacher, result in better performances.
Without the training data.
Distilling knowledge without access to the original training dataset was originally proposed in 2017 (Lopes et al., 2017) and ﬁrst experiments of leaving out single
MNIST classes from the training data were shown in (Hinton et al., 2015).
Yet, this paradigm is gaining importance as many recent advancements have been made possible due to extremely large proprietary datasets that are kept private.
This has lead to either the sole release of trained models (Radford et al., 2021; Ghadiyaram et al., 2019; Mahajan et al., 2018) or even more restricted access to only the model outputs via APIs (Brown et al., 2020).
While the original work (Lopes et al., 2017), still required activation statistics from the training dataset of the network, more recent works do not require this “meta-data”.
These approaches are typically generation based (Chen et al., 2019;
Ye et al., 2020; Micaelli & Storkey, 2019;
Yin et al., 2020) and e.g . yield datasets of synthetic images that maximally activate neurons in the ﬁnal layer of teacher.
Related to this, there are works which conduct “dataset” distillation, where the objective is to distill large-scale datasets into much smaller ones, such that models trained on it reach similar levels of performance as on the original data.
These methods generate synthetic images (Wang et al., 2018; Radosavovic et al., 2018; Liu et al., 2019; Zhao & Bilen, 2021; Cai et al., 2020), labels (Bohdal et al., 2020), or both (Nguyen et al., 2021), but due to their meta-learning nature have only been successfully applied to smallscale datasets.
In contrast to GAN- and inversion-based methods, as well as to dataset distillation, our approach does not require the knowledge of the weights and architecture of the teacher model, and instead works with black-box “API”-style teacher models and much smaller “datasets” of just a single datum plus augmentations.
This paper follows a radically different goal: We are interested in analysing the potential of extrapolating from a single image to the manifold of natural images, for which we choose KD as a well-suited and simple tool.
Prior knowledge in deep learning.
Finally, this work is related to several works which have analysed the infusion of prior knowledge to the training process.
For example the prior knowledge contained in neural network architectures (Jarrett et al., 2009; Zhou et al., 2019; Sreenivasan et al., 2021; Kim et al., 2021; Baek et al., 2021; Ulyanov et al., 2018) or image augmentations (Xiao et al., 2020).
In this work, we instead view a single image as a “prior” for all other natural images, which to the best of our knowledge has not been explored.
M ETHOD
We believe that simplicity is the key to demonstrating and analysing the question of how far a single image can take us.
We are inspired by recent work of Asano et al.
(2020), which “patchiﬁes” a single-image using augmentations, and the knowledge distillation method presented by Beyer et al.
(2022) and our technical contribution lies in unifying them to a single-image distillation framework.
i. Dataset generation.
In (Asano et al., 2020), a single “source” image is augmented many times to generate a static dataset of ﬁxed size.
This is done by applying the following augmentations in sequence: cropping, rotation and shearing, and color jittering.
For this, we follow their ofﬁcial implementation 1 , and do not change any hyperparameters
We do however analyze the choice of source images more thoroughly by additionally including a random noise, and a Hubble-telescope image to our analysis.
In addition, we also conduct experiments on audio classiﬁcation.
For generating a dataset of augmented audio-clips, we apply the set of audio-augmentations from (Bitton & Papakipos, 2021), which consist of operations, such as random volume increasing, background noise addition and pitch shifting to yield log-Mel spectrograms from raw waveforms (see Appendix for complete details).
ii.
Knowledge distillation.
The knowledge distillation objective is proposed in (Hinton et al., 2015) to transfer the knowledge of a pretrained teacher to a lower capacity student model.
In this
https
[Table] [Table]
case, the optimization objective for the student network is a weighted combination of dual losses: a standard supervised cross-entropy loss and a “distribution-matching” objective that aims to mimic the teacher’s output.
However, in our case there are no class-labels for the patches generated from a single image, so we solely use the second objective formulated as a Kullback–Leibler (KL) divergence between the student output p s and the teacher’s output p t :
[Equation]
where c are the teachers’ classes and the outputs of both student and teacher are temperature τ ﬂattened probabilities, p = softmax ( l/τ ) , that are generated from logits l .
For training, we follow (Beyer et al., 2022) in employing a function matching strategy, where the teacher and student models are fed consistently augmented instances, that include heavy augmentations, such as MixUp (Zhang et al., 2018) or CutMix (Yun et al., 2019).
However, in contrast to (Beyer et al., 2022), we neither have access to TPUs nor can train 10 K epochs on ImageNet-sized datasets.
While both of these would likely improve the quantitative results, we believe that this handicap is actually blessing in disguise: This means that the results we show in this paper are not speciﬁc to heavy-compute, or extremely large batch-sizes, but instead are fundamental.
E XPERIMENTS 4. I MPLEMENTATION DETAILS
Source data.
For the source data, we utilize the single images of (Asano et al., 2020), except for one image, which we replace by a similar one as we could not retrieve its licence.
These images are of sizes up to 2,560x1,920 (the “City” image of Fig. 1).
For the audio experiments, we use two short audio-clips from Youtube, a 5 mins BBC newsclip, as well as a 5 mins clip showing 11 Germanic languages.
Images and audio visualizations, sources and licences are provided in Appendix A.
Tasks.
For simplicity, we focus on classiﬁcation tasks here and provide results on a sample application of single-image based model compression in Appendix B.10.
For our ablations and smallscale experiments we focus on CIFAR-10/100 (Krizhevsky et al., 2009).
For larger-scale experiments using 224 × 224 sized images, we evaluate our method on datasets with varying number of classes (see Table 7) and additionally conduct experiments in the audio domain (see Table 5) and video (see Table 6).
Further implementation details are provided in Appendix C.
4. A BLATIONS
1-image vs full training set.
We ﬁrst examine the capability to extrapolate from one image to small-scale datasets, such as CIFAR-10 and CIFAR-100.
In Table 1, we compare various datasets for distilling a teacher model into a student.
We ﬁnd that while distillation using the source dataset always works best ( 95 .
26% and 78 .
06% ) on CIFAR-10/100, using a single image can yield models which almost reach this upper bar ( 94 .
1% and 73 .
Moreover, we ﬁnd that one image distillation even outperforms using 10 K images of CIFAR-10 when teaching CIFAR-100 classes even though these two datasets are remarkably similar.
To better understand why using a single image works,
we
[Table]
next disentangle the various components used in the training procedure: (a) the source image, (b) the generated image dataset size and (c) the augmentations used during distillation, corresponding to Tables 3a and 3c.
Choice of single image.
From Table 3a, we ﬁnd that the choice of source image content is crucial: Random noise or sparser images perform signiﬁcantly worse compared to the denser “City” and “Animals” pictures.
We compare 23 further images in Fig.
2 and ﬁnd that distillation quality roughly correlates with the density in images and JPEG sizes.
This is in contrast to self-supervised pretraining of (Asano et al., 2020), suggesting that the underlying mechanisms are different.
Varying loss functions.
In Table 3b, we show that the student can learn even with much degraded learning signals.
For example even if it receives only the top-5 predictions or solely the argmax prediction ( i.e . a hard label) of the teacher without any conﬁdence value, the student is still able to extrapolate at a signiﬁcant level ( > 91% / 60% for CIFAR-10/100).
Even at ImageNet scale, the performance remains at a high 43 .
8% top-1 accuracy (see Table 7 row (m) ).
This also suggests that copying models solely from API outputs is possible, similar to (Orekondy et al., 2019).
We also evaluated L1 and L2 distillation functions (see Table 9 for details) and found that these perform slightly worse.
This echoes the ﬁnding of (Tian et al., 2020a) that standard KD loss of Eq.
(1) is actually a strong baseline, hence we use this in the rest of our experimental analysis.
Augmentations.
In Table 3c, we ablate the augmentations we use during knowledge distillation.
Besides observing the general trend of “more augmentations are better”, we ﬁnd that CutMix performs better than MixUp on our single-image distillation task.
This might be because in our case the model is tasked with learning how to extrapolate towards real datapoints, while MixUp is derived and therefore might be more useful for interpolating between (real) data samples (Zhang et al., 2018;
Beyer et al., 2022).
Comparison to synthetic datasets.
In Table 2, we ﬁnd that there is something unique about using a single image, as our method outperforms several synthetic datasets, such as FractalDB Kataoka et al.
(2020), randomly initialized StyleGAN Baradad et al.
(2021), as well as the GAN-based approach of (Micaelli & Storkey, 2019).
This is despite the fact that these synthetic datasets contain ∼ 50 K images.
We provide insights on the characteristics of this 1-image dataset in Section 4.4.
[Table] [Table]
Next, using the insights gained in this section, we scale the experiments towards other network architectures, dataset domains and dataset sizes.
4. G ENERALISATION
Architectures In Table 4, we experiment with different common architectures on CIFAR-10 and
CIFAR-100.
On CIFAR-10, we ﬁnd that almost all architectures perform similarly well, except for the case of ResNet-56 to ResNet-20 which could be attributed to the lower capacity of the student model.
We ﬁnd that the distillation performance of WideR40-4 to WideR40-4 on CIFARis very close to distilling from original source data; achieving accuracy of 94 .
14% and only around one-percentage point lower than supervised training.
This analysis shows that our method generalizes to other architectures hints that larger capacity student models might be important for high performances.
Extension to audio.
So far, our proposed approach has shown success in distilling useful knowledge for images.
To further test the generalizability of our approach on other modalities, we conduct experiments on several audio recognition tasks of varying difﬁculty.
We perform distillation via 50 K randomly generated short audio clips from two ( i.e . Ours A and Ours B ), 5 mins YouTube videos (see Appendix A for more details).
In Table 5, we compare the results against the performance of the teacher model and distilling directly using source dataset.
We ﬁnd that even for audio, distilling with merely a single audio clip’s data provides enough supervisory signal to train the student model to reach above 80% accuracy in the majority of the cases.
In particular, we see signiﬁcant improvement in distillation performance when a single audio has a wide variety of sounds to boost accuracy on challenging Voxforge dataset from 72 .
8% to 78 .
Our results on audio recognition tasks demonstrate the modality agnostic nature of our approach and further highlight the capability to perform knowledge distillation in the absence of large amount of data.
Extension to video.
We conduct further experiments on video action recognition tasks.
As distillation data, we generate simple pseudo-videos of 12 frames that show a linear interpolation of two crop locations at taken at the beginning and end for the City image and use X3D video architectures (Feichtenhofer, 2020) as teachers and students, see Appendix B.1 for further details.
In
Top-1 accuracy is shown with 10temporal center-cropped clips per video.
Details in
[Table]
[Figure]
[Table]
Table 6, we ﬁnd that our approach indeed generalizes well to distilling video models with performances of 75.2% and 51.8% on UCF-101 (Soomro et al., 2012) and K400 (Carreira & Zisserman, 2017)–despite the fact that the source data is not even a real video.
4. S CALABILITY TO LARGE - SCALE DATASETS AND MODELS
From this section on, we scale our experiments to larger models utilizing 224 × 224-sized images, and evaluate these on various vision datasets.
Varying datasets.
In Table 7 we ﬁnd that, overall, a single image is not enough to fully recover the performance on more difﬁcult datasets.
This might be because of the ﬁne-grained nature of these datasets, e.g .
ImageNet includes more than 120 kinds of dogs and Flowers contains 102 types of ﬂowers.
Nevertheless, we ﬁnd a surprisingly high accuracy of 69% on ImageNet’s validation set, even though this dataset comprises 1000 classes, and the student only having seen heavily augmented crops of a single image.
In Table 8, we conduct further analyses on distilling ImageNet models.
Patchiﬁcation vs more data.
At the top part of the Table 8, we analyse the effect of using original vs patchiﬁed (p) random images from the training set of ImageNet.
We ﬁnd that while forand 100 source images, patchiﬁcation improves performance, this is not the case when a larger number of images is used.
This indicates that increased diversity is especially a crucial component for the small data regime, while capturing larger parts of objects becomes important only beyond this stage.
We also make the surprising observation that patches from a single high-quality image (“City”) obtain better performances than patches generated from 1000 ImageNet training images.
To understand why this might be the case, we note that the high coherency of training patches from a single image strikingly resembles what babies see during their early visual development, e.g ., looking at only few toys and people but from many angles (Bambach et al., 2018; Orhan et al., 2020).
It is hypothesized that this unique combination of coherence and variability is ideal for learning to recognize objects (Orhan et al., 2020).
Teacher/student architectures.
In Fig.
4, we analyze the performances of varying ResNet models by depth or width.
We ﬁnd that overall varying the width is a more parameter-efﬁcient way to obtain higher performances, almost reaching the teacher’s 69 .
5% performance with a ResNet-50x at 69 .
In addition, in the bottom half of Table 8 (rows g-n ), we ﬁnd surprising results: First, in contrast to normal KD, we ﬁnd that the teacher’s performance is not directly related to the ﬁnal student performance, e.g . a ResNet-50 is less well-suited for distilling than a ResNet-18 (rows i vs l ).
[Table]
[Figure]
Teacher is a ResNet-18, settings as in Table 7 with half batch size.
Second, we ﬁnd that the choice of source image is even more important for ImageNet classiﬁcation (see rows g-i ), as switching from the “City” even to the “Bridge” one decreases performance considerably and the Noise image does not train at all.
Third, we ﬁnd that the best settings are those in which the student’s capacity is higher than that of the teacher (rows i,n and Fig. 4).
In fact this
R18 → R50x2 setup (row n ) obtains a performance on ImageNet-12 of 69 . 0% which matches the teacher’s performance within 0 .
In Appendix B.3, we further compare distillations to ViT architectures (Dosovitskiy et al., 2021), but generally ﬁnd lower performances (up to 64%), showing that the inductive biases of CNNs might positively aid in learning.
4. A NALYSIS
To better understand how our method is achieving these performances, we next analyse the learned models.
Output conﬁdence scores.
In Fig.
3, we visualize the distribution of the conﬁdence scores of the predicted classes for student and teacher models for the training and evaluation data.
We observe that the student indeed learns to mimic the teacher’s predictions well for the patch training data.
However, during evaluation on CIFAR images, the student model exhibits a much broader distribution of values, some with extremely high conﬁdence scores.
This might indicate that the student has learned to identify many discriminative features of object classes from the training data, which are all triggered when shown real examples, leading to much higher conﬁdence values.
Feature space analysis.
Next, we analyse the network’s embedding of training patches when compared to validation-set inputs.
For this, we ﬁt a t -SNE (Van der Maaten & Hinton, 2008) using the features of 5 K training patches and 5 K test set images in Fig.
We ﬁnd that, as expected, the training patches mapped to individual CIFAR classes do not resemble real counter parts.
Moreover, we observe that all the training patches are embedded close to each other in the center, while CIFAR images are clustered towards the outside.
This shows that the network is indeed learning features that are well suited for effective extrapolation.
Neuron visualizations.
We next analyze a single-image distilled model trained to predict IN- 1 k classes.
In Fig.
6, we visualize four ﬁnal-layer neurons using the Lucid (Olah et al., 2017) library.
When we compare neurons of the standard ImageNet supervised and our distilled model, we ﬁnd that the neurons activate for very similar looking inputs.
The clear neuron visualizations for “panda” or “lifeboat” (and more are provided in Appendix B.6) are especially surprising since this network was trained using only patches from the City Image and has never seen any of these objects during its learning phase.
D ISCUSSION
The augmented image prior for extrapolation.
The results in this paper suggest a summary formulation as follows:
Within the space of all possible images I , a single real image x ∈ I and its augmentations A ( x ) can provide sufﬁcient diversity for extrapolating to semantic categories in real images.
Limitations.
In this paper, we were mainly concerned with showing that extrapolating from a single image works, empirically.
Due to the limited nature of our compute resources, we have not exhaustively analyzed the choice of initial image or patchiﬁcation augmentations, and longer training (as shown in (Beyer et al., 2022)) would further improve performances.
Potential negative societal impact.
While our research question is of fundamental nature, one possible negative impact could be that the method is used to steal models and thus intellectual property from API providers – although in practice the performance varies especially for large-scale datasets (see Table 7).
Conclusion and outlook.
In this work, we have analyzed whether it is possible to train neural networks to extrapolate to unseen semantic classes with the help of a supervisory signal provided by a pretrained teacher.
Our quantitative and qualitative results demonstrate that our novel single-image knowledge distillation framework can indeed enable training networks from scratch to achieve high accuracies on several architectures, datasets and domains.
This demonstrates that knowledge distillation can be done with just a single image plus augmentations, and also raises several further research questions, such as the dependency of the source image and the target semantic classes; how networks combine features for extrapolation
; and the role and informational content of augmentations; all of which we hope inspires further research
A A DDITIONAL V ISUALIZATIONS A. I NPUT IMAGES
Image sources and licences.
From top to bottom, the images in Fig.
7 have the following licences and sources: (a): CC-BY (we created it), (b) 2 public domain (NASA) (c) 3 CC BY-SA 3.0 (by David
Ball), (d) 4 pixabay licence (personal and commercial use allowed), (e) 5 personal use licence.
Images (b), (d) and (e) are identical to the ones in the repo of (Asano et al., 2020), while (a) we have recreated on our own and (c) is a replacement for a similar one whose licence we could not retrieve. [Figure]
Audio source.
For the experiment on audio representation distillation, we use a 5 . 5 mins English news-clip taken from BBC about how can Europe tackle climate change 6 and a 5 mins clip about
Germanic languages 7 .
A. T RAINING PATCHES AND PREDICTIONS
In Fig. 8, we show several training patches as they are being used during training along-side the teacher-network’s temperature-adjusted predictions.
Only very few training samples can be interpreted ( e.g . 3 rd row: “nail” or “slide rule”), while for most other patches, the teacher is being very “creative”. [Figure]
In Fig.
9, we show a similar plot for the training patches used in CIFAR-10 training.
Here we show the whole teaching signal, and highlight the top- 5 predictions of the teacher for visualization.
Finally, in Fig.
we show some some augmented training samples for the audio classiﬁcation experiment.
This ﬁgure shows how various spectrograms can be generated from a single clip by utilizing many augmentations. [Figure]
B A DDITIONAL E XPERIMENTS B. V IDEO ACTION RECOGNITION
We conduct further experiments on video action recognition tasks.
For this we use the common
UCF-101 (Soomro et al., 2012) dataset, more speciﬁcally the ﬁrst split.
As distillation data, we generate simple source videos of 12 frames that show a linear interpolation between two crops of the City image.
We generate 200K of these videos and apply AugMix (Hendrycks et al., 2019) and
CutMix (Yun et al., 2019) during the distillation for additional augmentations.
For the architectures we use the recent, state-of-the-art X3D architectures (Feichtenhofer, 2020) as they obtain strong performances and allow for ﬂexible scaling of network architectures.
We use a X3D-XS model as the teacher, which is trained with inputs of 160 × 160 and 4 frames and a temporal subsampling factor of 12 .
For the student model we scale this X3D-XS model in terms of width and depth by increasing the depth factor and width factor from 2 .
2 to 3 and 4 respectively, yielding a network with 18 .
1 M parameters compared to 3 .
2 M.
For UCF-101 we ﬁnetune the teacher model starting from supervised
Kinetics-400 (Carreira & Zisserman, 2017) pretraining, which achieves a 90 . 4 % performance.
We use a batch size of 128 per GPU on two GPUs, a learning rate of 10 − 3 , a temperature of 5 , cosine learning rate schedule with a warmup of 5 epochs, with no weight decay and no dropout.
As this experiment takes even longer than the image ones, we have not conducted systematic ablations on these parameters.
Instead, we merely picked ones which looked promising after a few epochs but believe that even this small experimental evidence is enough to investigate to what extent these 1-image “fake-videos” can be used for learning to extrapolate.
The results are given in Table 6.
We observe a performance of over 72 % on UCF-101 using just a single fake-video as the training data along with the pretrained teacher.
While this number is far below the state of the art or the teacher’s performance, it for example outperforms CLIP’s 69 .
8 % zero-shot performance with a ViT-B16 model and shows that the student is able to extrapolate to the action classes of UCF from just one datum.
B. V ARYING D ISTILLATION L OSS F UNCTIONS
In Table 9 we compare the standard KD loss against L1 and L2 losses (using scaled logits as in the
KD loss) for training the student.
We ﬁnd that while for CIFAR-10 the choice of the distillation method does not impact the performance, on the more difﬁcult ImageNet dataset, there is a small difference of around − 0 .
4% and − 0 .
5% for L1 and L2 losses as compared to the KD loss at epoch 30.
This shows ﬁrstly that our method is not dependend on a speciﬁc loss for distilling knowledge from a teacher to a student.
Secondly, it shows that the standard KD loss is actually a strong baseline, as was also reported in (Tian et al., 2020a).
[Table]
B. I MAGE N ET V ISION T RANSFORMER D ISTILLATIONS
In Table 10 we compare the best results from the main paper of distilling to ResNets to distilling to the ViT (Dosovitskiy et al., 2021) and CaiT (Touvron et al., 2021) architectures.
We observe that while convolutional networks learn signiﬁcantly faster, with e.g . differences of more than 30% at the tenth epoch.
While the ViT architecture almost catches up at epoch 200 with a ﬁnal performance of 63.9%, the CaiT model remains fairly low at 58.5%.
Nevertheless, this experiments shows that distilling across architecture types works and that even less constrained Vision Transformers can be trained with our method simply.
B. T RAINING WITH VARIOUS RANDOM NOISES
In Table 11, we experiment with adding various types of noise structures as inputs to the teacher and student instead of augmented patches from the images.
Notably, we either input random uniform noise between [0,1] (row ( w )), random normal noise with a mean of 0 and standard deviation of
[Table]
(row ( x )) and convex combinations of the augmented input patches with the normal noise.
We ﬁnd that just like the augmented patches from a random noise image (row ( g )), random noise as input also does not work for distilling semantic classes.
While generating new random noise at every iteration does create more variability, in practice, a random noise image with augmentations likely already achieve a high amount of variance, showing that this is not the crucial component for learning, but instead having structures from a real image.
This is further conﬁrmed by the steady increases in performance when moving from row ( x ) to ( z ).
[Table]
B. S TANDARD DEVIATIONS IN PERFORMANCE
In Table [Table] we report the performance of ﬁve independent runs on CIFAR-10 dataset to further highlight the robustness of our experimental results.
We notice consistent accuracy scores across different training runs with minor changes of ≈ ± 0 .
[Table]
N EURON ACTIVATION MAXIMIZATIONS
In Fig.
[Table]
we show further class-neuron activation maximization visualizations for the R50 → R distillation experiment and compare them to the corresponding ones of the supervised teacher model.
Similar to the examples in the main paper, we ﬁnd that the neurons visualizations are remarkably similar.
In Fig. [Table] we show further visualizations of the trained student model from the R50 → R50 distillation experiment.
We show ﬁve randomly chosen neurons for every layer in the ResNet.
[Figure]
We visualize representations of distilled student model on CIFAR-10 validation set in order to highlight semantic relevance of the features in Fig.
We extract the feature from penultimate layer of WideR16-4 model for computing 2 -D t-SNE embeddings.
The visualization clearly highlights that distilled features contain substantial amount of semantic information to correctly differentiate between different object categories.
B. C ENTERED K ERNEL A LIGNMENT COMPARISON
In Fig.
15, we visualize similarity between different layers of the models trained in a general supervised manner and with distillation using patches of 1 -image for CIFAR-10 and ImageNet datasets.
We use centered kernel alignment (Kornblith et al., 2019) method to identify the resemblance of features learned with these different training approaches.
We notice high similarity in representations, which reinforce our empirical evaluation that the student models indeed learn useful features required for differentiating between semantic categories.
B. S INGLE - IMAGE TRAINING DATA VS TODDLER DATA
In Fig.
16a, we compute the distances of GIST features (Oliva & Torralba, 2001) of 10 K training images of the “City” image at resolution 256 × 256 in Fig. 16a.
Following (Bambach et al., 2018), we
L2 normalize these GIST features before computing pair-wise distances and plotting the histogram of the values.
To make the comparison of our training data with the visual inputs of toddlers more concrete, we compare low-level GIST (Oliva & Torralba, 2001) features of our 1-image dataset against those computed from visual inputs of a toddler in (Bambach et al., 2018).
As we do not have access to the dataset of (Bambach et al., 2018), we copy their Figure 3b for reference in Fig. 16b.
While the GIST distances for ImageNet are very different with a mean around 0 .
75 (Bambach et al., 2018), we ﬁnd that our single image dataset’s distribution in Fig.
16a closely resembles that of the visual inputs of a toddler.
While this is one possible explanation for why this type of data might work well for developing visual representations, further research is still required. [Figure]
B. D ATA - FREE P RUNING AND Q UANTIZATION OF P RE - TRAINED M ODELS
Neural network compression has been studied extensively in the literature to produce light-weight models with the objective of improving computational efﬁciency.
In addition to knowledge distillation, network pruning and quantization are other well-known approaches.
In the former, the goal is to remove or sparsify (zeroing-out) the network’s weights based on a speciﬁc criteria, such as subset of weights with the lowest magnitude.
The later is concerned with reducing precision from 32 -bits ﬂoating numbers to 8 -bits and even single bit as binary neural networks.
In general, from practical standpoint post-training compression is largely employed in conjunction with ﬁne-tuning to avoid accuracy drop.
Here, we study utilize our proposed single-image distillation framework for post-training model compression without using any real data.
Speciﬁcally, we ask the question, whether a pre-trained model can be compressed when no in-domain data is available?
To this end, we utilize Tensorﬂow
Model Optimization Toolkit 8 to prune and quantize various pre-trained model on CIFAR-10 and 8 https://www tensorflow. /model optimization
[Figure]
a
[Figure] [Table]
CIFAR-100 datasets.
We use self-distillation, where a pre-trained model acts as teacher and student clone of it is compressed during ﬁne-tuning phase.
In Table 13, we report accuracy scores for the pruned and quantized models in comparison with the standard models and those compressed using real in-domain data.
In all the cases, we can notice that there is negligible loss in accuracy, i.e., < = 1% , while using random patches. [Table]
[Figure] Further, in Table 14, we vary sparsity ratio and can observe that a model can be pruned up to 75% sparsity with three-points drop in accuracy but with 25% pruning ratio the loss is merely noticeable.
In Figure 17, we analyze representational similarity of pruned WideRNet16-4 model with the original trained on CIFAR-100, the diagonal entries highlight that even after 50% sparsity the representations are very similar.
To summarize, with model compression use case, we barely scratched the surface of what is possible with our single-image distillation framework and we hope it will inspire further studies and applications in other domains. B. T RANSFER LEARNING EXPERIMENTS
In this section, we compare the performances on downstream tasks of our teacher network (ResNet18), our single-image distilled network (ResNet-50) and another ResNet-50, purely trained in a selfsupervised manner on our augmented single-image dataset. For this, we pretrain using the ofﬁcial code of MoCo-v2 (He et al., 2020) in the standard 200 epoch setting, while applying their default “v2” augmentations during training. Evaluation. We follow standard linear evaluation procedure from the MoCo-v2 repo.
which uses a batch size of 256 , learning rate of 30 which is multiplied by 0 .
1 at epochs 60 and 80 for a total of 90 epochs.
For the data-efﬁcient full-ﬁne tuning, we utilize the implementation from SwAV (Caron et al., 2019), which trains for 20 epochs using a cosine decay learning rate schedule and a batch size of 256 . For the data-efﬁcient SVM classiﬁcation experiments, we follow the implementation of (Li et al., 2021) and report average results from 5 trials and keep the SVM’s cost parameter ﬁxed at a value of 0 . 5 . For the COCO object detection experiments we use the detectron2 repo (Wu et al.,
We compare our distillation approach (R18 → R50) against pretraining via MoCo on the same
[Table]
architectures.
The mCE value mean Corruption
and the 1x schedule using a FPN (Lin et al., 2017) as in (He et al., 2020).
We additionally vary the learning rates from 1x and 4x as the models are trained with very different losses compared to the supervised variant for which the learning schedule is made.
The results are shown in the last row of Table 15 and show that our models beneﬁt from a higher initial learning rate, which also does not have any negative effect on training longer e.g . with the 2x schedule, while the MoCo pretrained model works best with the 1x learning rate.
We ﬁnd that freezing the backbone and only training a linear layer on top of it, can achieve performances of 68 .
7 % and 47 . 2 % for ImageNet and Places, respectively, vastly outperforming the self-supervised variant. A similar trend is observed for data-efﬁcient classiﬁcation reaching close but lower performances to the teacher model. B. R OBUSTNESS EXPERIMENTS
In Table 16, we report results on ImageNet-C
(Hendrycks & Dietterich, 2019).
B. A NALYSIS OF PER - CLASS ACCURACIES
Worst underperforming 10 classes.
The 10 classes that have the highest top-1 accuracy difference compared to the teacher model (for the R18 → R50 setting) are: ‘Tibetan terrier,
Lakeland terrier, golden retriever, dhole, Border terrier, cocker spaniel, Welsh springer spaniel, Brabancon griffon, collie, mountain bike‘ , with accuracies differences ranging from -34% to -20%.
From the underperforming classes, we ﬁnd that 9/10 of these are dog breeds.
This illustrates how out single-image trained model lacks behind in very ﬁne-grained classiﬁcation compared to the
ImageNet trained model.
This is likely because the single-image lacks the necessary structures and patterns for disambiguating these classes despite the help of augmentations.
In Fig.
18, we plot the validation performance against the frequency of how often the class appears as a top-1 prediction of the teacher during 1 epoch of training and ﬁnd that it is unrelated, showing the student is proﬁting heavily from the knowledge contained in the soft-predictions, echoing ﬁndings from (Hinton et al., 2015; Furlanello et al., 2018).
We further compare these per-class performances
[Figure] [Figure] against the teacher’s performance in Fig. 19 and ﬁnd
that there is no relationship between when the student under or overperforms the teacher vs how often a particular likeness of a class appears.
C I MPLEMENTATION D ETAILS
All code will be released open-source and is attached as supplementary information to this submission.
C. I NITIAL PATCH GENERATION
We do not tune the individual patch generation and instead adapt the procedure directly from (Asano et al., 2020) 9 .
The augmentations applied to the input image of size H × W to arrive at individual patches of size P × P is (in order) as follows:
1) RC(size=0.5*min
( H,W )) 2)
RRC(size=(1.42* P ), scale=(2e-3, 1)) 3) RandomAffine(degrees=30, shear=30) 4) RandomVFlip(p=0.5 ) 5) RandomHFlip(p=0.5 ) 6) CenterCrop(size=( P , P )) 7) ColorJitter(0.4,0.4, 0.4,0.1, p=0.5)
All transformations are standard operations in PyTorch: RC stands for RandomCrop, i.e . taking a random crop in the image with a speciﬁc size; RRC for RandomResizedCrop, i.e . taking a random sized crop withing the size speciﬁed in the scale tuple (relative to the input);
RandomAfﬁne for random afﬁne transformations (rotation and shear); RandomVFlip and RandomHFlip for random ﬂipping operations in the vertical and horizontal direction, and CenterCrop crops the image in the center to a square image of size P × P .
Finally, ColorJitter computes photometric jittering, where the parameters for the strengths are given in the order of brightness, contrast, saturation and hue and applied with a certain probability.
Similarly, for audio clips generation given a single audio, we use augmentation operations from (Bitton & Papakipos, 2021) with default settings.
Speciﬁcally, to create a single example we apply the following procedure: we randomly crop a segment of 2 -seconds, and use randomly sample an augmentation function to create transformed instances and save them in mono format.
In our work, we use these augmentations, all with their default settings:
1) add-background-noise 2) change-volume 3) clicks 4) clip 5) harmonic 6) high-pass-filter 7) low-pass-filter 8) normalize 9) peaking-equalizer 10) percussive 11) pitch-shift 12) reverb 13) speed 14) time-stretch
C. C OMPUTING LOG -M EL SPECTROGRAMS
The log-Mel spectrograms are generated on-the-ﬂy during training from a randomly selected 1second crop of an audio waveform as the model’s input.
We compute it by applying a short-time
Fourier transform with a window size of 25 ms and a hop size equal to 10 ms to extract 64 Melspaced frequency bins for each window.
During evaluation, we average over the predictions of non-overlapping segments of an entire audio clip.
C. A UDIO NEURAL NETWORK ARCHITECTURE
Our audio convolutional neural network is inspired by (Tagliasacchi et al., 2019) and it consists of four blocks.
We perform separate convolutions in each block with a kernel size of 4 .
One on the temporal and another on the frequency dimension, we concatenate their outputs afterward to perform a joint 1 × 1 convolution.
It allows model to capture ﬁne-grained features from each dimension and discover high-level features from shared output.
We apply L 2 regularization with a rate of 0 .in
each convolution layer and also use group normalization (Wu & He, 2018).
Between the blocks, we utilize max-pooling to reduce the time-frequency dimensions by a factor of 2 and use a spatial dropout rate with a rate of 0 .
2 to avoid over-ﬁtting.
We apply ReLU as a non-linear activation function and feature maps in the convolutions blocks are 24 , 32 , 64 , and 128 .
Finally, we aggregate the feature with a global max pooling layer which are fed into a fully-connected layer with number of units equivalent to the number of classes.
C. T RAINING C.4. O PTIMIZATION .
For each of these datasets, we ﬁrst train a teacher network in a usual supervised manner, that is then used for the distillation.
For the distillation to the student model, we use a temperature τ =motivated by ﬁndings in (Beyer et al., 2022).
We keep this temperature ﬁxed throughout the whole of the paper due to limited compute.
For optimization, we use AdamW (Loshchilov & Hutter, 2018).
C.4. S MALL - SCALE EXPERIMENTS
For experiments on CIFAR- 10 , CIFAR- 100 , and other smaller datasets, we use Tensorﬂow for running experiments on a single T 4 GPU with a batch size of 512 using an Adam (Kingma & Ba, 2015) optimizer with a ﬁxed learning rate of 0 .
We use standard augmentations including, random left right ﬂip, and random crop.
The supervised models also uses cutout augmentations with a cutout size of 16 × 16 .
For Mix-up, we sample α uniformly at random between zero and one.
In Cut-Mix, we use a ﬁxed value of 0 .
25 for α and β .
With the following setup, each of the single image distillation experiment of 1 K epochs took around 2 − 3 days.
The supervised and standard (using source data) distillation models are trained for 100 epochs (per epoch 2000 steps) with a batch size of 128 using SGD for optimization.
For VGG (Simonyan & Zisserman, 2014) and
ResNet (He et al., 2016) models, we use a learning rate schedule of 0 . 01 , 0 . 1 , 0 . 01 , 0 .
001 decayed at following steps 400 , 32000 , 48000 , 64000 with momentum of 0 . 9 .
For WideResNet (Zagoruyko & Komodakis, 2016a), we use a learning rate schedule of 0 . 1 , 0 .
0008 that is decayed at following steps 24000 , 48000 , 64000 , 80000 with Nesterov enabled.
In audio experiments, we use an Adam (Kingma & Ba, 2015) optimizer with a ﬁxed learning rate of 0 .
001 and use batch size of 128 and 512 for standard models and single-clip distillation, respectively.
Furthermore, we also utilize Mix-Up augmentation during our knowledge distillation experiments.
C.4. L ARGE - SCALE EXPERIMENTS
We use PyTorch’s DistributedDataParallel engine for running experiments on 2 A
GPUs in parallel with batch-sizes of 512 each.
For optimization we use AdamW (Loshchilov &
Hutter, 2018) with a learning rate of 0 .
01 and a weight-decay of 10 − 4 .
These values were determined by eyeballing the results from (Beyer et al., 2022) to ﬁnd a setting that might generalize across datasets, as we do not have enough compute to run hyper-parameter sweeps.
With this setup, each 200 epoch experiment took around 5 days.
We have also found that halving the batch-size performs equally well, and is more amenable to lower-memory GPUs.
For the distillation experiments, we use Cut-Mix (Yun et al., 2019) with its default parameters of α = β = 1 . 0 .
C. GIST FEATURES COMPARISON
In Fig.
16a, we compute the distances of GIST features (Oliva & Torralba, 2001) of 10 K training images of the “City” image at resolution 256 × 256 in Fig. 16a.
Following (Bambach et al., 2018), we
L2 normalize these GIST features before computing pair-wise distances and plotting the histogram of the values.