L EARNING FROM O NE AND O NLY O NE S HOT
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans can generalize from only a few examples and from little pre-training on similar tasks.
Yet, machine learning (ML) typically requires large data to learn or pre-learn to transfer.
Inspired by nativism, we directly model basic human-innate priors in abstract visual tasks e.g., character/doodle recognition.
This yields a whitebox model that learns transformation-based topological similarity by mimicking how humans naturally “distort” an object when ﬁrst seeing it.
Using the simple nearest-neighbor classiﬁer in this similarity space, our model achieves human-level character recognition using only 1 – 10 examples per class and nothing else (no pretraining).
This differs from few-shot learning (FSL) using signiﬁcant pre-training.
On standard benchmarks MNIST, EMNIST-letters, and the Omniglot challenge, our model outperforms both neural-network-based and classical ML in the “tiny-data” regime, including FSL pre-trained on large data.
Further, mimicking k -means but in a non-Euclidean space, our model enables unsupervised learning and generates human-interpretable archetypes as cluster “centroids”.
I NTRODUCTION
Modern machine learning (ML) has made remarkable progress, but this is accompanied by increasing model complexity, with hundreds of neural layers (e.g., ResNet-152) and millions of parameters (e.g., AlexNet:
3 M, VGG16: 138 M, BERT: 110 M, GTP-3: 175 B).
This results in a huge appetite for data and increasing difﬁculty in model interpretability—both for users to understand and for developers to tune (e.g., hyperparameters, architecture).
As such, AI researchers have pushed for ML models that are prior- and data-efﬁcient (Chollet, 2019), that are human-like (Lake et al., 2015), and that exhibit human-interpretable behaviors (Adadi & Berrada, 2018).
This poses the fundamental scientiﬁc question: How can humans learn so much from so little (May, 2015), whereas ML models, e.g., those achieving near-perfection on MNIST using all 60 k training images, deteriorate rapidly as (pre)training reduces?
Being data-hungry can pose a real challenge in data-scarce domains, e.g., in a rapidly evolving pandemic or a low-resource environment.
Focusing on its scientiﬁc contribution , this paper presents a theoretically sound, white-box model that learns like humans do, with initial success on a ﬁrst set of benchmarks. This includes our state-of-the-art results of hitting 80% / 90% MNIST accuracy using the only ﬁrst / ﬁrst four training images per class and achieving 6 .
75% error (human performance is 4 . 5% ) in the Omniglot one-shot learning challenge without pre-training—one and only one shot.
We follow the nativist principle: given an example, humans make abstractions : we envision an equivalence class of unseen examples, equivalent to the given, in several abstract senses.
Many abstraction abilities are inborn in humans and occur unconsciously.
When a baby sees a mug, (s)he can immediately recognize it regardless of whether it is translated, rotated, scaled, or deformed.
Such abstraction abilities are considered innate as Core Knowledge priors (Spelke & Kinzler, 2007), rather than acquired later in life such as learning that a mug is topologically a doughnut.
We computationally realize the above intuition via our so-called distortable canvas —imagining every image smoothly painted on a rubber canvas that can be distorted in many ways, e.g., bent, stretched, squeezed (Figure 1A).
Due to rubber’s viscosity, more distorted canvas transformations expend more energy.
This induces a topological similarity, or distance, based on minimal energy : two images are similar if one can almost transform into the other with little distortion (energy).
Distance is computed by minimizing color and canvas distortions.
In general, visual similarity involves not only a canvas but also a color transformation.
Here, we focus on canvas transformations and grayscale images only. [Figure]
Besides a ﬁnal desired transformation (and distance), we apply the minimal-energy principle to the entire transformation process, i.e., to keep canvas and color distortions small along the entire optimization process (Mesa et al., 2019).
When perceiving a translation, our mind does not process it as a sudden displacement from one location to another, but auto-completes a translation path— continuous and preferably short.
Gradient descent naturally ﬁts this goal by always choosing the steepest descent.
Yet, it suffers from the curse of local minima .
Our solution is to lift gradient descent to multiple levels of abstraction via multiscale canvas lattices and color blurring, mimicking human abstraction ability that is extremely ﬂexible in multiscale optimization.
This yields visualizable and interpretable transformation ﬂows (Figure 1B) that either match human intuition (e.g., what humans would naturally do to transform a “7” to a “1”) or provide human intuition to initially nonintuitive settings.
The latter case suggests new, human-interpretable transformations: “ah, I did not realize this other way of transforming ‘7’ into ‘1’, but now I see it and it makes perfect sense!”
We show initial success on abstract visual tasks such as character and doodle recognition (future work generalizes to photorealistic images after a preprocessing technique to turn them into abstract doodles or “emojis”).
Our model can be used with the simple nearest-neighbor method to classify images and also in a simple k -means style to cluster images.
On benchmarks including MNIST (LeCun et al., 1998), EMNIST-letters (Cohen et al., 2017), and the more taxing Omniglot challenge (Lake et al., 2015), running nearest-neighbor on our learned similarity space outperforms both neural-networkbased and classical ML in the “tiny-data” or single-datum regime.
This includes beating few-shot models pre-trained on extra background data (which our model does not require).
On MNIST, we need one and only one training image per class to hit 80% accuracy and four to hit 90% .
In an unsupervised setting, our model enables k -means-like clustering, but on our learned similarity space.
We generate archetypes as cluster “centroids”, e.g., different ways of writing “7” or doodling giraffes.
Literature on data scarcity: few-shot learning (FSL).
One-/few-shot learning (Lake et al., 2011;
Wang et al., 2020) via transfer or meta learning (Pan & Yang, 2009; Finn et al., 2017; Hsu et al., 2019) has achieved impressive success in data-scarce scenarios. FSL’s success relies on the assumption that source and target tasks are similar enough for pre-training to be relevant (Storkey, 2009).
However, knowing a priori how relevant the tasks are and understanding what has been pre-learned are often considered “black art”. It remains challenging for FSL practitioners to pick proper source data/models for pre-training so as to transfer most effectively and avoid negative transfer (Pan & Yang, 2009; Meiseles & Rokach, 2020).
This is especially the case in new domains (e.g., discovering rising trends) over classical ones (e.g., vision, language).
So, rather than “big transfer” (Kolesnikov et al., 2020), the Omniglot challenge urges “small transfer”—advocating reduced pre-training (Lake et al., 2019).
This paper follows Omniglot’s pursuit and pushes such reduction to the limit: with absolutely zero pre-training, we still achieved near-human performance and human-interpretability.
Literature on data scarcity: transformation and data augmentation.
Our distortable canvas is conceptually akin to other transformation-based models, e.g., optimal-transport maps (Villani, 2009), transformation-induced descriptors (Lowe, 1999) and equivariances (Bronstein et al., 2017). Like all these models, we build transformations into the model rather than into the data as through data augmentation (Krizhevsky et al., 2012).
This has the native advantage of harnessing transformational properties directly rather than learning them from data. Models with built-in transformations may be viewed as those perfectly learned from inﬁnitely augmented data. Unlike transformations commonly considered in existing models and data augmentation techniques, we do not encode domain knowledge about preferred transformations such as translation, rotation, or scaling. Instead, our model considers all transformations while still maintaining efﬁciency via our abstracted gradient descent.
S MOOTH I MAGE ON D ISTORTABLE C ANVAS
We introduce a distortable canvas model: any image is thought of as smoothly painted on a rubber canvas that can be bent, stretched, etc.
We further introduce canvas transformations that can ﬂexibly “distort” an image as we naturally simulate in our mind.
More speciﬁcally, we deﬁne a smooth image by a piecewise differentiable M : R 2 → R + , where R 2 denotes an inﬁnite canvas and R + denotes color (grayscale in this paper).
We deﬁne a canvas transformation by α : R 2 → R 2 , which “reshapes” the underlying canvas of a smooth image.
Examples include translation, rotation, scaling, and more.
We also deﬁne a color transformation by χ : R + → R + , which “repaints” a color.
In this paper, we simplify color transformation and only use it to adjust image contrast via afﬁne χ ( c ) :
= ac + b .
Nevertheless, we do not restrict canvas transformation, but consider all 2D transformations. Given
M , α, χ , the composition χ ◦ M ◦ α denotes the transformed image of M by transformations α, χ .
To mimic innate human intuition about topological similarity, we introduce canvas distortion D V ( α ) for any canvas transformation α and color distortion D C ( M , M (cid:48) ) between two smooth images M , M (cid:48) .
Our idea is to search for a transformation that mimics what humans naturally do to transform one image into another.
That is, a low-distorted α which makes little difference in color between M and the transformed M (cid:48) , or more precisely, an α that minimizes both D V ( α ) and D C ( M , χ ◦M (cid:48) ◦ α ) .
Representating digital and smoothed images.
An m × n digital image is a discrete M : [ m ] × [ n ] → [0 , 1] , where [ k ] := { 0 , 1 , . . .
, k − 1 } .
We call [ m ] × [ n ] the canvas grid and any z ∈ [ m ] × [ n ] a grid point .
For any m × n digital image M , we smooth it to M via a sum of kernels :
[Equation]
where a kernel κ : R + → R + is a decaying function (e.g., linear, polynomial, Gaussian decay) and ρ is a metric on R 2 (e.g., (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ ).
In this paper, we use linear decay and (cid:96) ∞ , i.e., κ ( ρ ( z, x ))
= 1 − 1 ρ c (cid:107)
z − x (cid:107) ∞
if (cid:107) z − x (cid:107) ∞ <
ρ c (for some cutoff radius ρ c > 0 ) and κ ( ρ ( z, x ))
{+= 
 otherwise.+}
Note: M is deﬁned everywhere on R 2 .
This differs from Gaussian blurring as we do not discretize kernels.
It is key to use the smoothed image as input, which allows computing gradients analytically.
As such, we always smooth any digital image at ﬁrst and then only manipulate the smoothed image.
Representing arbitrary canvas transformations.
We consider all 2D transformations (including those without a formula), but how do we represent them in a computer?
With respect to the standard grid [ m ] × [ n ] , we use the transformed grid α ([ m ] × [ n ]) to represent α digitally.
Thus, any canvas transformation α is digitally represented by ( d = ) a matrix α ∈ R ( mn ) × 2 whose i th row is the 2D coordinate of the transformed i th grid point.
We use the lexicographical order of a 2D grid, e.g., with respect to [2] × [3] , the identify transformation id d = id
=
Any transformed image M◦ α d = M ( α ) :
= ( M ( α 0 ) , . . .
, M ( α ( mn − 1) ) )
∈ R ( mn ) , i.e., a (vectorized) digital image sampled from M at the transformed grid α .
Representing color and canvas distortions.
The color distortion D C measures the color discrepancy between M ( id ) and M (cid:48)
( α ) up to an afﬁne color transformation χ .
The canvas distortion D V measures the distortion between the original grid id and the transformed grid α .
Formally,
[Equation]
[Figure] ↵ Here, B E comprises all pairs of neighboring edges in a canvas lattice (introduced below).
Eq. (3) is derived from the mathematical deﬁnition of distortion of a function by discretizing it across the canvas lattice.
This formula measures how far an arbitrary transformation is from being conformal , which is ﬂexible for local isometries and scaling.
Given a canvas grid [ m ] × [ n ] , its corresponding canvas lattice is an undirected graph L =
( V, E ) , with the set of vertices V =
[ m ] × [ n ] and the set of edges obtained by connecting neighboring vertices in the (cid:96) ∞ sense: E = {{ i, j } | (cid:107) v
i − v j (cid:107) ∞
= 1 for v i , v j ∈ V } .
We say two edges are neighbors if they form a 45 ◦ angle (Figure 2).
Computing topological distance by minimizing distortions.
To minimize color and canvas distortions (2) and (3), we consider two dual views: minimizing D C among low-distorted α ’s or minimizing
D V among best-matching α ’s. We write the two views as the following two constrained optimization problems, together with their respective unconstrained equivalents: with (cid:15) → 0 + and µ → 0 + ,
[Equation]
We let the optima D (cid:63)C for (4) and D (cid:63)V for (5) denote two versions of our desired topological distance that mimics innate human intuition.
We call them D C -distance and D V -distance , respectively.
Transformation ﬂow.
To obtain both transformations and transformation processes that are humanlike, we run (projected) gradient descent.
The iterative gradient steps yield not only a transformation α (cid:63) in the end but also a transformation ﬂow id = α (0) → α (1) → · · · → α (cid:63) .
The resulting sequence of transformed images M (cid:48)
=
M (cid:48) ◦ α (0) →
M (cid:48) ◦ α
[Equation] 1) → · · · → M (cid:48) ◦ α (cid:63) ≈ M (we omit χ for simplicity) makes up an animation (Figure 1), which helps with human intuition on transforming [Equation] to M .
However, directly running (projected) gradient descent on (4) or (5) does not work, because it suffers from the curse of local minima, which we discuss and solve in the next section.
G RADIENT D ESCENT AT H IGHER L EVELS OF A BSTRACTION
The canvas distortion D V is invariant under a variety of transformations (e.g., D V ( α ) = 0 for any conformal α ), which nicely mimics humans’ ﬂexible transformation options. But this also implies lots of local/global minima and other critical points where the gradient is zero.
How much the color distortion D C ﬂuctuates as a function of α depends on the images M , M (cid:48) .
But in most cases, D C also has lots of local/global minima, the majority of which represent unwanted “short cuts”—unnatural transformations that make D C →
but would break the rubber canvas or create holes in it.
The curse of vanishing gradients can freeze gradient descent.
To unfreeze it, we lift gradient descent to higher levels, mimicking once again humans’ abstraction power, as our internal optimization system is quite ﬂexible in pursuing “gradient-descent” moves at multiple levels of abstraction.
We design two abstraction techniques: a chain of anchor lattices to make hierarchical abstractions of canvas transformations and a chain of color blurring to make hierarchical abstractions of image painting.
Anchor grids and lattices.
An anchor grid and its corresponding anchor lattice offer a simpler parameterization (i.e., an abstraction) of canvas transformations.
Without such an abstraction, any transformed [ m ] × [ n ] grid α ∈ R ( mn ) × 2 consists of 2 mn free parameters.
So, the optimization problems (4) and (5) are 2 mn +2 dimensional, which is not only computationally inefﬁcient for large images but also has too much room for vanishing gradient.
We use a simpler α -parameterization that regularizes transformation, lowers distortion, and agrees with our intuition on rubber transformations.
Formally, an anchor system ( G, ˆ G ) =
( M × N, ˆ M × ˆ N ) uses two layers of grids: an underlying grid G and an anchor grid ˆ G atop, satisfying ˆ M ⊆ M, ˆ N ⊆ N , and G ⊆
ConvexHull
( ˆ G ) .
Figure 3A shows [Figure] one example, where G = [5] × [6] = { 0 , . .
, 4 }×{ 0 , . .
, 5 } and ˆ G = { 0 , 2 , 4 }×{ 0 , 2 , 5 } .
Under an anchor system, we can uniquely represent any grid point g ∈ G via four anchors A g , B g , C g , D g ∈ ˆ G via proportional interpolation, or more precisely, the following double convex combination
[Equation]
Here, A g B g D g C g can be uniquely selected as the smallest rectangle in ˆ G ’s lattice containing g ; the two weight parameters λ g , ν g are computed based on relative position, e.g., as in Figure 3A.
The relation between grid points and anchors can be summarized by a weight matrix W ∈ R | G
|×| ˆ G | .
Its i th row stores weights for the i th grid point (say g in (6)) and contains at most four non-zero entries (i.e., coefﬁcients in (6)) located at the columns corresponding to A g , B g , C g , D g , respectively.
Given an anchor system ( G, ˆ G ) , any canvas transformation α d = α ∈ R | G |× 2 under G and d = ˆ α ∈ R | ˆ G |× 2 under ˆ G .
ˆ α is a submatrix of α , which induces an equivalence relation on the set of all canvas transformations: α , β are equivalent iff ˆ α = ˆ β , and ˆ α abstracts the equivalence class { β | ˆ β
= ˆ α } .
Based on the maximum entropy principle (Jaynes, 1957), a reasonable selection of a representative of this equivalence class is W ˆ α , because W ˆ α ∈ { β | ˆ β = ˆ α } and evenly distributes the transformed grid points.
Figure 3B illustrates this type of even distribution, which agrees with human intuition on how a rubber surface would naturally react when transforming forces are applied at anchors.
Using an anchor system in optimization problems (4) and (5) adds very little to computing distortions and gradients: we reuse the computation with α
= W ˆ α
and perform only one additional chain-rule step ∂ α /∂ ˆ α = W . By doing so, however, the number of optimization variables in (4) or (5) reduces from | G | +2 to | ˆ G | +2 (e.g., if G = [28] × [28] and ˆ G = { 0 , 27 }×{ 0 , 27 } , the number reduces from 1570 to 10 ).
It is important to note that using a simpler anchor grid is not the same as downsampling.
If it were, one would plug in α ← ˆ α , but we plug in α ← W ˆ α .
In our case, image colors are still sampled from the underlying grid rather than downsampled from the anchor grid.
So, using our anchor system is not information lossy while still beneﬁting from reduced optimization size.
Running gradient descent (w.r.t. anchors) in abstracted optimization spaces effectively bypasses critical points.
Blurring.
Another view to lifting gradient descent to a high-level, abstracted optimization space, is to blur the image.
Intuitively, blurring ignores low-level ﬂuctuation, similar to how humans naturally abstract an image.
Blurring helps remedy vanishing gradients and is done in our image smoothing process.
The cutoff radius ρ c in κ in (1) controls the blurring extent: larger ρ c means more blurred.
Guided gradient descent.
Mixing the two abstraction techniques yields our guided gradient descent proceeding from higher- to lower-level abstractions.
Given an anchor grid ˆ G and a cutoff radius ρ c , we denote the corresponding (4) and (5) by DC ( ˆ G, ρ c ) and DV ( ˆ G, ρ c ) , respectively.
For either, we solve for a ( ˆ G, ρ c ) -solution path, from coarser ˆ G and larger ρ c to ﬁner ˆ G and smaller ρ c .
Let ˆ G k be a k × k evenly distributed anchor grid and ˆ L k be its corresponding lattice.
Figure 3C shows a chain of anchor lattices { ˆ L 3 i +1 } i =0 , 1 , 2 ,... and cutoff radii { η j ρ c 0 } j =0 , 1 , 2 ,... .
It is easy initially to align two blurred blobs via small canvas adjustments, implying a small number of iterations to converge to D C ≈ D V ≈ 0 .
As we proceed along the solution path, the images restore more detail but the ﬁner ˆ L k helps manage that detail.
In a solution path, an earlier solution is used to warm start the subsequent solve step, which further alleviates the curse of vanishing gradients.
Notably, even the starting ˆ L 2 comprising only four corner anchors parameterizes a large family of transformations containing all afﬁne transformations.
Finer anchor grids/lattices express more ﬂexible transformations (including local, global, piecewise afﬁne, and more), approaching human-level ﬂexibility. [Figure] t acc [Figure]
r acy [Figure] [Figure] se t acc u r acy
[Figure] T es t se t acc u r acy MNIST [Figure]
+ da t a aug m en t t r an s f e r l ea r n i ng CNN + d r opou t C ap s u l e + S i a m e s e
We use our learned D C - / D V -distance in the simplest nearest-neighbor method to classify grayscale images, named D C - / D V -nearest-neighbor.
The whole process of metric learning and classiﬁcation is human intuitive and interpretable.
We show classiﬁcation performances on three standard benchmarks: the MNIST and EMNIST datasets of handwritten digits and letters restricted to the tiny-data regime, as well as the Omniglot challenge.
MNIST in the tiny-data regime.
The original benchmark has 60 k images for training and 10 k for testing, spanning 10 classes.
To evaluate how a model performs in the tiny data regime, we train the model on the ﬁrst N images per class from the original training set, test it on the full test set, and record test accuracy versus N = 1 , 2 , 3 , . . . .
We compare our model to both neural-network-based and classical ML models, including TextCaps (Jayasundara et al., 2019) with state-of-the-art performance in the small-data regime, SVM, nearest neighbor, etc.
Classical ML is included to show that nailing the tiny-data regime does not mean just using simple models.
For stochastic models, we record mean and standard deviation from 5 independent runs.
TextCaps only runs when N ≥ 4 and sometimes returns a random guess ( 10% ), so, we record trimmed mean and standard deviation from 11 runs (where we trim the best two and worst four).
We also copy results from other references that ran MNIST in a similar tiny-data setting, including FSL that uses extra data for pre-training (whereas all our other selected models do not).
These results are from the same training-testing sizes but not the same data sets, and hence, are considered indirect comparisons.
We present all results in Figure 4.
[Figure]
EMNIST-letters in the tiny-data regime.
The original benchmark has 4 . 8 k training images per class and 0 . 8 k test images per class, spanning 26 classes of case-insensitive English letters.
We keep the same experimental setting as in MNIST (except for TextCaps being more stable now: we do 7 independent runs for each N and trim the best and the worst).
Results are shown in Figure 5.
EMNIST-letters is harder, not only with more classes but also more intrinsic ambiguities, e.g., an l and an I can look identical, so can an h and an n when written carelessly.
Hence, all models perform signiﬁcantly worse than in MNIST.
The intrinsic ambiguity, as well as more labeling errors, narrows our superiority over other models as training size increases.
This is especially true for the state-ofthe-art TextCaps model, catching up quickly in Figure 5.
Being sensitive to ambiguities and outliers, however, is not a deﬁciency of our distortable canvas model, but a property of nearest-neighbor.
To improve, we may integrate our model with more robust classiﬁers, e.g., k -nearest-neighbor ( k -NN) with proper voting.
However, k -NN is not applicable in the tiny-data regime, not only because the training size can be as small as k but also there is little room to hold out a validation set for selecting k .
An adaptive k -NN may be desired, with k remaining 1 in the tiny-data regime and becoming tunable when training size increases to a level that affords a held-out validation set.
A related issue due to lacking validation data is about picking a proper model conﬁguration.
One may expect better results from any selected model in Figures 4 and 5 by attempting new conﬁgurations.
Yet, it is unclear what heuristics one may use.
For TextCaps, we used its original implementation and conﬁguration; for the rest, we used scikit-learn implementations with default conﬁgurations (except for small tweaks for the tiny-data regime e.g., neural-network size and stronger regularization).
By contrast, our distortable canvas model requires little to tune, other than the ( ˆ G, ρ c ) -solution path.
Theoretically, the more gradual the path, the better.
We picked ( ˆ G, ρ c ) based on only image size ( 28 here) and runtime.
The Omniglot challenge for one-shot classiﬁcation .
The Omniglot dataset contains handwritten characters from 50 different alphabets, which include historical, present, and artiﬁcial scripts (e.g.,
Hebrew, Korean, “Futurama”) and thus, are far more complex than MNIST digits and EMNIST letters.
The characters are stored as both images and stroke movements.
Unlike MNIST/EMNIST coming with large training data, the Omniglot challenge was specially designed for human-level concept learning from small data.
Its one-shot classiﬁcation task was benchmarked to evaluate how humans and machines can learn from a single example.
This benchmark contains 20 independent runs of 20 -way within-alphabet classiﬁcations.
The (2 k − 1) th and (2 k ) th runs for k = 1 , . . .
,use the same set of 20 characters from a single alphabet.
Each run uses 40 images: one training and one test image per character.
The unit task here is to predict for each test image, the character class to which it belongs (one of 20 ), based on the 20 training images.
In total, there are 400 independent unit tasks across all 20 runs.
Figure 6A shows a unit task (in
The Omniglot benchmark adopted the standard FSL setting, where it also provided a background set for pre-training.
The original background set contains 964 character classes from 30 alphabets; a reduced background set was proposed later to make the classiﬁcation task more challenging.
We run our D C -nearest-neighbor without any background set or any stroke-movement information.
In [Figure] other words, in each unit task, we predict the test image based on one and only that training image per character, and we read all images from their raw pixels.
Shown in Figure 6B, our model (with a 6 . 75% error rate) approaches human performance ( 4 . 5% ) and outperforms all models in the Omniglot leaderboard (Lake et al., 2019), except for BPL specially designed for the Omniglot challenge by making additional use of both the background set and the stroke-movement information.
U NSUPERVISED L EARNING : A RCHETYPE G ENERATION
Beyond use with a classiﬁer, our distortable canvas model can also perform k -means-style clustering, but within its human-intuitive topological similarity landscape.
As in other metric learning and non-Euclidean settings (Cuturi & Doucet, 2014), a naive way of explicitly computing distances and then running k -means is not realistic.
Learning distance in our model requires solving an optimization problem, which is not as cheap as computing Euclidean distance.
Further, computing a “centroid” in non-Euclidean space requires solving another optimization problem (i.e., minimizing the sum of within-cluster distances)—so
an optimization problem of optimization problems—which is not as simple as an arithmetic mean.
Our idea of transformation ﬂow between two images can be extended to multi-ﬂows among multiple images.
Under this multi-ﬂow extension, we do not explicitly compute pairwise distances, i.e., we do not solve the inner optimizations ﬁrst.
Instead, we solve the inner and outer optimizations at the same time, ﬂattening the nested optimizations into a single one.
Formally, given N images M 1 , . . .
, M N , to cluster them into K clusters, we solve minimize [Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] to [Equation] where C k denotes the k th cluster, M k ◦ α k denotes the k th centroid, and M i ◦ α
i denotes the i th transformed image ﬂowing to its corresponding centroid together with all other N − 1 transformed images.
One can check (7) is an extension of (4) where we omitted χ for simplicity.
Solving (7) is similar to k -means via alternating reﬁnement : the assignment step assigns each transformed image M i ◦ α i to C k (cid:63) according to k (cid:63) =
arg min k =1 ,...,K D C
( M k ◦ α k , M i ◦ α i ) ; the update step solves (7) for one gradient-descent step given the C k s. Upon convergence, we obtain C (cid:63) 1 , . . .
, C (cid:63)K as clusters and M 1 ◦ α 1 , . . .
, M K ◦ α K as centroids.
We treat the learned centroids as archetypes of the N images given at the beginning, which may be further deployed for education purposes.
We run the above procedure to generate archetypes of a particular image class. Like common practices used in k -means, we try different k .
For each k , we try multiple random starts and record the best within-cluster sum of distances (WCSD).
We use the elbow method to pick good values for k .
Figureshows the WCSD-versus- k curve obtained by running our clustering method on a set of 16 images of “7”s from MNIST.
The curve indicates k = 2 or 3 as a potential elbow point.
The resulting two clusters of “7” agree with human intuition regarding two general ways of writing “7”, depending on whether there is an extra stroke.
The resulting three clusters further divide the cluster of “simpler 7s” based on the angle of the transverse stroke.
Moving away from more strict symbol systems towards [Figure] free-form art, we try our model on doodle data where people were tasked with freely drawing abstract sketches of real-world objects.
Figure 7 also shows four ways of doodling a giraffe, learned from the ﬁrst 16 giraffes in Google’s Quick Draw dataset (Jongejan et al., 2016). The four learned archetypes cleanly separate outline sketches, pose orientations, as well as focused views of the neck.  C ONCLUSION , L IMITATION , AND F UTURE W ORK Focusing on its scientiﬁc contribution, this paper designs a human-intuitive model from ﬁrst principles to learn from few and only those few shots—in particular one and only one shot—requiring no extra data for pre-training.
Based on nativism, our introduced distortable canvas effectively models humans’ topological intuition and learns transformation-based visual similarity akin to how humans naturally “distort” objects for comparison.
This notion of similarity is formalized in our proposed optimization problem, which minimizes canvas and color distortions so as to transform one object to another with minimal distortion.
To remedy vanishing gradients and solve the optimization efﬁciently, we mimic human abstraction ability by chaining anchor lattices and image blurs into a solution path. This yields our gradient descent method capable of optimizing at multiple levels of abstraction. Our model outputs not only transformations but also transformation ﬂows that mimic human thought processes. We demonstrate initial empirical success in a ﬁrst set of benchmarks focused on abstract visual tasks such as character and doodle recognition.
By simply using 1 -NN, we achieved state-of-the-art results in the tiny-data and single-datum regime on MNIST/EMNIST and achieved near-human performance in the Omniglot challenge.
Our model also enables k -means-style clustering to generate humaninterpretable archetypes.
This paper is a ﬁrst step towards a general theory of a comprehensive, human-like framework for human-level performance in diverse applications.
The current paper focuses on an initial scope, but opens the pathway to future generalizations as detailed below.
Consider two general types of images: 1) images of abstract patterns, or abstract images , e.g., those of symbols and doodles; 2) photorealistic images of real-world objects, or real-world images , e.g., those in CIFAR10/100 (Krizhevsky & Hinton, 2009).
This paper focuses on the ﬁrst type, handling abstract images only, by modeling humans’ distortion-based intuition.
For real-world images, it may be more efﬁcient to ﬁrst model cognitive simpliﬁcation and then apply our current distortion model.
It is a reasonable assumption that humans have evolved to classify real-world images by ﬁrst converting them into abstract icons, or “e (picture)+moji (character)”
s (e.g., the emoji of a face, the outline of a mountain, the shape of a lake) and then comparing these simpliﬁcations.
Following this, an efﬁcient way to apply our method to real-world images is to follow this pipeline—preprocessing them ﬁrst into “emojis” and then comparing “emojis” using our distortion model.
In fact, abstract images (e.g., doodles of giraffes, hieroglyphics) are those that can be treated as “emojis” already.
There are baselines to attempt ﬁrst, e.g., smart edge detectors (Xie & Tu, 2015), but the human visual system does more than edge detection.
In the future, we will work on a complete theory of icon or “emoji” creation mimicking human capacity and deal with real-world images, 3D objects, and more.
Although our model has shown dominant classiﬁcation performance in the tiny-data regime of the presented benchmarks, its dominance diminishes when training size increases.
This is due to 1 -NN being 100% biased towards the “nearest neighbor” and hence fragile against noisy, erroneous, and ambiguous training examples. This suggests another future direction: our distortable canvas model may be designed jointly with a new, human-like classiﬁer that introduces a small amount of learning into classiﬁcation. The goal is to achieve state-of-the-art results on all training sizes, which is not merely about swapping in and out existing classiﬁers. We will not be using black-box models, but maintain model interpretability by modeling “the direct human way”, where we learn (from a small training set) a particular function to be integrated into our distortion formulas.
We will introduce such functions in future work and continue to improve them.
Figure 8 summarizes the pipeline for generalizing our current distortable canvas model into the future directions sketched above.