Functional Indirection Neural Estimator for Better Out-of-distribution Generalization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The capacity to achieve out-of-distribution (OOD) generalization is a hallmark ofhuman intelligence and yet remains out of reach for machines.
This remarkablecapability has been attributed to our abilities to make conceptual abstraction andanalogy, and to a mechanism known as indirection, which binds two representationsand uses one representation to refer to the other.
Inspired by these mechanisms,we hypothesize that OOD generalization may be achieved by performing analogymaking and indirection in the functional space instead of the data space as incurrent methods.
To realize this, we design FINE
(Functional Indirection Neural
Estimator), a neural framework that learns to compose functions that map datainput to output on-the-ﬂy.
FINE consists of a backbone network and a trainablesemantic memory of basis weight matrices.
Upon seeing a new input-outputdata pair, FINE dynamically constructs the backbone weights by mixing the basisweights.
The mixing coefﬁcients are indirectly computed through querying aseparate corresponding semantic memory using the data pair.
We demonstrateempirically that FINE can strongly improve out-of-distribution generalization on
IQ tasks that involve geometric transformations.
In particular, we train FINEand competing models on IQ tasks using images from the MNIST, Omniglotand CIFAR100 datasets and test on tasks with unseen image classes from one ordifferent datasets and unseen transformation rules.
FINE not only achieves the bestperformance on all tasks but also is able to adapt to small-scale data scenarios.
Introduction
Every computer science problem can be solved with a higher level of indirection.—Andrew Koenig, Butler Lampson, David J. Wheeler
Generalizing to new circumstances is a hallmark of intelligence [16, 4, 11].
In some Intelligence Quotient (IQ) tests–a popular benchmark for human intelligence–one must leverage their priorexperience to identify the hidden abstract rules out of a concrete example (e.g., a transformationof an image) and then apply the rules to the next (e.g., a new set of images of totally differentappearance).
These tasks necessitate several key capabilities, including conceptual abstraction andanalogy-making [22].
Abstraction allows us to extend a concept to novel situations.
It is also drivenby analogy-making, which maps the current situation to the previous experience stored in the memory.
Indeed, analogy-making has been argued to be one of the most important abilities of human cognition,or even further, “a concept is a package of analogies” [12].
The ability for humans to traverseseamlessly across concrete and abstraction levels suggests another mechanism known as indirectionto bind two representations and use one representation to refer to the other [16, 20].
x ...
... backbone layerbackbone layer t y f un c t i o n c o m p
o s i
t i o n i nd i r e c t i o n
W t ..
y t x +t =
W qt ..
M value t,
M value t, M value t,s
M keyt,
M keyt,
M key t,s a n a l o g y - m a k
i ng
MEMORY encoder encoder
The function composer geometric transformation deep learning models have successfully utilized analogy and indirection.
The Transformer[30] and RelationNet [28] learn analogies between data, through self-attention or pairwise functions.
The ESBN [33] goes further by incorporating the indirection mechanism to bind an entity to a symboland then reason on the symbols; and this has proved to be efﬁcient on tasks involving abstract rules,similar to those aforementioned IQ tests.
However, a common drawback of these approaches is thatthey operate on the data space, and thus are susceptible to out-of-distribution samples.
In this paper, we propose to perform analogy-making and indirection in functional spaces instead.
Weaim to learn to compose a functional mapping from a given input to an output on-the-ﬂy .
By doingso, we achieve two clear advantages.
First, since the class of possible mappings is often restricted,it may not require a large amount of training data to learn the distribution of functions.
Second,more importantly, since this approach performs indirection in functional spaces, it avoids bindingsbetween numerous entities and symbols in data spaces, thus may help improve the out-of-distributiongeneralization capability.
To this end, we introduce a new class of problems that requires functional analogy-making andindirection, which are deemed to be challenging for current data-oriented approaches.
The tasks aresimilar to popular IQ tasks in which the model is given hints about the hidden rules, then it has topredict the missing answer following the rules.
One reasonable approach is that models should beable to compare the current task to what they saw previously to identify the rules between appearingentities, and thus has to search on functional spaces instead of data spaces.
More concretely, weconstruct the IQ tasks by applying geometric transformations to images from MNIST dataset, handwritten Omniglot dataset and real-image CIFAR100 dataset, where the training set and test set containdisjoint image classes from the same or different datasets, and possibly disjoint transformation rules .
Second, we present a novel framework named Functional Indirection Neural Estimator (FINE) tosolve this class of problems (see Fig.
1 for the overall architecture of FINE ).
FINE consists of (a) aneural backbone to approximate the functions and (b) a trainable key-value memory module thatstores the basis of the network weights that spans the space of possible functions deﬁned by thebackbone.
The weight basis memories allow FINE to perform analogy-making and indirection in thefunction space.
More concretely, when a new IQ task arrives, FINE ﬁrst (1) takes the hint images tomake analogies with value memories, then (2) performs indirection to bind value memories with keymemories and ﬁnally (3) computes the approximated functions based on key memories.
Throughout
A
B
C
D
Rotation ?
A
B
C
D a comprehensive suite of experiments, we demonstrate that FINE outperforms the competing methodsand adapts more effectively in small data scenarios.
Tasks
For concreteness, we will focus on Intelligence Quotient (IQ) tests, which have been widely acceptedas one of the reliable benchmarks to measure human intelligence [26].
We will study a popular classof IQ tasks that provides hints following some hidden rules and requires the player to choose amonggiven choices to ﬁll in a placeholder so that the ﬁlled-in entity obeys the rules of the current task.
Inorder to succeed in these tasks, the player must be able to ﬁgure out the hidden rules and performanalogy-making to select the correct choice.
Moreover, once ﬁguring out the rules for the currenttask, a human player can almost always solve tasks with similar rules regardless of the appearingentities given in the tasks.
This remarkable ability of out-of-distribution generalization indicates thathumans treat objects and relations abstractly instead of relying on the raw sensory data.
We aim to solve IQ tasks that involve geometric transformations (e.g., see Fig.
2 for an example),which include afﬁne transformations (translation, rotation, shear, scale, and reﬂection), non-lineartransformations (ﬁsheye, horizontal wave) and syntactic transformations (black-white, swap).
Detailsof transformations are given in Supplementary.
In a task, the models are given 3 images x, y andx (cid:48) , where y is the result of x after applying a geometric transformation.
The models are thenasked to select y (cid:48) among 4 choices y 1 , y 2 , y 3 , y 4 so that ( x (cid:48) , y (cid:48) ) follows the same rule as ( x, y ) (i.e.if y = f
( x ) then y (cid:48)
=
f ( x (cid:48) ) for transformation f ).
The 4 choices include (i) one with correctobject/image and correct transformation (which is the solution), (ii) one with correct object/imageand incorrect transformation, (iii) one with incorrect object/image and correct transformation, and(iv) one with both incorrect object/image and transformation.
Inspired by human capability, a reasonable approach to solve these tasks is the models should be ableto ﬁgure out the transformation (or relation) between objects/images and apply the transformation tonovel objects/images.
The datasets can be classiﬁed into two main categories: single-transformationdatasets and multi-transformation datasets.
Single-transformation datasets are ones that only includea particular transformation, e.g. rotation.
Note that the individual transformations of the same typevary, e.g., rotations by different angles.
Multi-transformation datasets, on the other hand, consist ofseveral transformation types.
To test the generalization capability of the models, we build testing setsincluding classes of images that have never been seen during training (see Section 4.1 and Section 4.2),or even more challenging tasks including unseen rules and unseen datasets (see Section 4.3).
Modelsmust be able to leverage knowledge and memory gained from the training dataset to solve a new task.
Method
3. Functional Hypothesis
Let X
and Y be the data input and output spaces, respectively.
Denote by ( X train , Y train ) ⊂
( X , Y )the training set, and ( X test , Y test ) ⊂
( X , Y ) the non-overlapping test set.
Classical ML assumes that X train and X test are drawn from the same distribution.
Under this hypothesis, it is reasonable (forfrequentists) to ﬁnd a function f : X → Y in the functional space F that ﬁts both the train and the testsets (i.e., the discrepancy between f ( x ) and corresponding y is small for all ( x, y ) in ( X train , Y train )and ( X test , Y test ) ).
However, when X test is drawn from a different data distribution from that of X train ,it has been widely reported that current deep learning models fail drastically [2, 8, 13, 32, 33].
This is because the models are inferred exclusively from the observed data distribution.
Moreover, it couldbe the case that relations between x and y in testing samples are unseen during training, which raisesquestions on the feasibility of learning a single function when dealing with out-of-distribution tasks.
A natural solution for this problem would be to train the model to learn the functions adaptively.
Formally speaking, the model will learn a function composer φ : X × Y → F that maps each pairof
( x, y ) ∈ X × Y (where y is the associated output of x ) to a function φ x,y in F so that φ x,yapproximates the true relation between x and y .
As discussed, training models this way leads to twoclear advantages: (1) it can help to handle the cases when there are multiple (and possibly disjoint)relations between inputs and outputs within the training and testing datasets; and (2) models are lessdependent on data and thus can achieve more stable results on different training and testing sets.
Empirical evidence for these points will be given in Section 4.
Since there are an inﬁnite number of functions that can map an input to an output with any givendegree of precision, the key is to design φ so that it can map a given input-output pair to a “goodenough” function.
We humans may draw analogies between the current situation and our experiencesand then work out the most suitable options [12].
For example, we can recognize a math problem inexam to be similar to a previous exercise in class with different variable names.
We may ﬁnd thepresented idea in a paper related to that in another paper we read before, as someone even says “newideas are just re-distribution of old ideas”.
All these examples illustrate that analogy-making is apowerful strategy in human thinking process.
Inspired by this mechanism, we equip the functioncomposer φ with a semantic memory to store past knowledge, on which analogy-making is performed.
The memory also plays the role of constraining the searching region for φ , so that φ only looks forfunctions in the subspaces spanned by the memories instead of the whole functional space F .
Furtheranalysis is given in Supplementary.
A remaining question is how to design the memory.
Let us be inspired again by human thinkingprocess: When we see an animal, we compare its face, legs or tail with things we know and ﬁnallyconclude it’s a “dog”.
Here “dog” is an abstract concept bound with primary characteristics (e.g. face,legs, tail, etc.)
; once a new entity arrives, we compare its properties with these characteristics, and ifthey are similar, we utilize the indirection mechanism to infer it is indeed the concept we are considering.
In our case, the concepts are functions, or more speciﬁcally, the geometric transformations.
We thus maintain a key-value memory structure, in which the keys represent abstract concepts andvalues the associated characteristics of the concepts.
A new input-output pair matches with somevalues, and the indirection enables us to compute the functions based on corresponding keys.
Note that we have swapped the role of keys and values of the traditional memory (where the query ismatched against the key, not value).
This is to emphasize that we perform indirection to map concretefunctional values to abstract functional keys.
3. Functional Indirection Neural Estimator
In this section, we present the Functional Indirection Neural Estimator (FINE), an neural architectureto realize the general idea laid out in Section 3.1.
FINE learns to compose a function mapping aninput embedding x to an output embedding y on-the-ﬂy.
The function is drawn from a parametricfamily speciﬁed by a backbone neural network.
Coupled with the backbone is a function composerφ , which is trained to compute the parameters of the backbone.
More speciﬁcally, φ maps a datainput-output pair to the weights of the neural networks.
FINE solves the proposed IQ tasks as follows:
(1) Given two images x, y from the hint, FINE uses φto produce the function transforming x to y ; denoted by φ x,y .
(2) Then, we feed the third image x (cid:48)to φ x,y and get output y ∗ =
φ x,y ( x (cid:48) ) .
(3) We deﬁne a similarity metric to compare y ∗ with givenchoices.
The choice with the closest distance from y ∗ is model’s answer.
The function composer andthe similarity metric are speciﬁed as follows.
Encoder : Images are encoded by a trainable encoder (see Fig. 1).
To effectively solve IQ tasksintroduced in Section 2, we use the p 4 -CNN [7] encoder to serve as an inductive bias for geometrictransformations.
Throughout this paper, we refer to the images by their embeddings.
The functional memory
Let us focus a weight matrix W t ∈ R d in t × d out t at the t -th layer of the backbone network.
In practice,
W t belongs to the huge d in t ×
d out t -dimensional real matrix space M d in t ,d out t .
To reduce the complexityof W t , we assume that W t only belongs to a s -dimensional subspace of M d in t ,d out t , where s (cid:28) d in t d out t .
This subspace has a basis of s matrices which are trainable and stored in FINE ’s memory, and W twill be written as a linear combination of these matrices.
Denote by M t the memory for the t -th backbone layer.
M t includes two sub-memories: the key memory M key t = { M key t, 1 , . . .
, M key t,s } and the corresponding value memory M value t = { M value t, 1 , . . .
, M value t,s } .
Elements of M key t and M value t are trainable matrices of size d in t × d out t .
We further let x t ∈ R d in t andy t ∈ R d out t be the associated input and output, respectively, where x t is output of the ( t − 1) -th layerand y t is the pseudo-output computed by a trainable 1-layer neural network y t = γ t ( y ) .
With this design, we control the complexity of functional hypothesis space by either constraining theform of the backbone or the capacity of the functional memory.
Memory reading
By the virtue of simplicity, we aim to ﬁnd a simple query that can demonstrate the relation betweenx t and y t .
Although the exact relation may be non-linear, we found that a query induced from linearrelation is enough to efﬁciently read from memory.
Formally, we want to ﬁnd a query W qt such that
W qt x t = y t .
The best-approximated solution is W qt =
y t x + t , where x + t is the pseudo-inverse of x tand can be efﬁciently approximated by the iterative Ben-Israel and Cohen algorithm [3].
This way ofquery computing requires no parameter as opposed to other methods, where the input is often fedinto a trainable neural network to compute the query.
With the query in hand, the next step is to perform analogy-making.
In FINE , the query W qt representsfor the current situation and the value memory M value t consists of past experiences.
The concretequery interacts with value memories to measure how close the current situation is to each of theexperiences.
The similarities are computed as dot products between the query and value memoriesand normalized by a factor of (cid:112) d in t d out t :
[Equation]
where the flatten operator ﬂattens the matrix W qt of size d in t × d out t into a vector of size d in t d out t ,while the fconcat operator ﬁrst ﬂattens all matrices in M value t , then concatenates them together toform the value matrix of size ( d in t d out t ) × s .
The resulting a t is a s -dimensional vector measuringthe similarities between the query and the entries in the value memory.
Here we omit the softmaxoperator as in usual attention to allow the similarities with more freedom.
The same idea is shared inthe ESBN [33], where the softmax similarities are scaled by a sigmoidal factor.
Finally, the value memories are bound with their associated key memories via indirection.
This canbe understood as moving forward from the concrete space of data and value memories to the abstractspace of functions and key memories.
With the key memories and the similarity vector a t , the weight
W t of current backbone layer can be computed as the linear combination of key memories:
[Equation]
where the reshape operator reshapes the vector of size d in t d out t to a matrix of size d in t × d out t .
Since thesoftmax is omitted when calculating the similarities, W t is not constrained to be in the convex hull ofthe key memories and indeed can lie anywhere in the subspace spanned by those key memories.
Memory update
The key and value memories are updated using gradient descent:
[Equation]
where λ > 0 is the learning rate and L is the loss of the training step.
The similarity metric
After determining φ x,y , the model is given a new input x (cid:48) and being asked to select the correctassociated output y (cid:48) among 4 choices y (cid:48) 1 , y (cid:48) 2 , y (cid:48) 3 , y (cid:48)
so that ( x (cid:48) , y (cid:48) ) follows the same transformationrule as ( x, y ) .
This problem can be cast as ﬁnding the choice that is the most similar with y ∗ =
φ x,y ( x (cid:48) ) .
We consider the weighted Euclidean metric that measures the distance between two vectorsu, v ∈ R d : η ( u, v ) =
d (cid:88) i
= α i ( u i − v i ) 2 , where { α i } di =1 ≥ 0 are trainable scalars, i.e., each component of u and v contributes with differentimportance.
Finally, the probability to pick a choice is computed as:p ( y (cid:48) i | x (cid:48) ) = exp( − η ( y (cid:48)
i , y ∗ )) 4 (cid:88)
j = exp( − η ( y (cid:48)
j , y ∗ )) , for i
= 1 , 2 , 3 , 4 .
Experiments
We conduct experiments to show the out-of-distribution generalization capability of FINE whenperforming tasks introduced in Section 2.
For non-invertible transformations (e.g. reﬂection), we usea simple 2-layer MLP as the backbone.
For invertible transformations, we use the NICE architecture[9] as backbone to serve as an inductive bias for invertibility.
Since in each NICE layer, only half ofthe input is transformed, we use the same memory for two consecutive layers, i.e., M 2 t =
M 2 t +1 .
To balance between the backbone complexity and computational cost, we use 4 NICE layers in allexperiments.
We compare FINE with three major classes of models: (a) models that make analogies in the dataspace, including Transformer [30], PrediNet [29] and RelationNet [28]; (b) models that leverageindirection to bind feature vectors with associated symbols and reason on the symbols, including the
ESBN [33]; and (c) models that aim to learn a mapping from data to the functional space, includingthe HyperNetworks [14].
For HyperNetworks, we still use the NICE backbone and just apply theirfast-weight generation method for fair comparisons.
Except for FINE and HyperNetworks, all modelsare trained with context normalization [32], which has been proved to be effective in improving thegeneralization ability.
Datasets & implementation : We generate data for IQ tasks described in Section 2 using images from
Omniglot dataset [5], which includes 1,623 handwritten characters, and real-image CIFAR100 dataset[17].
If not speciﬁed, models are trained with p 4 -CNN encoder [7].
Experiments are conducted using
PyTorch on a single GPU with Adam optimizer.
Reported results are averages of 10 runs.
4. Results on Omniglot Dataset
[Table]
We use 100 characters for training and other 800 characters for testing.
The train and test set size is10,000 and 20,000, respectively.
For FINE , we use 4 NICE layers with 48 memories for each pairof NICE layers.
Experimental results for single-transformation tasks are shown in Table 1.
Overall,
FINE dominates others with large margins.
For example, the gap to the runner-up on the Rotationtask is nearly 13%.
With test accuracy over 75% on all tasks, FINE shows a strong capability ofout-of-distribution generalization.
We further conduct multi-afﬁne-transformation experiments.
In this case, the training set includesmultiple types of afﬁne transformations, while other settings are similar to the single-transformationcase.
Results are also reported in Table 1.
FINE continues to outperform other models.
This isbecause only FINE explicitly assumes the existence of multiple good functions that represent thetransformation from input to output data.
We note that although HyperNet also makes a similar assumption by generating data-speciﬁc weights, it does not utilize analogy-making and indirectionand thus, fails to generalize to unseen images.
[Table] [Table]
For CIFAR100 dataset, we follow similar settings as in experiments on the Omniglot dataset, exceptthat we use 50 classes for training and 50 remaining classes for testing.
We also conduct experimentson single-transformation and multi-afﬁne-transformation tasks.
Results for single-transformationtasks are reported in Table 2.
Again, FINE outperforms all other models on all tasks, especially on
Reﬂection where the gap is nearly 30%.
Although FINE does not show good performance on Swaptask as in Omniglot experiments, it is still slightly better than other models.
On the remaining tasks,
FINE achieves test accuracy of more than 80%.
For the multi-afﬁne-transformation task, we report the performances of the models when trainedwith different encoder architectures, including the p 4 -CNN, 3-layer ResNet and 2-layer MLP, in
Table 3.
The results show two superior characteristics of FINE : ﬁrst, FINE is consistently better thanother models across different encoders; second, FINE is more stable with small standard deviations.
This empirical result supports the functional hypothesis stated in Section 3.1, where we suggestthat focusing on functions distribution instead of data distribution can boost model’s generalizationcapability and stability.
4. More Extreme OOD Tasks
[Table]
Previous tasks only include unseen classes of objects during testing. In this section, we further test FINE and related models on more challenging OOD tasks: tasks with unseen rules during trainingand even ones with images from unseen datasets.
The training and testing sets are either imagesfrom CIFAR100, Omniglot or MNIST datasets, while the hidden rules are either translation, rotation or shear.
For translation, training problems consist of translation vectors ( a, b ) with | a | , | b | ≤ 3 ,and models are tested with either | a | > 3 or | b | > 3 ; for rotation, model are trained with anglesα ≤ 180 ◦ and tested with α > 180 ◦ ; for shear, training angles ( α, β ) are ones with | α | , | β | ≤ 30 ◦ ,while testing ones are either | α | > 30 ◦ or | β | > 30 ◦ .
All tasks have 5,000 data points for trainingand 10,000 for testing. We report results of FINE with NICE backbone and related models in Table 4.
As expected, FINE continues to outperform other models on most of the tasks, even on extreme OODtasks with unseen datasets and unseen rules where performances of all models drop signiﬁcantly. This demonstrates FINE is capable of effectively learning the basis weights, which are stored in thememory, to represent novel rules.
4.4 Model Analysis and Ablation Study
Clustering on functional spaces
[Figure]
We study how the transformations are distributed in the functional space.
We use the FINE modeltrained on multi-afﬁne transformation tasks.
For an input-output pair ( x, y ) , we ﬂatten and concatenateall weights of NICE layers to form a vector representing φ x,y .
We then use the UMAP [21] to projectφ x,y ’s vectors onto the 2D plane.
Results are shown in Fig.
It is interesting to see that shears withthe same horizontal or vertical angles are positioned closely; and scales are separated into 2 “big”clusters, one for smaller scale and one for bigger scale.
In contrast, reﬂection representations seemnot to be clustered properly, which is worth further investigation in future work.
Number of training data points, memories, and backbone layers
The performances of FINE can be improved if the number of memories and backbone layer areincreased.
Furthermore, when trained with small number of data points, FINE continues to outperformcompeting models, which shows a strong superiority of FINE on OOD generalization tasks.
Pleaserefer to the Supplementary for more details.
Related work
In recent years, there has been a strong interest in designing models that are capable of generalizingsystematically.
The Module Networks [1], which dynamically compose neural networks out of trainable modules, have been shown to possess some degree of systematic generalization [2].
Parascandoloet al. [24] proposed a method to train multiple competing experts to explain image transformationson MNIST and Omniglot datasets, yet the transformations are simpler than ones in our FINE datasetand it is not clear whether the proposed method can deal with unseen transformations. The Neural Interpreter [25] uses attention mechanism to recompose functional modules for each input-output pairand test their method on abstract reasoning tasks, yet tends to require a large amount of data to learn.
Switch Transformer [10] mitigates the communication and computational cost in mixture of expertsmodels.
Recently, ESBN [33], which utilizes the indirection mechanism, shows a great promise on OOD tasks.
Both methods achieve their degree of systematic generalization by injecting symbolicbiases into the models.
Our model FINE follows the same strategy, but it performs analogy-makingand indirection in functional spaces instead of data spaces, and this has proved to boost both theperformance and stability.
IQ tests are powerful testbeds for visual and abstract reasoning.
Inspired by Raven’s Progressive
Matrices (RPM), the RAVEN dataset [34] was proposed as a testbed for visual reasoning models.
However, this dataset does not focus on testing the ability of OOD generalization.
Webb et al.
[33]propose a series of IQ tasks with Unicode characters to show the effectiveness of indirection in tasksinvolving abstract rules, however these tasks are relatively simple since they only require the modelsto understand the same-different relation.
The ARC dataset [6] aims to serve as a benchmark forgeneral intelligence and includes various psychometric IQ tasks in the form of grid structures.
In thispaper, we propose IQ tasks involving geometric transformations as introduced in Section 2.
Thesetasks are not only ﬂexible so that we can include images from different datasets or create/combinenumerous transformations, but also challenging to test OOD generalization abilities of models.
The weight composition feature of FINE links back to the concept of fast weights [19], the ideaof computing data-speciﬁc network weights on-the-ﬂy.
HyperNetworks [14] stylizes this idea bycomputing the fast weights using a separate trainable (slow weight) network.
The Meta-learned Neural
Memory [23] uses the pseudo-target technique (which we also leverage in FINE ) and appropriatelyupdates the short-term memory once new input arrives.
The Neural Stored-Program Memory (NSM)[18] proposes a hybrid approach between slow-weight and fast-weight to compute network weightson-the-ﬂy based on slow-weight key and value memories.
However, NSM only performs on sequentiallearning tasks, while FINE aims to solve OOD IQ tasks requiring abstract cognition.
Moreover, FINEcomputes the query based on the input and the (pseudo-) output, while NSM’s query is computedbased on the input only.
Memories have been to be versatile in meta-learning and few-shot learning[15, 27, 31] due to the ability to rapidly store past examples and adapt to new situations.
In our case,an IQ task can be thought of as a one-shot learning scenario in which FINE has to make use of thelong-term key-value memory to adapt to the current task.
Limitations
Operating in functional spaces requires FINE ’s memory to store several trainable weight matricesto infer the data-speciﬁc weights for the backbone.
Moreover, each layer of the backbone requiresits own memory, thus FINE may need a large number of parameters for very deep backbones.
Thiscould be addressed by parameter-sharing across layers and limiting the rank of the weight matrices.
It remains to design the backbone architecture optimally.
We have used the NICE architecture asbackbone for invertible transformations and a 2-layer MLP backbone for non-invertible cases.
Wefurther investigated our model’s performances with larger number of training points (up to 1M) andobserved that FINE with the NICE backbone peaks at 100,000 training points with around 85%test accuracy, while FINE with MLP backbone continues to improve and achieves around 95% testaccuracy at 1M training points.
This calls for further theoretical analysis to guide the architecturaldesign of the backbone network.
Finally, testing FINE on IQ tasks in the visual space may limit its potential on IQ tasks involving othermodalities.
For example, one may consider a textual IQ problem: “if abc → abd, then mmnnpp → ? ”.
It is worth to emphasize that the concepts of analogy-making and indirection in functional spaces areindeed general and thus the idea of FINE should be applicable to various scenarios.
Conclusion
To study the out-of-distribution (OOD) generalization capability of models, we have proposed IQtasks that require models to rapidly recognize the hidden rules of geometric transformations betweena pair of images and transfer the rules to a new pair of different image classes.
Such tasks wouldnecessitate human-like abilities for conceptual abstraction, analogy-making, and utilizing indirection.
We put forward a hypothesis that these mechanisms should be performed in the functional spaceinstead of data space as in current deep learning models.
To realize the hypothesis, we then proposed
FINE , a memory-augmented neural architecture that learns to compose functions mapping an inputto an output on-the-ﬂy.
The memory has two trainable components: the value sub-memory and thebinding key sub-memory, where the keys are basis weight matrices that span the space of functions.
For an IQ task, when given a hint in the form of an input-output pair, and FINE estimates the analogybetween the pair and the values as mixing coefﬁcients.
These coefﬁcients are then used to mix thebinding keys via indirection to generate the weights of the backbone neural net which computes the intended function.
For a test input of different class, the function is used to estimate the mostcompatible output, thus solving the IO task.
Through an extensive suite of experiments using imagesfrom the Omniglot and CIFAR100 datasets to construct the IQ tasks, FINE is found to be reliable inﬁguring out the hidden relational pattern in each IQ task and thus is able to solve new tasks, evenwith unseen image classes.
Importantly, FINE outperforms other models in all experiments, and cangeneralize well in small data regimes.
Future works will include making FINE robust against OOD in transformations without catastrophicforgetting when new transformations are continually introduced
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] Our main contribution is to perform analogy-making and indirection on functional spaces for better OOD generalization.
(b) Did you describe the limitations of your work?
[Yes] see Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] see Supplemen- tary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] see Section 4(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] see Section 4 and Appendix C(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] see Section
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use codes of [33] to run our experiments for the RelationNet, PrediNet, Transformer and ESBN.
(b) Did you mention the license of the assets?
[Yes] see Supplementary
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include codes for our tasks and models.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] see Supplementary
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] see Supplementary
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]