Neural Structure Mapping For Learning Abstract Visual Analogies
Abstract
Building conceptual abstractions from sensory information and then reasoning about them is central to human intelligence.
Abstract reasoning both relies on, and is facilitated by, our ability to make analogies about concepts from known domains to novel domains.
Structure Mapping Theory of human analogical reasoning posits that analogical mappings rely on (higher-order) relations and not on the sensory content of the domain.
This enables humans to reason systematically about novel domains, a problem with which machine learning (ML) models tend to struggle.
We introduce a two-stage neural framework, which we call Neural Structure Mapping (NSM), to learn visual analogies from Raven’s Progressive Matrices, an abstract visual reasoning test of ﬂuid intelligence.
Our framework uses (1) a multi-task visual relationship encoder to extract constituent concepts from raw visual input in the source domain, and (2) a neural module net-based analogy inference engine to reason compositionally about the inferred relation in the target domain.
Our
NSM approach (a) isolates the relational structure from the source domain with high accuracy, and (b) successfully utilizes this structure for analogical reasoning in the target domain.
Introduction
The ability to form abstractions of ‘concepts’ from information and then reason about them is central to human intelligence (Lake et al., 2015).
Over the last decade, deep learning (DL) models that learn visual abstractions from raw images show strong validation performance on curated test datasets for image recognition (Krizhevsky et al., 2012), object detection (Ren et al., 2015) and scene classiﬁcation (Zhou et al., 2017).
However, unlike humans, DL models struggle with isolating these abstractions and systematically applying them to out of distribution test scenarios (Greff et al., 2020).
One important way in which humans both build and reason about abstractions is through analogies (Mitchell, 2021).
Structure Mapping Theory (SMT) of human analogical reasoning posits that perceptual information can be broken down into a domain consisting of objects and attributes, and structural relations between the attributes in the domain (Gentner, 1983).
Consequently, SMT deﬁnes an analogy as a mapping between the structural relations across two domains, with no mapping of the attributes.
For example, to draw an analogy between the solar system and an atom, we can map the relational structure ( Planet revolves around Sun ùñ Electron revolves around Nucleus , Sun more massive than Planets ùñ Nucleus more massive than Electrons ).
However, the domain attributes, such as the Sun being yellow or hot are not mapped to the Nucleus in drawing an analogy.
This idea behind structure mapping was utilized by Hill et al.
(2019) as a prior in training ML models for learning analogies.
They ﬁrst constructed a dataset on learning abstract visual analogies from Raven’s
SVRHM
[Figure]
CNN CNN
CNN
LSTM
FC 128 LSTM
LSTM
FC FC
FC
Relation Classifier
Object Classifier
Attribute Classifier
OR
Type
Line
Q U E S
T
I O N
C A N D
I D A T
E S
Analogy Inference Engine
Answer: progressive matrices (RPMs) (Raven, 2000).
The RPM task is made up of two rows of ﬁve context panels, and four possible candidates to complete the analogy.
The learner must understand the higherorder relation constituted by the ﬁrst row of panels, and then choose the candidate satisfying the same relation with the the bottom row (shown inset in Figure 1).
Hill et al.
(2019) carefully curated the candidates presented during training to maximize differences in structural relations while being perceptually plausible.
It was found that incorporating structural differences to maximize the importance of structure mapping during learning enabled DL models to learn visual analogies more accurately.
However, it is not feasible to build or curate datasets to always exploit a prior in the dataset to maximize structure learning.
This would involve knowing combinatorially large possible relational structures beforehand and curating data to maximize learning these structures, which scales poorly with an increase in possible relational structures.
To address this issue, we propose a two-stage Neural Structure Mapping (NSM) framework to learn abstract visual analogies (Figure 1).
Our ﬁrst-stage involves a visual relationship encoder that takes as input the visual context from the source domain and predicts the component visual concepts: objects, attributes and relationship in a multi-task manner.
We then use the derived relation to dynamically build an analogy inference engine.
Similar to structure mapping in humans, our engine utilizes only the relation from the source domain to pick the candidate panel that best ﬁts the analogy.
We incorporate compositional reasoning into our approach by using a neural module network (Andreas et al., 2016; Johnson et al., 2017b) as our engine.
Our modular neural engine dynamically adapt its structure to align (Xu et al., 2020) with the relationship , thus explicitly incorporating the structure mapping process in our NSM model.
Problem Setup
Each abstract visual analogy task consists of ﬁve context panels P 1 ´ 5 con , and four candidate panels
P 1 ´ 4 can .
The context panels are arranged in a matrix with two rows of three columns each, with the ﬁnal panel in the second row missing.
The ﬁrst row of panels P 1 ´ 3 con express a semantically related triplet R “ t o, a, r u , that is composed of two lower-order perceptual visual concepts: object o and attribute a , and one higher-order semantic visual concept: relationship r .
The objective is to choose a candidate c P Ť 4 i “ 1 t P i can u in place of the missing panel which represents the same relationship as the ﬁrst row of panels.
The object ( o P { line , shape }) and attribute ( a P { quantity , colour , type , size , position }) together constitute the visual domain d of the triplet R .
Each possible attribute can take ten values v p a q P t 1 , 2 , ..., 9 , 10 u (normalized).
The domain of the ﬁrst row of panels P 1 ´ 3 con deﬁnes the source domain d source of the analogy, and the domain of the second row of panels t P 4 ´ 5 con , c u deﬁnes the target domain d target .
The dataset consists of seven unique possible domains such that d P {shape quantity, shape colour, shape type, shape size, shape position, line type, line colour} .
To test the systematic generalization in learning visual analogies, the dataset has ﬁve different generalization splits that require the ability to recognize the relationship across both novel domain d and attribute a values:
• Novel Domain Transfer: The training set consists of 42 ordered pairs of d source and d target while the remaining 7 (7*7 - 42) pairs are present only in the test set i.e. d trainsource Ś d traintarget X d testsource Ś d testtarget “ H .
• Novel Domain Type ( line type and shape color ) :
The training set does not contain any problems with the held-out domain
i
e. d train R {line type} or d train R {shape colour} , while each problem in the test set involves the held-out domain.
• Novel Domain Values (Interpolation and Extrapolation): The training and test sets are made of two mutually exclusive sets of attribute values.
For the extrapolation split, v p a q train P t 1 , 2 , 3 , 4 , 5 u while v p a q test P t 6 , 7 , 8 , 9 , 10 u .
For interpolation, v p a q train P t 2 , 4 , 6 , 8 , 10 u while v p a q test P t 1 , 3 , 5 , 7 , 9 u .
[Figure]
The relationship ( r P { progression, AND, OR, XOR }) deﬁnes the semantics across a row of panels and is the core abstraction of analogical reasoning.
It can be of classiﬁed into two types:
Unary and Binary.
The Unary relation, progression , is composed of a function applied to a single panel to produce the next panel in a row.
On the other hand, the Binary relations, ( AND, OR, XOR ), are composed of a function applied to the ﬁrst two panels in a row to generate the ﬁnal panel.
Hill et al.
(2019) hypothesized that introducing a prior on the learning process that requires the learner to correctly identify the relationship would align with how humans learn structure for analogical mapping.
They designed this prior by carefully controlling the candidate panels P 1 ´ 4 can presented during model training (Figure 2) to consist of candidates that are all semantically possible to complete the visual analogy (called Learning Analogies By Contrasting (LABC)).
Each candidate satisﬁes one possible relationship with the target domain panels but only the correct panel satisﬁes the relationship from the source domain panels.
It was found that LABC training signiﬁcantly improved the ability to learn analogies in DL models compared to training with merely perceptually possible candidates (Normal training).
In fact, even a mixed training regime consisting of both LABC and Normal problems showed signiﬁcant improvement on the Normal training regime.
Method
In our work, we take a complimentary approach to Hill et al.
for learning abstract visual analogies in RPMs.
Instead of depending on explicitly labeled candidates for mapping relational structure, we build a model that dynamically conﬁgures its structure based on the relational structure extracted from the source domain.
Internally, our approach is made up of two different neural networks that correspond to the two separate tasks in our pipeline.
We provide detailed discussion on each of these networks in our pipeline next (for implementation details of the networks please refer to Appendix D).
3.1 Visual Relationship Encoder
The Visual Relationship Encoder, R pred “ φ p P 1 ´ 3 con q , segregates the abstract relationship triplet in the source domain panels into its constituent object, attribute , and relationship (corresponding to the structure extraction phase in SMT).
The encoder φ is made up of a multi-task neural network.
We use a convolutional neural network (CNN)
C φ to ﬁrst extract a visual feature vector vis iφ “ C φ p P i con q for each context panel.
The extracted visual features are then sequentially fed into a long short term memory network (LSTM)
R φ to generate visual sequence features seq φ of the source domain.
The hidden layer vector after the third sequence panel is taken as the encoded features of the source domain seq φ “ R φ p vis 1 ´ 3 q .
The source domain feature vector seq φ is then passed to a fully connected layer l shared φ
[Equation]
p seq
[Equation]
q that is shared between all three visual components.
The shared linear feature vector l shared φ is then passed through three different fully connected layers l
[Equation]
l
[Equation]
q for each task t P {object, attribute, relation} classiﬁcation.
Finally, each l task φ is passed through a fully connected layer out task φ “ FC out task φ p l task φ q of sizes 2, 5, and 4 for the object, attribute, and relation prediction tasks respectively.
A Softmax over the outputs out task φ yields a probability distribution over the possible t o, a, r u values that constitute the triplet R .
Training The Visual Relationship Encoder φ is trained using the source domain context panels
P 1 ´ 3 con and the visual relationship labels R “ t o, a, r u .
The cross-entropy loss for each task is obtained using the multi-task network’s predictions and triplet labels.
The total encoder loss is calculated as a weighted sum of the individual losses for object, attribute, and relation prediction:
[Equation]
Empirically, we found that the values of α “ 0 . 5 , β “ 0 . 5 , γ “ 2 . 0 led to higher validation accuracy.
The model was trained for 100 epochs using the Adam optimizer with a learning rate of 1 e ´ 4 .
The training was performed on a local cluster with 10 Nvidia GTX2080Ti GPUs.
For each generalization split, the model was trained with contrasting, normal, as well as mixed candidates.
The model with the best validation accuracy was chosen for testing (the validation set consisted of a similar set of analogy candidates as the training set).
3.2 Analogy Inference Engine
Classifier
Unary
Unary Unary
Binary Binary
Stem Stem
Stem Stem
Stem Stem
Unary Unary
Unary
Binary
Classifier
[Figure]
The Analogy Inference Engine, c “ π p P 4 ´ 5 con , P
[Equation]
4 can , r q , maps the source relationship r extracted by the Visual
Relationship Encoder to the target domain P 4 ´ 5 con to identify the correct candidate c that completes the visual analogy.
This step of our NSM model corresponds to the structure mapping phase in SMT, in which humans apply relations extracted from the source domain to a new domain to reason about concepts in the target domain.
To generalize systematically to new domains and attribute values, we utilize a modular approach (Andreas et al., 2016) to build our analogy inference engine.
Similar to Johnson et al.
(2017b)
, we utilize modules with a generic architecture in order to enable computational reasoning and parameter reuse.
The layout of the module network is chosen adaptively from two possible layouts based on the source domain relation at inference time.
Each layout ﬁrst processes the individual panels through the Stem module to extract visual features, followed by the Unary module.
In the ﬁrst layout, the Unary module outputs for the target domain panels P 4 ´ 5 con are combined using a Binary module, and the outputs for the second panel P 5 con and each candidate panel P 1 ´ 4 can are combined using another Binary module (Figure 3a).
The outputs of both the Binary modules are then fed into the Classifier module.
In the second layout, the Unary module outputs for the target domain panels P 4 ´ 5 con are similarly combined using a Binary module.
However, the output of this Binary module and the Unary candidate output is directly fed into the Classifier module (Figure 3b).
(Further discussion on modules and layouts provided in Appendix B.)
Training The Analogy Inference Engine π is trained using the target domain context panels P 4 ´ 5 con , the candidate panels P 1 ´ 4 can , the ground truth candidate label, and the ground truth visual relationship label r .
The engine was trained for 100 epochs using the Adam optimizer with a learning rate of 1 e ´ 4 .
The training was performed on a local cluster with 10 Nvidia GTX2080Ti GPUs.
Similar to the ﬁrst stage, the engine was also trained with contrasting, normal, as well as mixed candidates.
The model with the highest validation accuracy was chosen for testing (validation set consisted of a similar set of analogy candidates as the training set).
Testing For testing, we rely on the relationship label r pred predicted by our encoder instead of ground truth r .
Each candidate panel is fed parallely to the engine to generate its predictions for all four relations.
In order to generate the candidate selection, we subset the probability of relationship r
( r pred during inference) for all candidates, and select the candidate with the highest corresponding probability.
Results
[Figure]
Visual Relationship Encoder Results We test the relationship prediction accuracy of our encoder across all ﬁve generalization splits, with each possible set of analogy candidates (Contrasting, Normal, Mixed).
Our relationship encoder was able to isolate the component relationship from the source domain with a high degree of accuracy (min: 82.4%, max: 86.04%) across all generalization splits and all three types of candidates (Figure 4).
Furthermore, since the encoder utilizes only the source domain information in determining the relation, we observe no drop between training and test accuracy across all generalization splits.
By identifying the visual relationship in the source domain
[Table]
the output of the visual relationship encoder is able to determine the layout for the subsequent analogy inference engine.
Since the inference engine chooses between only two possible layouts, a baseline that always chooses the majority class layout would lead to a correct choice in 75% of problems.
Hence, our encoder must achieve a high degree of accuracy in identifying the correct layout to beat this baseline.
Our encoder is indeed able to identify the layout matching the reasoning problem with near perfect accuracy (min: 98.42%, max: 99.9%).
The full results for layout prediction across all generalization splits and training regimes are in Table 15 in Appendix E
[Table]
Analogy Inference Engine Results
Does the modular analogy inference engine generalize systematically?
[Table]
for Novel Domain and Novel Attribute values.
Our approach achieves the best generalization accuracy across all models in the highest number of possible candidate scenarios (4/12), and is within ă 1% of the best generalization performance in half of all scenarios (6/12).
Higher accuracy =
=
Better systematic generalization.
We hypothesize that choosing a modular engine for analogy inference enables compositional reasoning about novel domain values (objects and attributes).
In Table 2 we compare the generalization performance of our engine on the Novel Attribute Value: Extrapolation and Novel Domain: Line
Type test splits of the analogy dataset.
Our approach achieves the highest test accuracy across all three training candidate regimes in the Novel Domain setting when tested with contrasting candidates.
Furthermore, our approach yields high test accuracy (highest or within ă 1% ) for the Contrasting and
Normal training candidate regimes in the Novel Attribute Value setting.
Thus, our NSM approach is better at systematic generalization to novel visual domains than monolithic neural networks.
For completeness, we provide the results for both our evaluation setups across all training and test regimes in Tables 16 and 17 in Appendix E.
Does our model’s prior enable it to learn structure mapping?
We speciﬁcally evaluate the Normal training scenario where the candidates presented to the learner are drawn randomly and lack the contrasting prior which explicitly promotes structure learning.
Since our NSM model explicitly captures and maps the structure of the visual analogy, thus making up for the lack of a prior in the candidates, we anticipate that the test performance of our model trained with normal candidates would outperform other baselines.
We visualize the generalization performance of this training scenario for the Novel Domain Value: Extrapolation and Novel Domain:
Line Type generalization splits in Figure 5.
We observe that our model outperforms other networks when averaged across both types of candidates during testing, demonstrating that our model’s prior enables it to learns structure mapping better than other models.
[Figure]
How important is structure mapping in drawing the correct analogy?
We verify the importance of explicitly mapping structure on our NSM approach by comparing our results when the relationship label inferred in the ﬁrst step matches the ground truth relationship label.
We present the confusion matrix in Table 2 where it can be seen that the test candidate selection accuracy for the correct structure mapping (diagonal entries) is signiﬁcantly higher than incorrect structure mapping.
[Table]
XOR OR
AND
[Table] [Table]
[Table] [Table]
88 XOR
[Table]
[Table] [Table]
OR
[Table] [Table]
[Table]
01 AND
[Table] [Table]
[Table]
Test Accuracy
[Table]
Conclusions and Future Work
In this work, we introduce a two-stage neural framework for learning abstract visual analogies based on the Structure Mapping Theory of human analogy making.
Our approach is able to generalize systematically to novel target domains, compensate for the lack of a prior in candidate selection that maximizes structure learning, and successfully exploit the process of extracting and mapping the relationship structure.
In future work, we plan to investigate search-based (e.g. Neural Architecture
Search) as well as differentiable (e.g. attention-based) methods for generating the layouts of our
Analogy Inference Engine.
Furthermore, we would like explore approaches that combine our structure mapping framework into general methods for reasoning about Raven’s Progressive Matrices.
[Table]
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] Our claims about introducing a neural structure mapping approach and its applications to learning abstract visual analogies are accurate.
(b) Did you describe the limitations of your work?
[No] Our work is limited by the availability of structural relationship labels.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is not directly impactful for societal applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] The authors have read and conform to the ethics review guidelines 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We have included our model’s implementation details in Appendix D, and our experimental setup in Section 3.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We explain the data splits in Section 2, and our hyperparameters in Section 4 and Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We did not have sufﬁcient time or GPU resources for multiple runs (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Lines 507 and4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite Hill et al.
in our Related Works (Appendix A) and at multiple points in the main paper.
(b) Did you mention the license of the assets?
[Yes] Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We do not release any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] Appendix A (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
The dataset we are using does not pertain to persons.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Related Work
Learning Analogies Gentner (1983)’s
SMT deﬁnes an analogy as a comparison in which relational predicates, but few or no object attributes, can be mapped from base to target domains.
Symbolic models based on SMT (Falkenhainer et al., 1986), including those designed for RPMs (Lovett &
Forbus, 2017), rely on ﬁrst extracting a rule based representation of the domains from the perceptual input.
Other cognitive analogy models based on Active Symbol theory (Hofstadter & Mitchell, 1994;
Mitchell, 1993) or High-Level Perception theory (Chalmers et al., 1992) do not separate the structure extraction and mapping process, nor rely on mapping the structure syntax across domains.
Our approach is not directly related to these models.
Hill et al.
(2019) introduced the dataset for learning analogies from RPMs and utilized several neural network architectures to directly learn from visual data.
Their key contribution was to introduce the SMT prior into the dataset during candidate selection.
Webb et al.
(2020) introduced Temporal Context Normalization to explicitly learn visual features that can support extrapolation, and tested it on the Visual Analogy Extrapolation Challenge (VAEC) dataset.
Both of these approaches are complementary to ours.
Other DL approaches most similar to ours are the Part-Composition Model (Ichien et al., 2021) and Chen et al.
(2019)’s approach.
However, unlike our model, these models rely on availability of very structured intermediate representations like semantic segmentation maps, and do not use the extracted structural representations to explicitly conﬁgure the reasoning model.
Abstract Reasoning Hill et al.
’s dataset (released publicly for commercial use under Apache
License 2.0) is directly derived from the Procedurally Generated Matrices (PGM) dataset introduced by Barrett et al.
(2018) to test the ability of neural networks to perform abstract reasoning on RPM problems.
RPM tests are an important measure of ﬂuid intelligence in humans (Raven, 2000) and the
PGM dataset provided a sufﬁciently large sample size for training neural networks on this problem.
Following this work, two new RPM datasets were also released: RAVEN (Zhang et al., 2019), which utilized additional rules and structured rule annotations, and V-PROM (Teney et al., 2020), which utilized real images.
DL approaches to RPM tasks can be roughly categorized into three types.
The ﬁrst is relation learning approaches, such as WReN (Barrett et al., 2018), which models all pairwise relationships between the matrix panels using relation networks (Santoro et al., 2017), and
MXGNet (Wang et al., 2020), which learns row-based node embeddings and then performs a graph classiﬁcation on the resulting candidate graphs.
Second is rule learning approaches, such as DRT (Zhang et al., 2019), that learns a structured Stochastic Image Grammar of the abstract rules, and
LEN (Zheng et al., 2019) that utilizes a logic embedding network along with a curriculum to learn with increasingly distracting features.
Third is object-centric learning, such as Rel-AIR (Spratley et al., 2020), that obtains object embeddings generated via Attend-Infer-Repeat for each panel, and
Pekar et al.
(2020)’s method that utilizes both Variational AutoEncoders and an adversarial loss for candidate generation.
Our model can be broadly ﬁt into rule learning by way of the encoder, followed by relation learning via the analogy inference engine.
However, our general approach towards explicitly extracting structure and mapping it is complimentary to these ideas and can be used in conjunction with several of them.
Systematic Generalization Systematic generalization refers to the ability to generalize to novel concepts (out of distribution data) by understanding them as compositions of known concepts (Bahdanau et al., 2018).
It is underpinned by three important characteristics: (1) systematicity, the ability to generalize to semantically related concepts; (2) productivity, the ability to iteratively apply compounding to generalize from constituent concepts to their recurrence; and (3) localism, the ability to iteratively apply reductionism to generalize from repeated constituent concepts to the singular.
The ability to generalize systematically has been previously explored in human cognition (Fodor &
Pylyshyn, 1988), natural language processing models (Lake & Baroni, 2018), and language-grounded computer vision models (Bahdanau et al., 2018).
It has its origins in human cognition and was formalized by Fodor (1975) under the ‘Language of Thought’ hypothesis.
Cognitive scientists have expressed reservations about the ability of connectionist models to generalize systematicitally (Fodor & Pylyshyn, 1988).
Investigation of neural language models showed that systematic generalization still poses a challenge for DL (Lake & Baroni, 2018;
Loula et al., 2018).
A key reason deep neural networks lack systematicity is due to their propensity towards ‘shortcut learning’
(Geirhos et al., 2020).
This suggests that DL models rely on exploiting a few number of predictive features instead of considering all possible information about the data while drawing conclusions.
Consequently, this leads to poor generalization when the key predictive features are changed, even though the central evidence remains the same.
For example, DL approaches to solving RPM problems were discovered to rely on evaluating the mode across all candidates in the RAVEN dataset and a signiﬁcant drop in generalization performance was observed when the models were tested without the shortcut on the
RAVEN-Fair dataset (Spratley et al., 2020).
On the other hand, neural module networks (Andreas et al., 2016; Johnson et al., 2017b) generalized well systematically when their layout aligned well with speciﬁc language-grounded reasoning problems (Bahdanau et al., 2018).
Hence, we utilize a modular approach in building our analogy inference engine, which in turn utilizes the relationship structure inferred by our encoder to automatically align itself to the problem.
This is a form of conditional computation.
B Analogy Inference Engine Details and Ablations
Our engine is made up of four different types of neural modules:
• The Stem module performs a series of convolution operations with stride=2.
It takes the original grayscale images of size 1 ˆ 160 ˆ 160 as input and returns a visual feature map of size 8 ˆ 9 ˆ 9 p C ˆ H ˆ W q .
Each of P 4 ´ 5 con and P 1 ´ 4 can panels is ﬁrst passed through the Stem module to extract visual features before being processed by the rest of the engine.
The Unary module is a residual block with two 3 ˆ 3 convolution layers.
It receives one feature map p C
ˆ H
ˆ W q as input, and returns one feature map of p C
ˆ H
ˆ W q as output.
• The Binary module receives two feature maps, concatenates them along the channel dimension, and uses a 1 ˆ 1 convolution to project them to C dimensions.
It then passes the combined feature maps through a residual block, and returns one feature map of p C
ˆ H
ˆ W q as output.
• The Classifier module also receives two feature maps, and projects them to C dimensions.
It then ﬂattens the feature maps and passes them through two fully connected layers to generate a probability distribution over the relations.
The generic architecture of our modules allows for a combinatorially large number of possible architectural layouts of our inference engine.
Previous work in language-grounded reasoning tasks like VQA (Andreas et al., 2016; Johnson et al., 2017b) uses a semantic parse tree generated from natural language to generate layouts for neural module networks.
Since our inference engine π p r q relies only on the relationship label from the source domain, it lacks the rich compositional structure available in language-grounded reasoning to arrange the modular network layout.
However, generating the proper layout for neural module networks is fundamental to their ability to generalize systematically (Bahdanau et al., 2018).
As discussed in Section 3.2, we dynamically chose between two layouts of our neural module network based on the relationship type.
Our rationale behind choosing these layouts comes from the knowledge of the possible relationships our engine has to reason about.
Each layout aligns algorithmically (Xu et al., 2020) with a reasoning algorithm for identifying the relationship across three panels.
The ﬁrst layout models Algorithm 1 to identify a Unary relationship like progression (not to be confused with a Unary module) between panels 1, 2 and 3.
The second layout models
Algorithm 2 to identify a Binary relationship ( OR, AND, XOR ) across panels 1 and 2 in the last panel.
Algorithm 1 To Identify Unary Relation
Require: Panels P 1 , P 2 , P1: D
Ð extractObjectsAndAttributes ( P
: D 2 Ð extractObjectsAndAttributes ( P 2 ) 3: D 3 Ð extractObjectsAndAttributes ( P 3 ) 4: R 12 Ð artihmeticOperation ( D 1 , D 2 )
R
Ð
artihmeticOperation ( D 2 , D 3 )
6: Result Ð isConsistent ( R 12 , R 23 ) 7: return Result
Algorithm 2 To Identify Binary Relation
Require: Panels P 1 , P 2 , P1: D 1 Ð extractObjectsAndAttributes ( P 1 ) 2: D 2 Ð extractObjectsAndAttributes ( P 2 ) 3:
D 3 Ð extractObjectsAndAttributes ( P 3 ) 4: R 12 Ð binaryOperation ( D 1 , D 2 ) 5: Result Ð isConsistent ( R 12 , D 3 ) 6: return Result
We do not explicitly train our modules to correspond to any individual functional form.
Instead, we expect the module instantiations to learn the corresponding function in the reasoning step simply by learning from the classiﬁcation loss.
For example, in the reasoning algorithms above, a Unary module can learn extractObjectsAndAttributes() , Binary module can learn artihmeticOperation() and binaryOperation() , and Classifier module can learn isConsistent() functions.
This enables the modules to share parameters across relation functions (such as two different instantiations of binaryOperation() for AND and OR ), as well as across objects and attributes.
This also enables us to utilize general purpose modules which could potentially be arranged in several possible combinations depending on the reasoning problem, and can be easily extended beyond the abstract visual analogy setting of only four relationships to large number of possible relationships in real-world datasets like Scene Graph (Johnson et al., 2015) and CLEVR (Johnson et al., 2017a).
Do algorithmically aligned engine layouts learn corresponding relations better?
We chose two possible layouts for our modular engine based on their alignment with algorithms for identifying a Unary or Binary relation across three panels in RPM problems.
We expect Layout-A (Figure 3a) to better align with Unary ( progression ) problems and Layout-B (Figure 3b) to better align with Binary ( AND, OR, XOR ) problems.
In order to verify this empirically we train each layout individually with similar training hyperparameters as the full engine.
We control the overall validation accuracy to select trained models which perform similarly across the full dataset.
Next, we compare the test accuracy for Unary and Binary relations for the trained Layout-A and Layout-B models in
Table 3.
[Table]
In practice, we found signiﬁcant stochasticity in the performance between Unary and Binary relation types across both layouts.
From the observed results, we cannot conclusively say that our layouts necessarily align better with one or the other relationship problem.
One possible explanation for this could be that there can be alternate algorithms for evaluating the same relation, which align algorithmically with these layouts.
This is similar to Xu et al.
(2020)’s observation that a neural network which would have not aligned with the Bellman-Ford algorithm was found to align well with a different algorithm for the same dynamic programming problem.
Does having an adaptive engine enable better analogy inference?
We use an adaptive modular engine for analogy inference which chooses between two possible module net layouts at inference time.
This enables our engine to compositionally reason about the underlying relation.
We expect our adaptive setup to outperform a static layout since the engine can then chose to use a better ﬁt layout according to the mapped relationship structure.
To verify our hypothesis, we compare our input-adaptive engine to a ﬁxed engine.
We chose each candidate layout described in Section 3.2 as a possible inference engine.
We call these ﬁxed models Engine-A and
Engine-B, and train them similar to our full model.
We then compare the test accuracy performance of our full model, which we label Engine-Full, against the test accuracy ﬁxed models.
[Table]
Empirically, we found that the adaptive engine was not able to outperform the ﬁxed engines, contrary to our expectation.
We believe there are two main reasons for this: (1) Our individual engines are not sufﬁciently differentiated in their performance on their respective reasoning tasks (as shown in Table 3), so there is not a signiﬁcant performance gain in choosing one over the other, and (2) the number of possible reasoning scenarios is quite limited in the abstract visual analogy problem, and hence the adaptive engine is not able to beneﬁt from learning a large number of mixture of experts.
C Visual Relationship Encoder Ablations C.1 Multi-task versus Multi-label Visual Relationship Encoder
In our two-step structure mapping approach, we treat the structure extraction problem as a multi-task learning problem where the relationship , object , attribute prediction are treated as separate learning tasks.
We could have alternatively treated the structure extraction problem as a multi-label classiﬁcation problem and instead predicted the full triplet R “ t o, a, r u from our model.
A multilabel problem could in principle learn from added network interactions between the ﬁnal classiﬁcation layer weights for the domain components.
In order to draw a comparison between both these learning approaches, we trained a multi-label visual relationship encoder which predicts the triplet R “ t o, a, r u as its output.
We replaced the l task φ “
FC task φ p l shared φ q layers in out multi-task encoder with a l shared 2 φ “ FC
[Equation]
q layer, and the ﬁnal classiﬁers out task φ
[Equation]
p
[Equation]
q of sizes 2, 5, and 4, with one classiﬁer out
[Equation]
FC
[Equation]
q of size 11 (2+5+4).
For training, we used exactly the same procedure as the multi-task encoder (described in Section ?? ).
We compared the performance of these models by evaluating there test accuracy across three different generalization splits: Novel Domain Transfer, Novel Domain: Line Type, and Novel Attribute Value:
Extrapolation.
Since our encoder is not trained with the candidate panels, we restrict ourselves to one training regime (Contrasting) for this comparison.
Our observations are reported below in Table 6.
[Table]
We found the performance of the two types of relationship encoders to be quite similar in terms of the test accuracy for predicting the relationship from the source domain panels.
In fact, the multi-label encoder was slightly better than the multi-task encoder across all generalization splits.
Despite this, we still went ahead with using the multi-task encoder in our two-step approach since the multi-task learning network scales much better with the number of objects, attributes
[Table]
and relationships in the perceptual domain.
C.2 Multi-task versus Relationship-Only Visual Relationship Encoder
We also performed an ablation study on predicting only the visual relationship from the encoder and comparing it with the full-task of predicting relationship , object , attribute .
We found that predicting only the relationship structure is competitive with a multi-task predictor in the Novel
Domain Transfer regime
[Table]
and since our encoder does not perform generalization across domains or candidate types this should hold for other regimes as well.
This does not take away from our central contribution of incorporating structure mapping into our model, rather it only highlights the importance of structure extraction as the ﬁrst-step of our model
[Table]
D Implementation Details D.1 Visual Relationship Encoder
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
| c |
[Equation]
[Table]
[Equation]
[Table]
[Equation]
E Supplementary Results E.1 Visual Relationship Encoder
[Table]
[Table]
E.2 Analogy Inference Engine
[Table]
[Table] [Table]
NovelDomainTransfer
[Table]