Original: N EURAL S TRUCTURE M APPING F OR L EARNING A BSTRACT V ISUAL A NALOGIES
Revised: N EURAL S TRUCTURE M APPING F OR L EARNING A BSTRACT V ISUAL A NALOGIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Building conceptual abstractions from sensory information and then reasoning about them is central to human intelligence.
Revised: Building conceptual abstractions from sensory information and then reasoning about them is central to human intelligence.

Original: Abstract reasoning both relies on, and is facilitated by, our ability to make analogies about concepts from known domains to novel domains.
Revised: Abstract reasoning both relies on, and is facilitated by, our ability to make analogies about concepts from known domains to novel domains.

Original: Structure Mapping Theory of human analogical reasoning posits that analogical mappings rely on (higher-order) relations and not on the sensory content of the domain.
Revised: Structure Mapping Theory of human analogical reasoning posits that analogical mappings rely on (higher-order) relations and not on the sensory content of the domain.

Original: This enables humans to reason systematically about novel domains, a problem with which machine learning (ML) models tend to struggle.
Revised: This enables humans to reason systematically about novel domains, a problem with which machine learning (ML) models tend to struggle.

Original: We introduce a two-stage neural framework, which we call Neural
Revised: We introduce a two-stage neural framework, which we call Neural

Original: Structure Mapping (NSM), to learn visual analogies from Raven’s Progressive
Revised: Structure Mapping (NSM), to learn visual analogies from Raven’s Progressive

Original: Matrices, an abstract visual reasoning test of ﬂuid intelligence.
Revised: Matrices, an abstract visual reasoning test of ﬂuid intelligence.

Original: Our framework uses (1) a multi-task visual relationship encoder to extract constituent concepts from raw visual input in the source domain and (2) a neural module net-based analogy inference engine to reason compositionally about the inferred relation in the target domain.
Revised: Our framework uses (1) a multi-task visual relationship encoder to extract constituent concepts from raw visual input in the source domain and (2) a neural module net-based analogy inference engine to reason compositionally about the inferred relation in the target domain.

Original: Our NSM approach (a) isolates the relational structure from the source domain with high accuracy, and (b) successfully utilizes this structure for analogical reasoning in the target domain.
Revised: Our NSM approach (a) isolates the relational structure from the source domain with high accuracy, and (b) successfully utilizes this structure for analogical reasoning in the target domain.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to form abstractions of ‘concepts’ from information and then reason about them is central to human intelligence (Lake et al., 2015).
Revised: The ability to form abstractions of ‘concepts’ from information and then reason about them is central to human intelligence (Lake et al., 2015).

Original: Abstractions enable humans to quickly learn concepts from few examples, and then reason systematically about new concepts by composing previously understood concepts (Nam & McClelland, 2021).
Revised: Abstractions enable humans to quickly learn concepts from few examples, and then reason systematically about new concepts by composing previously understood concepts (Nam & McClelland, 2021).

Original: Over the last decade, artiﬁcial neural networks have demonstrated human-level performance in building useful abstractions from data when tested on well-deﬁned and constrained tasks.
Revised: Over the last decade, artiﬁcial neural networks have demonstrated human-level performance in building useful abstractions from data when tested on well-deﬁned and constrained tasks.

Original: In computer vision, deep learning (DL) models that learn visual abstractions from raw images show strong validation performance on curated test datasets for image recognition (Krizhevsky et al., 2012), object detection (Ren et al., 2015), and scene classiﬁcation (Zhou et al., 2017).
Revised: In computer vision, deep learning (DL) models that learn visual abstractions from raw images show strong validation performance on curated test datasets for image recognition (Krizhevsky et al., 2012), object detection (Ren et al., 2015), and scene classiﬁcation (Zhou et al., 2017).

Original: However, unlike humans, DL models struggle with isolating these abstractions and systematically applying them to out of distribution test scenarios (Greff et al., 2020).
Revised: However, unlike humans, DL models struggle with isolating these abstractions and systematically applying them to out of distribution test scenarios (Greff et al., 2020).

Original: [Figure]
Revised: [Figure]

Original: One important way in which humans both build and reason about abstractions is through analogies (Mitchell, 2021).
Revised: One important way in which humans both build and reason about abstractions is through analogies (Mitchell, 2021).

Original: There are several theories in human cognitive science about how humans perform analogical reasoning.
Revised: There are several theories in human cognitive science about how humans perform analogical reasoning.

Original: Structure Mapping Theory (SMT) posits that perceptual information can be broken down into a domain consisting of objects and attributes, and structural relations between the attributes in the domain (Gentner, 1983).
Revised: Structure Mapping Theory (SMT) posits that perceptual information can be broken down into a domain consisting of objects and attributes, and structural relations between the attributes in the domain (Gentner, 1983).

Original: Consequently, SMT deﬁnes an analogy as a mapping between the structural relations across two domains, with no mapping of the attributes.
Revised: Consequently, SMT deﬁnes an analogy as a mapping between the structural relations across two domains, with no mapping of the attributes.

Original: For example, to draw an analogy between the solar system and an atom, we can map the relational structure ( Planet revolves around Sun ùñ Electron revolves around Nucleus , Sun more massive than Planets ùñ Nucleus more massive than Electrons ).
Revised: For example, to draw an analogy between the solar system and an atom, we can map the relational structure ( Planet revolves around Sun ùñ Electron revolves around Nucleus , Sun more massive than Planets ùñ Nucleus more massive than Electrons ).

Original: However, the domain attributes, such as the Sun being yellow or hot are not mapped to the Nucleus in drawing an analogy.
Revised: However, the domain attributes, such as the Sun being yellow or hot are not mapped to the Nucleus in drawing an analogy.

Original: [Figure]
Revised: [Figure]

Original: Type
Revised: Type

Original: Line
Revised: Line

Original: Q U E
Revised: Q U E

Original: S T
Revised: S T

Original: I
Revised: I

Original: O N
Revised: O N

Original: C A N
Revised: C A N

Original: D I D
Revised: D I D

Original: A T E
Revised: A T E

Original: S
Revised: S

Original: Analogy Inference Engine
Revised: Analogy Inference Engine

Original: Answer:
Revised: Answer:

Original: This idea behind structure mapping was utilized by Hill et al.
Revised: This idea behind structure mapping was utilized by Hill et al.

Original: (2019) as a prior in training ML models for learning analogies.
Revised: (2019) as a prior in training ML models for learning analogies.

Original: They ﬁrst constructed a dataset on learning abstract visual analogies from Raven’s progressive matrices (RPMs) (Raven, 2000).
Revised: They ﬁrst constructed a dataset on learning abstract visual analogies from Raven’s progressive matrices (RPMs) (Raven, 2000).

Original: The abstract visual analogy task is made up of two rows of ﬁve context panels, and four possible candidate panels to complete the visual analogy.
Revised: The abstract visual analogy task is made up of two rows of ﬁve context panels, and four possible candidate panels to complete the visual analogy.

Original: The learner must understand the higher-order relation constituted by the ﬁrst row of three context panels, and then choose the candidate that satisﬁes the same relation with the two context panels in the bottom row as shown in Figure 1.
Revised: The learner must understand the higher-order relation constituted by the ﬁrst row of three context panels, and then choose the candidate that satisﬁes the same relation with the two context panels in the bottom row as shown in Figure 1.

Original: Hill et al.
Revised: Hill et al.

Original: (2019) showed that if the candidates presented during training were curated to maximize differences in structural relations, while being perceptually similar, the models performed more accurately in learning visual analogies.
Revised: (2019) showed that if the candidates presented during training were curated to maximize differences in structural relations, while being perceptually similar, the models performed more accurately in learning visual analogies.

Original: Thus, the authors verifed that incorporating structural differences to maximize the importance of structure mapping in the learning procedure also works for ML models.
Revised: Thus, the authors verifed that incorporating structural differences to maximize the importance of structure mapping in the learning procedure also works for ML models.

Original: However, it is not feasible to build or curate datasets to always exploit  the  structure mapping prior.
Revised: However, it is not feasible to build or curate datasets to always exploit a prior in the dataset to maximize structure learning.

Original: This would involve knowing a combinatorially large number of possible relational structures beforehand and manually selecting data to maximize learning these structures, a requirement that scales poorly with an increase in possible relational structures.
Revised: This would involve knowing a combinatorially large number of possible relational structures beforehand and manually selecting data to maximize learning these structures, a requirement that scales poorly with an increase in possible relational structures.

Original: To address this issue, we propose a twostage Neural Structure Mapping (NSM) framework to learn abstract visual analogies (Figure 2).
Revised: To address this issue, we propose a two-stage Neural Structure Mapping (NSM) framework to learn abstract visual analogies (Figure 2).

Original: Our ﬁrst-stage involves a visual relationship encoder that takes as input the visual context from the source domain and predicts the component visual concepts: objects, attributes and relationship in a multi-task manner.
Revised: Our ﬁrst-stage involves a visual relationship encoder that takes as input the visual context from the source domain and predicts the component visual concepts: objects, attributes and relationship in a multi-task manner.

Original: We then use the derived relation to dynamically build an analogy inference engine.
Revised: We then use the derived relation to dynamically build an analogy inference engine.

Original: Similar to structure mapping in humans, our engine utilizes only the relation from the source domain to pick the candidate panel that best ﬁts the analogy.
Revised: Similar to structure mapping in humans, our engine utilizes only the relation from the source domain to pick the candidate panel that best ﬁts the analogy.

Original: We incorporate compositional reasoning into our approach by using a neural module network (Andreas et al., 2016;
Revised: We incorporate compositional reasoning into our approach by using a neural module network (Andreas et al., 2016;

Original: Johnson et al., 2017b; Csord´as et al., 2021) as our engine.
Revised: Johnson et al., 2017b; Csord´as et al., 2021) as our engine.

Original: Our modular neural engine dynamically adapt its structure to align (Xu et al., 2020) with the structural relationship , thus explicitly incorporating the structure mapping process in our NSM model.
Revised: Our modular neural engine dynamically adapt its structure to align (Xu et al., 2020) with the structural relationship , thus explicitly incorporating the structure mapping process in our NSM model.

Original: We train and evaluate our approach on ﬁve different generalization splits of the visual analogy dataset.
Revised: We train and evaluate our approach on ﬁve different generalization splits of the visual analogy dataset.

Original: These splits were proposed by Hill et al.
Revised: These splits were proposed by Hill et al.

Original: (2019) to test systematic generalization (Bahdanau et al., 2018) across object and attribute values in learning visual analogies.
Revised: (2019) to test systematic generalization (Bahdanau et al., 2018) across object and attribute values in learning visual analogies.

Original: We show that our relationship encoder achieves a high degree of accuracy in isolating the relationship from the source domain, and, as expected, there is no performance drop across generalization splits or training regimes.
Revised: We show that our relationship encoder achieves a high degree of accuracy in isolating the relationship from the source domain, and, as expected, there is no performance drop across generalization splits or training regimes.

Original: For our analogy inference engine, we empirically show that it explicitly captures the structure mapping process, while generalizing systematically in learning the structural relation across visual domains.
Revised: For our analogy inference engine, we empirically show that it explicitly captures the structure mapping process, while generalizing systematically in learning the structural relation across visual domains.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learning Analogies Gentner (1983)’s
Revised: Learning Analogies Gentner (1983)’s

Original: SMT deﬁnes an analogy as a comparison in which relational predicates, but few or no object attributes, can be mapped from base to target domains.
Revised: SMT deﬁnes an analogy as a comparison in which relational predicates, but few or no object attributes, can be mapped from base to target domains.

Original: Symbolic models based on SMT (Falkenhainer et al., 1986), including those designed for RPMs (Lovett & Forbus, 2017), rely on ﬁrst extracting a rule based representation of the domains from the perceptual input.
Revised: Symbolic models based on SMT (Falkenhainer et al., 1986), including those designed for RPMs (Lovett & Forbus, 2017), rely on ﬁrst extracting a rule based representation of the domains from the perceptual input.

Original: Other cognitive analogy models based on Active Symbol theory (Hofstadter &
Revised: Other cognitive analogy models based on Active Symbol theory (Hofstadter &

Original: Mitchell, 1994; Mitchell, 1993) or High-Level Perception theory (Chalmers et al., 1992) do not separate the structure extraction and mapping process, nor rely on mapping the structure syntax across domains.
Revised: Mitchell, 1994; Mitchell, 1993) or High-Level Perception theory (Chalmers et al., 1992) do not separate the structure extraction and mapping process, nor rely on mapping the structure syntax across domains.

Original: Our approach is not directly related to these models.
Revised: Our approach is not directly related to these models.

Original: Hill et al.
Revised: Hill et al.

Original: (2019) introduced the dataset for learning analogies from RPMs and utilized several neural network architectures to directly learn from visual data.
Revised: (2019) introduced the dataset for learning analogies from RPMs and utilized several neural network architectures to directly learn from visual data.

Original: Their key contribution was to introduce the
Revised: Their key contribution was to introduce the

Original: SMT prior into the dataset during candidate selection.
Revised: SMT prior into the dataset during candidate selection.

Original: Webb et al.
Revised: Webb et al.

Original: (2020) introduced Temporal
Revised: (2020) introduced Temporal

Original: Context Normalization to explicitly learn visual features that can support extrapolation, and tested it on the Visual Analogy Extrapolation Challenge (VAEC) dataset.
Revised: Context Normalization to explicitly learn visual features that can support extrapolation, and tested it on the Visual Analogy Extrapolation Challenge (VAEC) dataset.

Original: Both of these approaches are complementary to ours.
Revised: Both of these approaches are complementary to ours.

Original: Other DL approaches most similar to ours are the Part-Composition Model (Ichien et al., 2021) and Chen et al.
Revised: Other DL approaches most similar to ours are the Part-Composition Model (Ichien et al., 2021) and Chen et al.

Original: (2019)’s approach.
Revised: (2019)’s approach.

Original: However, unlike our model, these models rely on availability of very structured intermediate representations like semantic segmentation maps, and do not use the extracted structural representations to explicitly conﬁgure the reasoning model.
Revised: However, unlike our model, these models rely on availability of very structured intermediate representations like semantic segmentation maps, and do not use the extracted structural representations to explicitly conﬁgure the reasoning model.

Original: Abstract Reasoning Hill et al.
Revised: Abstract Reasoning Hill et al.

Original: ’s dataset is directly derived from the Procedurally Generated Matrices (PGM) dataset introduced by Barrett et al.
Revised: ’s dataset is directly derived from the Procedurally Generated Matrices (PGM) dataset introduced by Barrett et al.

Original: (2018) to test the ability of neural networks to perform abstract reasoning on RPM problems.
Revised: (2018) to test the ability of neural networks to perform abstract reasoning on RPM problems.

Original: RPM tests are an important measure of ﬂuid intelligence in humans (Raven, 2000) and the PGM dataset provided a sufﬁciently large sample size for training neural networks on this problem.
Revised: RPM tests are an important measure of ﬂuid intelligence in humans (Raven, 2000) and the PGM dataset provided a sufﬁciently large sample size for training neural networks on this problem.

Original: Following this work, two new RPM datasets were also released: RAVEN
Revised: Following this work, two new RPM datasets were also released: RAVEN

Original: (Zhang et al., 2019), which utilized additional rules and structured rule annotations, and V-PROM (Teney et al., 2020), which utilized real images.
Revised: (Zhang et al., 2019), which utilized additional rules and structured rule annotations, and V-PROM (Teney et al., 2020), which utilized real images.

Original: DL approaches to RPM tasks can be roughly categorized into three types.
Revised: DL approaches to RPM tasks can be roughly categorized into three types.

Original: The ﬁrst is relation learning approaches, such as WReN (Barrett et al., 2018), which models all pairwise relationships between the matrix panels using relation networks (Santoro et al., 2017), and MXGNet (Wang et al., 2020), which learns row-based node embeddings and then performs a graph classiﬁcation on the resulting candidate graphs.
Revised: The ﬁrst is relation learning approaches, such as WReN (Barrett et al., 2018), which models all pairwise relationships between the matrix panels using relation networks (Santoro et al., 2017), and MXGNet (Wang et al., 2020), which learns row-based node embeddings and then performs a graph classiﬁcation on the resulting candidate graphs.

Original: Second is rule learning approaches, such as DRT (Zhang et al., 2019), that learns a structured Stochastic Image Grammar of the abstract rules, and LEN (Zheng et al., 2019) that utilizes a logic embedding network along with a curriculum to learn with increasingly distracting features.
Revised: Second is rule learning approaches, such as DRT (Zhang et al., 2019), that learns a structured Stochastic Image Grammar of the abstract rules, and LEN (Zheng et al., 2019) that utilizes a logic embedding network along with a curriculum to learn with increasingly distracting features.

Original: Third is object-centric learning, such as Rel-AIR (Spratley et al., 2020), that obtains object embeddings generated via Attend-Infer-Repeat for each panel, and Pekar et al.
Revised: Third is object-centric learning, such as Rel-AIR (Spratley et al., 2020), that obtains object embeddings generated via Attend-Infer-Repeat for each panel, and Pekar et al.

Original: (2020)’s method that utilizes both Variational AutoEncoders and an adversarial loss for candidate generation.
Revised: (2020)’s method that utilizes both Variational AutoEncoders and an adversarial loss for candidate generation.

Original: Our model can be broadly ﬁt into rule learning by way of the encoder, followed by relation learning via the analogy inference engine.
Revised: Our model can be broadly ﬁt into rule learning by way of the encoder, followed by relation learning via the analogy inference engine.

Original: However, our general approach towards explicitly extracting structure and mapping it is complimentary to these ideas and can be used in conjunction with several of them.
Revised: However, our general approach towards explicitly extracting structure and mapping it is complimentary to these ideas and can be used in conjunction with several of them.

Original: Systematic Generalization Systematic generalization refers to the ability to generalize to novel concepts (out of distribution data) by understanding them as compositions of known concepts (Bahdanau et al., 2018).
Revised: Systematic Generalization Systematic generalization refers to the ability to generalize to novel concepts (out of distribution data) by understanding them as compositions of known concepts (Bahdanau et al., 2018).

Original: It is underpinned by three important characteristics: (1) systematicity, the ability to generalize to semantically related concepts; (2) productivity, the ability to iteratively apply compounding to generalize from constituent concepts to their recurrence; and (3) localism, the ability to iteratively apply reductionism to generalize from repeated constituent concepts to the singular.
Revised: It is underpinned by three important characteristics: (1) systematicity, the ability to generalize to semantically related concepts; (2) productivity, the ability to iteratively apply compounding to generalize from constituent concepts to their recurrence; and (3) localism, the ability to iteratively apply reductionism to generalize from repeated constituent concepts to the singular.

Original: The ability to generalize systematically has been previously explored in human cognition (Fodor & Pylyshyn, 1988), natural language processing models (Lake & Baroni, 2018), and languagegrounded computer vision models (Agrawal et al., 2017).
Revised: The ability to generalize systematically has been previously explored in human cognition (Fodor & Pylyshyn, 1988), natural language processing models (Lake & Baroni, 2018), and languagegrounded computer vision models (Agrawal et al., 2017).

Original: It has its origins in human cognition and was formalized by Fodor (1975) under the ‘Language of Thought’ hypothesis.
Revised: It has its origins in human cognition and was formalized by Fodor (1975) under the ‘Language of Thought’ hypothesis.

Original: Cognitive scientists have expressed reservations about the ability of connectionist models to generalize systematicitally (Fodor & Pylyshyn, 1988; Marcus, 2019).
Revised: Cognitive scientists have expressed reservations about the ability of connectionist models to generalize systematicitally (Fodor & Pylyshyn, 1988; Marcus, 2019).

Original: Investigation of neural language models showed that systematic generalization still poses a challenge for DL (Lake & Baroni, 2018;
Revised: Investigation of neural language models showed that systematic generalization still poses a challenge for DL (Lake & Baroni, 2018;

Original: Loula et al., 2018).
Revised: Loula et al., 2018).

Original: A key reason deep neural networks lack systematicity is due to their propensity towards ‘shortcut learning’
Revised: A key reason deep neural networks lack systematicity is due to their propensity towards ‘shortcut learning’

Original: (Geirhos et al., 2018; 2020).
Revised: (Geirhos et al., 2018; 2020).

Original: This suggests that DL models rely on exploiting a few number of predictive features instead of considering all possible information about the data while drawing conclusions.
Revised: This suggests that DL models rely on exploiting a few number of predictive features instead of considering all possible information about the data while drawing conclusions.

Original: Consequently, this leads to poor generalization when the key predictive features are changed, even though the central evidence remains the same.
Revised: Consequently, this leads to poor generalization when the key predictive features are changed, even though the central evidence remains the same.

Original: For example,
Revised: For example,

Original: DL approaches to solving RPM problems were discovered to rely on evaluating the mode across all candidates in the RAVEN dataset and a signiﬁcant drop in generalization performance was observed when the models were tested without the shortcut on the RAVEN-Fair dataset (Spratley et al., 2020).
Revised: DL approaches to solving RPM problems were discovered to rely on evaluating the mode across all candidates in the RAVEN dataset and a signiﬁcant drop in generalization performance was observed when the models were tested without the shortcut on the RAVEN-Fair dataset (Spratley et al., 2020).

Original: On the other hand, neural module networks (Andreas et al., 2016; Johnson et al., 2017b) generalized well systematically when their layout aligned well with speciﬁc language-grounded reasoning problems (Bahdanau et al., 2018).
Revised: On the other hand, neural module networks (Andreas et al., 2016; Johnson et al., 2017b) generalized well systematically when their layout aligned well with speciﬁc language-grounded reasoning problems (Bahdanau et al., 2018).

Original: Hence, we utilize a modular approach in building our analogy inference engine, which in turn utilizes the relationship structure inferred by our encoder to automatically align itself to the problem.
Revised: Hence, we utilize a modular approach in building our analogy inference engine, which in turn utilizes the relationship structure inferred by our encoder to automatically align itself to the problem.

Original: This is a form of conditional computation.
Revised: This is a form of conditional computation.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: The abstract visual analogy task (Hill et al., 2019) falls under the umbrella of Raven’s Progressive
Revised: The abstract visual analogy task (Hill et al., 2019) falls under the umbrella of Raven’s Progressive

Original: Matrices (Raven, 2000) designed to test abstract reasoning and ﬂuid intelligence in human and artiﬁcial agents.
Revised: Matrices (Raven, 2000) designed to test abstract reasoning and ﬂuid intelligence in human and artiﬁcial agents.

Original: Each question consists of ﬁve context panels P 1 ´ 5 con , and four candidate panels
Revised: Each question consists of ﬁve context panels P 1 ´ 5 con , and four candidate panels

Original: P 1 ´ 4 can .
Revised: P 1 ´ 4 can .

Original: The context panels are arranged in a matrix with two rows of three columns each, with the ﬁnal panel in the second row missing.
Revised: The context panels are arranged in a matrix with two rows of three columns each, with the ﬁnal panel in the second row missing.

Original: The ﬁrst row of panels P 1 ´ 3 con express a semantically related triplet R “ t o, a, r u , that is composed of two lower-order perceptual visual concepts: object o and attribute a , and one higher-order semantic visual concept: relationship r .
Revised: The ﬁrst row of panels P 1 ´ 3 con express a semantically related triplet R “ t o, a, r u , that is composed of two lower-order perceptual visual concepts: object o and attribute a , and one higher-order semantic visual concept: relationship r .

Original: The objective is to choose a candidate c P Ť 4 i “ 1 t P i can u that, upon substitution in place of the missing panel, represents the same relationship as the ﬁrst row of panels.
Revised: The objective is to choose a candidate c P Ť 4 i “ 1 t P i can u that, upon substitution in place of the missing panel, represents the same relationship as the ﬁrst row of panels.

Original: The object ( o
Revised: The object ( o

Original: P { line, shape } ) and attribute ( a
Revised: P { line, shape } ) and attribute ( a

Original: P { quantity,colour,type,size,position } ) together constitute the visual domain d of the triplet R .
Revised: P { quantity,colour,type,size,position } ) together constitute the visual domain d of the triplet R .

Original: Each possible attribute can take ten values v p a q P t 1 , 2 , ..., 9 , 10 u (normalized).
Revised: Each possible attribute can take ten values v p a q P t 1 , 2 , ..., 9 , 10 u (normalized).

Original: The domain of the ﬁrst row of panels P 1 ´ 3 con deﬁnes the source domain d source of the analogy, and the domain of the second row of panels t P 4 ´ 5 con , c u deﬁnes the target domain d target .
Revised: The domain of the ﬁrst row of panels P 1 ´ 3 con deﬁnes the source domain d source of the analogy, and the domain of the second row of panels t P 4 ´ 5 con , c u deﬁnes the target domain d target .

Original: The dataset consists of seven unique possible domains such that d P { shape quantity, shape colour, shape type, shape size, shape position, line type, line colour } .
Revised: The dataset consists of seven unique possible domains such that d P { shape quantity, shape colour, shape type, shape size, shape position, line type, line colour } .

Original: To test the systematic generalization in learning visual analogies, the dataset has ﬁve different generalization splits that require the ability to recognize the relationship across both novel domain d and attribute a values:
Revised: To test the systematic generalization in learning visual analogies, the dataset has ﬁve different generalization splits that require the ability to recognize the relationship across both novel domain d and attribute a values:

Original: • Novel Domain Transfer: The training set consists of 42 ordered pairs of d source and d target while the remaining 7 (7*7 - 42) pairs are present only in the test set i.e. d trainsource Ś d traintarget X d testsource Ś d testtarget “ H .
Revised: • Novel Domain Transfer: The training set consists of 42 ordered pairs of d source and d target while the remaining 7 (7*7 - 42) pairs are present only in the test set i.e. d trainsource Ś d traintarget X d testsource Ś d testtarget “ H .

Original: • Novel Domain Type ( line type and shape color ) :
Revised: • Novel Domain Type ( line type and shape color ) :

Original: The training set does not contain any problems with the held-out domain i.e. d train R { line type } or d train R { shape colour } , while each problem in the test set involves the held-out domain.
Revised: The training set does not contain any problems with the held-out domain i.e. d train R { line type } or d train R { shape colour } , while each problem in the test set involves the held-out domain.

Original: • Novel Domain Values (Interpolation and Extrapolation): The training and test sets are made of two mutually exclusive sets of attribute values.
Revised: • Novel Domain Values (Interpolation and Extrapolation): The training and test sets are made of two mutually exclusive sets of attribute values.

Original: For the extrapolation split, v p a q train P t 1 , 2 , 3 , 4 , 5 u while v p a q test P t 6 , 7 , 8 , 9 , 10 u .
Revised: For the extrapolation split, v p a q train P t 1 , 2 , 3 , 4 , 5 u while v p a q test P t 6 , 7 , 8 , 9 , 10 u .

Original: For interpolation, v p a q train P t 2 , 4 , 6 , 8 , 10 u while v p a q test P t 1 , 3 , 5 , 7 , 9 u .
Revised: For interpolation, v p a q train P t 2 , 4 , 6 , 8 , 10 u while v p a q test P t 1 , 3 , 5 , 7 , 9 u .

Original: Each of the splits presents a unique challenge in terms of generalization, yet shares the idea that understanding an analogy involves identifying the higher-order visual relation from the source domain and applying it to a target domain.
Revised: Each of the splits presents a unique challenge in terms of generalization, yet shares the idea that understanding an analogy involves identifying the higher-order visual relation from the source domain and applying it to a target domain.

Original: The relationship ( r P { progression, AND,
Revised: The relationship ( r P { progression, AND,

Original: OR, XOR } ) in the visual analogy dataset can be of classiﬁed into two types: Unary and Binary.
Revised: OR, XOR } ) in the visual analogy dataset can be of classiﬁed into two types: Unary and Binary.

Original: The Unary relation, progression , is composed of a function applied to a single panel to produce the next panel in a row.
Revised: The Unary relation, progression , is composed of a function applied to a single panel to produce the next panel in a row.

Original: On the other hand, the Binary relations, ( AND, OR, XOR ), are composed of a function applied to the ﬁrst two panels in a row to generate the ﬁnal panel.
Revised: On the other hand, the Binary relations, ( AND, OR, XOR ), are composed of a function applied to the ﬁrst two panels in a row to generate the ﬁnal panel.

Original: Thus, the relationship deﬁnes the semantics across a row of panels and is the core abstraction of analogical reasoning.
Revised: Thus, the relationship deﬁnes the semantics across a row of panels and is the core abstraction of analogical reasoning.

Original: Hill et al.
Revised: Hill et al.

Original: (2019) hypothesized that introducing a prior on the learning process that requires the learner to correctly identify the relationship would align with how humans learn structure for analogical mapping.
Revised: (2019) hypothesized that introducing a prior on the learning process that requires the learner to correctly identify the relationship would align with how humans learn structure for analogical mapping.

Original: They designed this prior by carefully controlling the candidate panels P 1 ´ 4 can presented during model training (Figure 3).
Revised: They designed this prior by carefully controlling the candidate panels P 1 ´ 4 can presented during model training (Figure 3).

Original: This regime, called Learning Analogies By Contrasting
Revised: This regime, called Learning Analogies By Contrasting

Original: [Figure]
Revised: [Figure]

Original: Each candidate satisﬁes one of the four possible relationships with the target domain panels.
Revised: Each candidate satisﬁes one of the four possible relationships with the target domain panels.

Original: However, only the correct panel satisﬁes the relationship in the source domain panels.
Revised: However, only the correct panel satisﬁes the relationship in the source domain panels.

Original: The authors found that LABC training signiﬁcantly improved the ability to learn analogies in DL models compared to training with merely perceptually possible candidates (Normal training).
Revised: The authors found that LABC training signiﬁcantly improved the ability to learn analogies in DL models compared to training with merely perceptually possible candidates (Normal training).

Original: In fact, even a mixed training regime consisting of both LABC and Normal problems showed signiﬁcant improvement on the Normal training regime.
Revised: In fact, even a mixed training regime consisting of both LABC and Normal problems showed signiﬁcant improvement on the Normal training regime.

Original: To simplify presentation, in what follows we will use the term “Contrasting” for the use of such examples during training (LABC) and in evaluation.
Revised: To simplify presentation, in what follows we will use the term “Contrasting” for the use of such examples during training (LABC) and in evaluation.

Original:  M ETHOD
Revised:  M ETHOD

Original: In our work, we take a complimentary approach to Hill et al.
Revised: In our work, we take a complimentary approach to Hill et al.

Original: for learning abstract visual analogies in RPMs.
Revised: for learning abstract visual analogies in RPMs.

Original: Instead of depending on explicitly labeled candidates for mapping relational structure, we build a model that dynamically conﬁgures its structure based on the relational structure extracted from the source domain.
Revised: Instead of depending on explicitly labeled candidates for mapping relational structure, we build a model that dynamically conﬁgures its structure based on the relational structure extracted from the source domain.

Original: Internally, our approach is made up of two different neural networks that correspond to the two separate tasks in our pipeline.
Revised: Internally, our approach is made up of two different neural networks that correspond to the two separate tasks in our pipeline.

Original: The ﬁrst step is the
Revised: The ﬁrst step is the

Original: Visual Relationship Encoder, R pred “ φ p P 1 ´ 3 con q , that predicts the visual relationship triplet
Revised: Visual Relationship Encoder, R pred “ φ p P 1 ´ 3 con q , that predicts the visual relationship triplet

Original: R “ t o, a, r u encoded in the source domain panels P 1 ´ 3 con .
Revised: R “ t o, a, r u encoded in the source domain panels P 1 ´ 3 con .

Original: Next is the Analogy Inference Engine, c “ π p P 4 ´ 5 con , P 1 ´ 4 can , r q , that takes the predicted source relation r , and selects the candidate c P Ť 4 i “ 1 t P i can u that maximizes the probability of this relation in the target domain P 4 ´ 5 con .
Revised: Next is the Analogy Inference Engine, c “ π p P 4 ´ 5 con , P 1 ´ 4 can , r q , that takes the predicted source relation r , and selects the candidate c P Ť 4 i “ 1 t P i can u that maximizes the probability of this relation in the target domain P 4 ´ 5 con .

Original: We next provide detailed discussion on each of these networks in our pipeline.
Revised: We next provide detailed discussion on each of these networks in our pipeline.

Original: For implementation details of the networks please refer to Appendix C.
Revised: For implementation details of the networks please refer to Appendix C.

Original: 4. V ISUAL R ELATIONSHIP E NCODER
Revised: 4. V ISUAL R ELATIONSHIP E NCODER

Original: The Visual Relationship Encoder, R pred “ φ p P 1 ´ 3 con q , segregates the abstract relationship triplet in the source domain panels into its constituent object, attribute , and relationship .
Revised: The Visual Relationship Encoder, R pred “ φ p P 1 ´ 3 con q , segregates the abstract relationship triplet in the source domain panels into its constituent object, attribute , and relationship .

Original: This step of our NSM model corresponds to the structure extraction phase in SMT in which humans segregate the higher-order relation concepts in perceptual information from the underlying domain concepts.
Revised: This step of our NSM model corresponds to the structure extraction phase in SMT in which humans segregate the higher-order relation concepts in perceptual information from the underlying domain concepts.

Original: The encoder φ is made up of a multi-task neural network.
Revised: The encoder φ is made up of a multi-task neural network.

Original: We use a convolutional neural network (CNN)
Revised: We use a convolutional neural network (CNN)

Original: C φ to ﬁrst extract a visual feature vector vis i φ “ C φ p P
Revised: C φ to ﬁrst extract a visual feature vector vis i φ “ C φ p P

Original: i con q for each context panel.
Revised: i con q for each context panel.

Original: The extracted visual features are then sequentially fed into a long short term memory network (LSTM)
Revised: The extracted visual features are then sequentially fed into a long short term memory network (LSTM)

Original: R φ to generate visual sequence features seq φ of the source domain.
Revised: R φ to generate visual sequence features seq φ of the source domain.

Original: The hidden layer vector after the third sequence panel is taken as the encoded features of the source domain seq φ “ R φ p vis 1 ´ 3 q .
Revised: The hidden layer vector after the third sequence panel is taken as the encoded features of the source domain seq φ “ R φ p vis 1 ´ 3 q .

Original: The source domain feature vector seq φ is then passed to a fully connected layer l shared φ “ FC shared φ p seq φ q that is shared between all three visual components.
Revised: The source domain feature vector seq φ is then passed to a fully connected layer l shared φ “ FC shared φ p seq φ q that is shared between all three visual components.

Original: The shared linear fea-
Revised: The shared linear fea-

Original: [Figure]
Revised: [Figure]

Original: is then passed through three different fully connected for each task t P { object, attribute, relation classiﬁcation.
Revised: is then passed through three different fully connected for each task t P { object, attribute, relation classiﬁcation.

Original: Finally, each l passed through a fully connected layer out
Revised: Finally, each l passed through a fully connected layer out

Original: “ FC out task φ p l task φ q of sizes 2, 5, and 4 for the object, attribute, and relation prediction tasks respectively.
Revised: “ FC out task φ p l task φ q of sizes 2, 5, and 4 for the object, attribute, and relation prediction tasks respectively.

Original: A Softmax over the outputs out task yields a probability distribution over the possible t o, a, r u values that constitute the triplet R .
Revised: A Softmax over the outputs out task yields a probability distribution over the possible t o, a, r u values that constitute the triplet R .

Original: 4. A NALOGY I NFERENCE E NGINE
Revised: 4. A NALOGY I NFERENCE E NGINE

Original: The Analogy Inference Engine, c “ π p P 4 ´ 5 con , P 1 ´ 4 can , r q , maps the source relationship r extracted by the Visual Relationship Encoder to the target domain P 4 ´ 5 con to identify the correct candidate c that completes the visual analogy.
Revised: The Analogy Inference Engine, c “ π p P 4 ´ 5 con , P 1 ´ 4 can , r q , maps the source relationship r extracted by the Visual Relationship Encoder to the target domain P 4 ´ 5 con to identify the correct candidate c that completes the visual analogy.

Original: This step of our NSM model corresponds to the structure mapping phase in SMT, in which humans apply relations extracted from the source domain to a new domain to reason about concepts in the target domain.
Revised: This step of our NSM model corresponds to the structure mapping phase in SMT, in which humans apply relations extracted from the source domain to a new domain to reason about concepts in the target domain.

Original: To generalize systematically to new domains and attribute values, we utilize a modular approach (Andreas et al., 2016) to build our analogy inference engine.
Revised: To generalize systematically to new domains and attribute values, we utilize a modular approach (Andreas et al., 2016) to build our analogy inference engine.

Original: Similar to Johnson et al.
Revised: Similar to Johnson et al.

Original: (2017b)
Revised: (2017b)

Original: and Bahdanau et al.
Revised: and Bahdanau et al.

Original: (2018), we use modules with a generic architecture.
Revised: (2018), we use modules with a generic architecture.

Original: Our engine is made up of four different types of neural modules:
Revised: Our engine is made up of four different types of neural modules:

Original: • The Stem module performs a series of convolution operations with stride=2.
Revised: • The Stem module performs a series of convolution operations with stride=2.

Original: It takes the original grayscale images of size 1 ˆ 160 ˆ 160 as input and returns a visual feature map of size 8 ˆ 9 ˆ 9 p C ˆ H ˆ W q .
Revised: It takes the original grayscale images of size 1 ˆ 160 ˆ 160 as input and returns a visual feature map of size 8 ˆ 9 ˆ 9 p C ˆ H ˆ W q .

Original: Each of P 4 ´ 5 con and P 1 ´ 4 can panels is ﬁrst passed through the Stem module to extract visual features before being processed by the rest of the engine.
Revised: Each of P 4 ´ 5 con and P 1 ´ 4 can panels is ﬁrst passed through the Stem module to extract visual features before being processed by the rest of the engine.

Original: The Unary module is a residual block with two 3 ˆ 3 convolution layers.
Revised: The Unary module is a residual block with two 3 ˆ 3 convolution layers.

Original: It receives one feature map p C
Revised: It receives one feature map p C

Original: ˆ H
Revised: ˆ H

Original: ˆ W q as input, and returns one feature map of p C
Revised: ˆ W q as input, and returns one feature map of p C

Original: ˆ H
Revised: ˆ H

Original: ˆ W q as output.
Revised: ˆ W q as output.

Original: • The Binary module receives two feature maps, concatenates them along the channel dimension, and uses a 1 ˆ 1 convolution to project them to C dimensions.
Revised: • The Binary module receives two feature maps, concatenates them along the channel dimension, and uses a 1 ˆ 1 convolution to project them to C dimensions.

Original: It then passes the combined feature maps through a residual block, and returns one feature map of p C
Revised: It then passes the combined feature maps through a residual block, and returns one feature map of p C

Original: ˆ H
Revised: ˆ H

Original: ˆ W q as output.
Revised: ˆ W q as output.

Original: • The Classifier module also receives two feature maps, and projects them to C dimensions.
Revised: • The Classifier module also receives two feature maps, and projects them to C dimensions.

Original: It then ﬂattens the feature maps and passes them through two fully connected layers to generate a probability distribution over the relations.
Revised: It then ﬂattens the feature maps and passes them through two fully connected layers to generate a probability distribution over the relations.

Original: Each candidate panel is fed parallely to the engine to generate its predictions for all four relations.
Revised: Each candidate panel is fed parallely to the engine to generate its predictions for all four relations.

Original: In order to generate the candidate selection, we subset the probability of relationship r
Revised: In order to generate the candidate selection, we subset the probability of relationship r

Original: ( r pred during inference) for all candidates, and select the candidate with the highest corresponding probability.
Revised: ( r pred during inference) for all candidates, and select the candidate with the highest corresponding probability.

Original: The layout of the module network is chosen adaptively from two possible layouts based on the source domain relation at inference time.
Revised: The layout of the module network is chosen adaptively from two possible layouts based on the source domain relation at inference time.

Original: Each layout ﬁrst processes the individual panels through the Stem module to extract visual features, followed by the Unary module.
Revised: Each layout ﬁrst processes the individual panels through the Stem module to extract visual features, followed by the Unary module.

Original: In the ﬁrst layout, the
Revised: In the ﬁrst layout, the

Original: Unary module outputs for the target domain panels P 4 ´ 5 con are combined using a Binary module, and the outputs for the second panel P 5 con and each candidate panel P 1 ´ 4 can are combined using another
Revised: Unary module outputs for the target domain panels P 4 ´ 5 con are combined using a Binary module, and the outputs for the second panel P 5 con and each candidate panel P 1 ´ 4 can are combined using another

Original: Binary module (Figure 4a).
Revised: Binary module (Figure 4a).

Original: The outputs of both the Binary modules are then fed into the
Revised: The outputs of both the Binary modules are then fed into the

Original: Classifier module.
Revised: Classifier module.

Original: In the second layout, the Unary module outputs for the target domain panels P 4 ´ 5 con are similarly combined using a Binary module.
Revised: In the second layout, the Unary module outputs for the target domain panels P 4 ´ 5 con are similarly combined using a Binary module.

Original: However, the output of this Binary module and the Unary candidate output is directly fed into the Classifier module (Figure 4b).
Revised: However, the output of this Binary module and the Unary candidate output is directly fed into the Classifier module (Figure 4b).

Original: We provide further discussion on these layouts in Appendix A.
Revised: We provide further discussion on these layouts in Appendix A.

Original:  E XPERIMENTS AND R ESULTS 5. V ISUAL R ELATIONSHIP E NCODER
Revised:  E XPERIMENTS AND R ESULTS 5. V ISUAL R ELATIONSHIP E NCODER

Original: Training The Visual Relationship Encoder φ is trained using the source domain context panels
Revised: Training The Visual Relationship Encoder φ is trained using the source domain context panels

Original: P 1 ´ 3 con and the visual relationship labels R “ t o, a, r u .
Revised: P 1 ´ 3 con and the visual relationship labels R “ t o, a, r u .

Original: The cross-entropy loss for each task is obtained using the multi-task network’s predictions and triplet labels.
Revised: The cross-entropy loss for each task is obtained using the multi-task network’s predictions and triplet labels.

Original: The total encoder loss is calculated as a weighted sum of the individual losses for object, attribute, and relation prediction:
Revised: The total encoder loss is calculated as a weighted sum of the individual losses for object, attribute, and relation prediction:

Original: [Equation]
Revised: [Equation]

Original: encoder
Revised: encoder

Original: [Equation]
Revised: [Equation]

Original: Empirically, we found that the values of α “ 0 . 5 , β “ 0 . 5 , γ “ 2 . 0 led to higher validation accuracy.
Revised: Empirically, we found that the values of α “ 0 . 5 , β “ 0 . 5 , γ “ 2 . 0 led to higher validation accuracy.

Original: The model was trained for 100 epochs using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 1 e ´ 4 .
Revised: The model was trained for 100 epochs using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 1 e ´ 4 .

Original: For each generalization split, the model was trained with contrasting, normal, as well as mixed candidates.
Revised: For each generalization split, the model was trained with contrasting, normal, as well as mixed candidates.

Original: The model with the best validation accuracy was chosen for testing (the validation set consisted of a similar set of analogy candidates as the training set).
Revised: The model with the best validation accuracy was chosen for testing (the validation set consisted of a similar set of analogy candidates as the training set).

Original: 5.1. R ESULTS
Revised: 5.1. R ESULTS

Original: We test the relationship prediction accuracy of our encoder across all ﬁve generalization splits, with each possible set of analogy candidates (Contrasting, Normal, Mixed).
Revised: We test the relationship prediction accuracy of our encoder across all ﬁve generalization splits, with each possible set of analogy candidates (Contrasting, Normal, Mixed).

Original: Our relationship encoder was able to isolate the component relationship from the source domain with a high degree of accuracy (min: 82.4%, max: 86.04%) across all generalization splits and all three types of candidates (Figure 5).
Revised: Our relationship encoder was able to isolate the component relationship from the source domain with a high degree of accuracy (min: 82.4%, max: 86.04%) across all generalization splits and all three types of candidates (Figure 5).

Original: Furthermore, since the encoder utilizes only the source domain information in determining the relation, we observe no drop between training and test accuracy across all generalization splits.
Revised: Furthermore, since the encoder utilizes only the source domain information in determining the relation, we observe no drop between training and test accuracy across all generalization splits.

Original: By identifying the visual relationship in the source domain, the output of the visual relationship encoder is able to determine the layout for the subsequent analogy inference engine.
Revised: By identifying the visual relationship in the source domain, the output of the visual relationship encoder is able to determine the layout for the subsequent analogy inference engine.

Original: Since the inference engine chooses between only two possible layouts, a baseline that always chooses the majority class layout would lead to a correct choice in 75% of problems.
Revised: Since the inference engine chooses between only two possible layouts, a baseline that always chooses the majority class layout would lead to a correct choice in 75% of problems.

Original: Hence, our encoder must
Revised: Hence, our encoder must

Original: [Figure]
Revised: [Figure]

Original: achieve a high degree of accuracy in identifying the correct layout to beat this baseline.
Revised: achieve a high degree of accuracy in identifying the correct layout to beat this baseline.

Original: Our encoder is indeed able to identify the layout matching the reasoning problem with near perfect accuracy (min: 98.42%, max: 99.9%).
Revised: Our encoder is indeed able to identify the layout matching the reasoning problem with near perfect accuracy (min: 98.42%, max: 99.9%).

Original: The full results for layout prediction across all generalization splits and training regimes are in Table 15 in Appendix D.
Revised: The full results for layout prediction across all generalization splits and training regimes are in Table 15 in Appendix D.

Original: A NALOGY I NFERENCE E NGINE
Revised: A NALOGY I NFERENCE E NGINE

Original: Training The Analogy Inference Engine π is trained using the target domain context panels P 4 ´ 5 con , the candidate panels P 1 ´ 4 can , the ground truth candidate label, and the ground truth visual relationship label r .
Revised: Training The Analogy Inference Engine π is trained using the target domain context panels P 4 ´ 5 con , the candidate panels P 1 ´ 4 can , the ground truth candidate label, and the ground truth visual relationship label r .

Original: The engine was trained for 100 epochs using the Adam optimizer with a learning rate of 1 e ´ 4 .
Revised: The engine was trained for 100 epochs using the Adam optimizer with a learning rate of 1 e ´ 4 .

Original: Similar to the ﬁrst stage, the engine was also trained with contrasting, normal, as well as mixed candidates.
Revised: Similar to the ﬁrst stage, the engine was also trained with contrasting, normal, as well as mixed candidates.

Original: The model with the highest validation accuracy was chosen for testing (validation set consisted of a similar set of analogy candidates as the training set).
Revised: The model with the highest validation accuracy was chosen for testing (validation set consisted of a similar set of analogy candidates as the training set).

Original: Testing For testing, we rely on the relationship label r pred predicted by our encoder instead of ground truth r .
Revised: Testing For testing, we rely on the relationship label r pred predicted by our encoder instead of ground truth r .

Original: While all baselines (see below) are trained with the full set of context panels P 1 ´ 5 con , our Analogy Inference Engine only utilizes the target domain panels P 4 ´ 5 con for context.
Revised: While all baselines (see below) are trained with the full set of context panels P 1 ´ 5 con , our Analogy Inference Engine only utilizes the target domain panels P 4 ´ 5 con for context.

Original: Hence, we also provide the results of a full context model which utilizes all context panels P 1 ´ 5 con .
Revised: Hence, we also provide the results of a full context model which utilizes all context panels P 1 ´ 5 con .

Original: For this purpose, we leverage our encoder which is trained with the remaining context panels P 1 con – P 3 con .
Revised: For this purpose, we leverage our encoder which is trained with the remaining context panels P 1 con – P 3 con .

Original: We use it to ﬁrst extract the relationship r from the source domain as discussed in Section 4.1.
Revised: We use it to ﬁrst extract the relationship r from the source domain as discussed in Section 4.1.

Original: In addition, we also generate candidate probabilities from the encoder during inference, by considering each candidate in parallel followed by a Softmax over the probability of r across all candidates as described in Section 4.2.
Revised: In addition, we also generate candidate probabilities from the encoder during inference, by considering each candidate in parallel followed by a Softmax over the probability of r across all candidates as described in Section 4.2.

Original: The candidate probabilities of the full context NSM model are calculated as a 1:1 ensemble between the probabilities from both the inference engine and the encoder.
Revised: The candidate probabilities of the full context NSM model are calculated as a 1:1 ensemble between the probabilities from both the inference engine and the encoder.

Original: Baselines We compare the candidate selection accuracy of our analogy inference engine with each model utilized by Hill et al.
Revised: Baselines We compare the candidate selection accuracy of our analogy inference engine with each model utilized by Hill et al.

Original: These baselines include a CNN-LSTM model, a standard
Revised: These baselines include a CNN-LSTM model, a standard

Original: ResNet50 (with 9 input channels for 9 panels), a parallel ResNet50 (with 6 channels and each candidate fed in parallel)
Revised: ResNet50 (with 9 input channels for 9 panels), a parallel ResNet50 (with 6 channels and each candidate fed in parallel)

Original: [Table]
Revised: [Table]

Original: and a Wide Relation Network.
Revised: and a Wide Relation Network.

Original: Each baseline is trained for 100 epochs using the Adam optimizer with a learning rate of 1 e ´ 4 , with contrasting, normal, and mixed candidates.
Revised: Each baseline is trained for 100 epochs using the Adam optimizer with a learning rate of 1 e ´ 4 , with contrasting, normal, and mixed candidates.

Original: 5.2. R ESULTS
Revised: 5.2. R ESULTS

Original: Does the modular analogy inference engine generalize systematically?
Revised: Does the modular analogy inference engine generalize systematically?

Original: We hypothesize that choosing a modular engine for analogy inference enables compositional reasoning about novel domain values (objects and attributes).
Revised: We hypothesize that choosing a modular engine for analogy inference enables compositional reasoning about novel domain values (objects and attributes).

Original: In Table 2 we compare the generalization performance of our engine on the Novel Attribute Value: Extrapolation and Novel Domain: Line
Revised: In Table 2 we compare the generalization performance of our engine on the Novel Attribute Value: Extrapolation and Novel Domain: Line

Original: Type test splits of the analogy dataset.
Revised: Type test splits of the analogy dataset.

Original: Our approach achieves the highest test accuracy across all three training candidate regimes in the Novel Domain setting when tested with contrasting candidates.
Revised: Our approach achieves the highest test accuracy across all three training candidate regimes in the Novel Domain setting when tested with contrasting candidates.

Original: Furthermore, our approach yields high test accuracy (highest or within ă 1% ) for the
Revised: Furthermore, our approach yields high test accuracy (highest or within ă 1% ) for the

Original: Contrasting and Normal training candidate regimes in the Novel Attribute Value setting.
Revised: Contrasting and Normal training candidate regimes in the Novel Attribute Value setting.

Original: Thus, our
Revised: Thus, our

Original: NSM approach is better at systematic generalization to novel visual domains than monolithic neural
Revised: NSM approach is better at systematic generalization to novel visual domains than monolithic neural

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: networks.
Revised: networks.

Original: For completeness, we provide the results for both our evaluation setups across all training and test regimes in Tables 16 and 17 in Appendix D.
Revised: For completeness, we provide the results for both our evaluation setups across all training and test regimes in Tables 16 and 17 in Appendix D.

Original: Does our model learnthe structure mapping prior?
Revised: Does our model’s prior enable it learn structure mapping?

Original: We speciﬁcally evaluate the Normal training scenario where the candidates presented to the learner are drawn randomly and lack the structure mapping prior  explicitly present when training with Contrasting.
Revised: We speciﬁcally evaluate the Normal training scenario where the candidates presented to the learner are drawn randomly and lack the contrasting prior which explicitly promotes structure learning.

Original: Since our NSM model explicitly captures and maps the structure of the visual analogy, thus making up for the lack of the structural prior in the candidates, we anticipate that the test performance of our model trained with normal candidates would outperform other baselines.
Revised: Since our NSM model explicitly captures and maps the structure of the visual analogy, thus making up for the lack of a prior in the candidates, we anticipate that the test performance of our model trained with normal candidates would outperform other baselines.

Original: We visualize the generalization performance of this training scenario for the Novel Domain Value: Extrapolation and Novel Domain: Line Type generalization splits in Figure 6.
Revised: We visualize the generalization performance of this training scenario for the Novel Domain Value: Extrapolation and Novel Domain: Line Type generalization splits in Figure 6.

Original: We observe that our model outperforms other networks when averaged across both types of candidates during testing, demonstrating that  it  learnsthe structure mappingprior better than other models.
Revised: We observe that our model outperforms other networks when averaged across both types of candidates during testing, demonstrating that our model’s prior enables it to learns structure mapping better than other models.

Original: How important is structure mapping in drawing the correct analogy?
Revised: How important is structure mapping in drawing the correct analogy?

Original: We verify the importance of explicitly mapping structure on our NSM approach by comparing our results when the relationship label inferred in the ﬁrst step matches the ground truth relationship label.
Revised: We verify the importance of explicitly mapping structure on our NSM approach by comparing our results when the relationship label inferred in the ﬁrst step matches the ground truth relationship label.

Original: We present the confusion matrix in Table 2 where it can be seen that the test candidate selection accuracy for the correct structure mapping (diagonal entries) is signiﬁcantly higher than incorrect structure mapping.
Revised: We present the confusion matrix in Table 2 where it can be seen that the test candidate selection accuracy for the correct structure mapping (diagonal entries) is signiﬁcantly higher than incorrect structure mapping.

Original: Prog.
Revised: Prog.

Original: XOR OR
Revised: XOR OR

Original: AND 
Revised: AND Prog

Original: [Table]
Revised: [Table]

Original:  OR43.
Revised: 88 XOR [Table] OR

Original: [Table]
Revised: [Table]

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this work
Revised: In this work

Original: [Table]
Revised: [Table]

Original: we introduce a two-stage neural framework for learning abstract visual analogies based on the Structure Mapping Theory of human analogy making.
Revised: we introduce a two-stage neural framework for learning abstract visual analogies based on the Structure Mapping Theory of human analogy making.

Original: Our ﬁrst stage is a multi-task Visual
Revised: Our ﬁrst stage is a multi-task Visual

Original: Relationship Encoder that corresponds to structure extraction from perceptual information in humans.
Revised: Relationship Encoder that corresponds to structure extraction from perceptual information in humans.

Original: Our second stage is a modular Analogy Inference Engine that corresponds to mapping higherorder relations for structure mapping in humans.
Revised: Our second stage is a modular Analogy Inference Engine that corresponds to mapping higherorder relations for structure mapping in humans.

Original: Our approach is able to generalize systematically to novel target domains, compensate for the lack of astructure mapping prior in candidate selection
Revised: Our approach is able to generalize systematically to novel target domains, compensate for the lack of a prior in candidate selection

Original: [Table]
Revised: [Table]

Original: and successfully exploit the process of extracting and mapping the relationship structure.
Revised: and successfully exploit the process of extracting and mapping the relationship structure.

Original: In future work
Revised: In future work

Original: [Table]
Revised: [Table]

Original: we plan to investigate search-based (e
Revised: we plan to investigate search-based (e

Original: [Table]
Revised: [Table]

Original: g. Neural Architecture Search) as well as differentiable (e
Revised: g. Neural Architecture Search) as well as differentiable (e

Original: [Table]
Revised: [Table]

Original: g. attention-based) methods for generating the layouts of our Analogy Inference Engine.
Revised: g. attentionbased) methods for generating the layouts of our Analogy Inference Engine.

Original: Furthermore, we would like explore approaches that combine our structure mapping framework into existing methods for reasoning about Raven’s Progressive Matrices beyond analogy making [Table]
Revised: Furthermore, we would like explore approaches that combine our structure mapping framework into existing methods for reasoning about Raven’s Progressive Matrices beyond analogy making.

Original: A H OW DO WE CHOSE THE MODULE NETWORK LAYOUTS ?
Revised: A H OW DO WE CHOSE THE MODULE NETWORK LAYOUTS ?

Original: The generic architecture of our modules allows for a combinatorially large number of possible architectural layouts of our inference engine.
Revised: The generic architecture of our modules allows for a combinatorially large number of possible architectural layouts of our inference engine.

Original: Previous work in language-grounded reasoning tasks like VQA (Johnson et al., 2017b) or referring expression comprehension (Yu et al., 2018) uses a semantic parse tree generated from natural language to generate layouts for neural module networks.
Revised: Previous work in language-grounded reasoning tasks like VQA (Johnson et al., 2017b) or referring expression comprehension (Yu et al., 2018) uses a semantic parse tree generated from natural language to generate layouts for neural module networks.

Original: Since our inference engine π p r q relies only on the relationship label from the source domain, it lacks the rich compositional structure available in language-grounded reasoning to arrange the modular network layout.
Revised: Since our inference engine π p r q relies only on the relationship label from the source domain, it lacks the rich compositional structure available in language-grounded reasoning to arrange the modular network layout.

Original: However, generating the proper layout for neural module networks is fundamental to their ability to generalize systematically (Bahdanau et al., 2018).
Revised: However, generating the proper layout for neural module networks is fundamental to their ability to generalize systematically (Bahdanau et al., 2018).

Original: As discussed in Section 4.2, we dynamically chose between two layouts of our neural module network based on the relationship type.
Revised: As discussed in Section 4.2, we dynamically chose between two layouts of our neural module network based on the relationship type.

Original: Our rationale behind choosing these layouts comes from the knowledge of the possible relationships our engine has to reason about.
Revised: Our rationale behind choosing these layouts comes from the knowledge of the possible relationships our engine has to reason about.

Original: Each layout aligns algorithmically (Xu et al., 2020) with a reasoning algorithm for identifying the relationship across three panels.
Revised: Each layout aligns algorithmically (Xu et al., 2020) with a reasoning algorithm for identifying the relationship across three panels.

Original: The ﬁrst layout models Algorithm 1 to identify a Unary relationship like progression (not to be confused with a Unary module) between panels 1, 2 and 3.
Revised: The ﬁrst layout models Algorithm 1 to identify a Unary relationship like progression (not to be confused with a Unary module) between panels 1, 2 and 3.

Original: The second layout models Algorithm 2 to identify a Binary relationship ( OR, AND, XOR ) across panels 1 and 2 in the last panel.
Revised: The second layout models Algorithm 2 to identify a Binary relationship ( OR, AND, XOR ) across panels 1 and 2 in the last panel.

Original: Algorithm 1 To Identify Unary Relation
Revised: Algorithm 1 To Identify Unary Relation

Original: Require: Panels P 1 , P 2 , P1: D 1 Ð extractObjectsAndAttributes ( P 1 ) 2: D 2 Ð extractObjectsAndAttributes ( P 2 ) 3:
Revised: Require: Panels P 1 , P 2 , P1: D 1 Ð extractObjectsAndAttributes ( P 1 ) 2: D 2 Ð extractObjectsAndAttributes ( P 2 ) 3:

Original: D 3 Ð extractObjectsAndAttributes ( P 3 ) 4: R 12 Ð artihmeticOperation ( D 1 , D 2 ) 5: R 23 Ð artihmeticOperation ( D 2 , D 3 ) 6: Result Ð isConsistent ( R 12 , R 23 ) 7: return Result
Revised: D 3 Ð extractObjectsAndAttributes ( P 3 ) 4: R 12 Ð artihmeticOperation ( D 1 , D 2 ) 5: R 23 Ð artihmeticOperation ( D 2 , D 3 ) 6: Result Ð isConsistent ( R 12 , R 23 ) 7: return Result

Original: Algorithm 2 To Identify Binary Relation
Revised: Algorithm 2 To Identify Binary Relation

Original: Require: Panels P 1 , P 2 , P1: D 1 Ð extractObjectsAndAttributes ( P 1 )
Revised: Require: Panels P 1 , P 2 , P1: D 1 Ð extractObjectsAndAttributes ( P 1 )

Original: Ð extractObjectsAndAttributes
Revised: Ð extractObjectsAndAttributes

Original: Ð extractObjectsAndAttributes
Revised: Ð extractObjectsAndAttributes

Original: We do not explicitly train our modules to correspond to any individual functional form.
Revised: We do not explicitly train our modules to correspond to any individual functional form.

Original: Instead, we expect the module instantiations to learn the corresponding function in the reasoning step simply by learning from the classiﬁcation loss.
Revised: Instead, we expect the module instantiations to learn the corresponding function in the reasoning step simply by learning from the classiﬁcation loss.

Original: For example, in the reasoning algorithms above, a Unary module can learn extractObjectsAndAttributes() , Binary module can learn artihmeticOperation() and binaryOperation() , and Classifier module can learn isConsistent() functions.
Revised: For example, in the reasoning algorithms above, a Unary module can learn extractObjectsAndAttributes() , Binary module can learn artihmeticOperation() and binaryOperation() , and Classifier module can learn isConsistent() functions.

Original: This enables the modules to share parameters across relation functions (such as two different instantiations of binaryOperation() for AND and OR ), as well as across objects and attributes.
Revised: This enables the modules to share parameters across relation functions (such as two different instantiations of binaryOperation() for AND and OR ), as well as across objects and attributes.

Original: This also enables us to utilize general purpose modules which could potentially be arranged in several possible combinations depending on the reasoning problem, and can be easily extended beyond the abstract visual analogy setting of only four relationships to large number of possible relationships in real-world datasets like Scene Graph (Johnson et al., 2015) and CLEVR (Johnson et al., 2017a).
Revised: This also enables us to utilize general purpose modules which could potentially be arranged in several possible combinations depending on the reasoning problem, and can be easily extended beyond the abstract visual analogy setting of only four relationships to large number of possible relationships in real-world datasets like Scene Graph (Johnson et al., 2015) and CLEVR (Johnson et al., 2017a).

Original: Do algorithmically aligned engine layouts learn corresponding relations better?
Revised: Do algorithmically aligned engine layouts learn corresponding relations better?

Original: We chose two possible layouts for our modular engine based on their alignment with algorithms for identifying a Unary or Binary relation across three panels in RPM problems.
Revised: We chose two possible layouts for our modular engine based on their alignment with algorithms for identifying a Unary or Binary relation across three panels in RPM problems.

Original: We expect LayoutA (Figure 4a) to better align with Unary ( progression ) problems and Layout-B (Figure 4b) to better align with Binary ( AND, OR, XOR ) problems.
Revised: We expect LayoutA (Figure 4a) to better align with Unary ( progression ) problems and Layout-B (Figure 4b) to better align with Binary ( AND, OR, XOR ) problems.

Original: In order to verify this empirically we train each layout individually with similar training hyperparameters as the full engine.
Revised: In order to verify this empirically we train each layout individually with similar training hyperparameters as the full engine.

Original: We control the overall validation accuracy to select trained models which perform similarly across the full dataset.
Revised: We control the overall validation accuracy to select trained models which perform similarly across the full dataset.

Original: Next, we compare the test accuracy for Unary and Binary relations for the trained Layout-A and
Revised: Next, we compare the test accuracy for Unary and Binary relations for the trained Layout-A and

Original: Layout-B models in Table 3.
Revised: Layout-B models in Table 3.

Original: In practice, we found signiﬁcant stochasticity in the performance between Unary and Binary relation types across both layouts.
Revised: In practice, we found signiﬁcant stochasticity in the performance between Unary and Binary relation types across both layouts.

Original: From the observed results, we cannot conclusively say that our layouts necessarily align better with one or the other relationship problem.
Revised: From the observed results, we cannot conclusively say that our layouts necessarily align better with one or the other relationship problem.

Original: One possible explanation for this could be that there can be alternate algorithms for evaluating the same relation, which align algorithmically with these layouts.
Revised: One possible explanation for this could be that there can be alternate algorithms for evaluating the same relation, which align algorithmically with these layouts.

Original: This is similar to Xu et al.
Revised: This is similar to Xu et al.

Original: (2020)’s observation that a neural network which would have not aligned with the Bellman-Ford algorithm was found to align well with a different algorithm for the same dynamic programming problem.
Revised: (2020)’s observation that a neural network which would have not aligned with the Bellman-Ford algorithm was found to align well with a different algorithm for the same dynamic programming problem.

Original: [Table]
Revised: [Table]

Original: Does having an adaptive engine enable better analogy inference?
Revised: Does having an adaptive engine enable better analogy inference?

Original: We use an adaptive modular engine for analogy inference which chooses between two possible module net layouts at inference time.
Revised: We use an adaptive modular engine for analogy inference which chooses between two possible module net layouts at inference time.

Original: This enables our engine to compositionally reason about the underlying relation.
Revised: This enables our engine to compositionally reason about the underlying relation.

Original: We expect our adaptive setup to outperform a static layout since the engine can then chose to use a better ﬁt layout according to the mapped relationship structure.
Revised: We expect our adaptive setup to outperform a static layout since the engine can then chose to use a better ﬁt layout according to the mapped relationship structure.

Original: To verify our hypothesis, we compare our input-adaptive engine to a ﬁxed engine.
Revised: To verify our hypothesis, we compare our input-adaptive engine to a ﬁxed engine.

Original: We chose each candidate layout described in Section 4.2 as a possible inference engine.
Revised: We chose each candidate layout described in Section 4.2 as a possible inference engine.

Original: We call these ﬁxed models
Revised: We call these ﬁxed models

Original: Engine-A and Engine-B
Revised: Engine-A and Engine-B

Original: [Table]
Revised: [Table]

Original: and train them similar to our full model.
Revised: and train them similar to our full model.

Original: We then compare the test accuracy performance of our full model, which we label Engine-Full, against the test accuracy ﬁxed models.
Revised: We then compare the test accuracy performance of our full model, which we label Engine-Full, against the test accuracy ﬁxed models.

Original: [Table]
Revised: [Table]

Original: Empirically, we found that the adaptive engine was not able to outperform the ﬁxed engines, contrary to our expectation.
Revised: Empirically, we found that the adaptive engine was not able to outperform the ﬁxed engines, contrary to our expectation.

Original: We believe there are two main reasons for this: (1) Our individual engines are not sufﬁciently differentiated in their performance on their respective reasoning tasks (as shown in
Revised: We believe there are two main reasons for this: (1) Our individual engines are not sufﬁciently differentiated in their performance on their respective reasoning tasks (as shown in

Original: Table 3), so there is not a signiﬁcant performance gain in choosing one over the other, and (2) the number of possible reasoning scenarios is quite limited in the abstract visual analogy problem, and hence the adaptive engine is not able to beneﬁt from learning a large number of mixture of experts.
Revised: Table 3), so there is not a signiﬁcant performance gain in choosing one over the other, and (2) the number of possible reasoning scenarios is quite limited in the abstract visual analogy problem, and hence the adaptive engine is not able to beneﬁt from learning a large number of mixture of experts.

Original: B V ISUAL R ELATIONSHIP E NCODER A BLATIONS B. M ULTI - TASK VERSUS M ULTI - LABEL V ISUAL R ELATIONSHIP E NCODER
Revised: B V ISUAL R ELATIONSHIP E NCODER A BLATIONS B. M ULTI - TASK VERSUS M ULTI - LABEL V ISUAL R ELATIONSHIP E NCODER

Original: In our two-step structure mapping approach, we treat the structure extraction problem as a multitask learning problem where the relationship , object , attribute prediction are treated as separate learning tasks.
Revised: In our two-step structure mapping approach, we treat the structure extraction problem as a multitask learning problem where the relationship , object , attribute prediction are treated as separate learning tasks.

Original: We could have alternatively treated the structure extraction problem as a multi-label classiﬁcation problem and instead predicted the full triplet R “ t o, a, r u from our model.
Revised: We could have alternatively treated the structure extraction problem as a multi-label classiﬁcation problem and instead predicted the full triplet R “ t o, a, r u from our model.

Original: A multi-label problem could in principle learn from added network interactions between the ﬁnal classiﬁcation layer weights for the domain components.
Revised: A multi-label problem could in principle learn from added network interactions between the ﬁnal classiﬁcation layer weights for the domain components.

Original: In order to draw a comparison between both these learning approaches, we trained a multi-label visual relationship encoder which predicts the triplet R “ t o, a, r u as its output.
Revised: In order to draw a comparison between both these learning approaches, we trained a multi-label visual relationship encoder which predicts the triplet R “ t o, a, r u as its output.

Original: We replaced the l task φ
Revised: We replaced the l task φ

Original: [Equation]
Revised: [Equation]

Original: q layers in out multi-task encoder with a l shared 2 φ “ FC
Revised: q layers in out multi-task encoder with a l shared 2 φ “ FC

Original: [Equation]
Revised: [Equation]

Original: φ p l
Revised: φ p l

Original: [Equation]
Revised: [Equation]

Original: φ q layer, and the ﬁnal classiﬁers out task
Revised: φ q layer, and the ﬁnal classiﬁers out task

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: q of sizes 2, 5, and 4, with one classiﬁer out φ “
Revised: q of sizes 2, 5, and 4, with one classiﬁer out φ “

Original: FC
Revised: FC

Original: [Equation]
Revised: [Equation]

Original: q of size 11 (2+5+4).
Revised: q of size 11 (2+5+4).

Original: For training, we used exactly the same procedure as the multitask encoder (described in Section 5.1).
Revised: For training, we used exactly the same procedure as the multitask encoder (described in Section 5.1).

Original: We compared the performance of these models by evaluating there test accuracy across three different generalization splits: Novel Domain Transfer, Novel Domain: Line Type, and Novel Attribute
Revised: We compared the performance of these models by evaluating there test accuracy across three different generalization splits: Novel Domain Transfer, Novel Domain: Line Type, and Novel Attribute

Original: Value: Extrapolation.
Revised: Value: Extrapolation.

Original: Since our encoder is not trained with the candidate panels, we restrict ourselves to one training regime (Contrasting) for this comparison.
Revised: Since our encoder is not trained with the candidate panels, we restrict ourselves to one training regime (Contrasting) for this comparison.

Original: Our observations are reported below in Table 6.
Revised: Our observations are reported below in Table 6.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We found the performance of the two types of relationship encoders to be quite similar in terms of the test accuracy for predicting the relationship from the source domain panels.
Revised: We found the performance of the two types of relationship encoders to be quite similar in terms of the test accuracy for predicting the relationship from the source domain panels.

Original: In fact, the multilabel encoder was slightly better than the multi-task encoder across all generalization splits.
Revised: In fact, the multilabel encoder was slightly better than the multi-task encoder across all generalization splits.

Original: Despite this, we still went ahead with using the multi-task encoder in our two-step approach since the multitask learning network scales much better with the number of objects, attributes, and relationships in the perceptual domain.
Revised: Despite this, we still went ahead with using the multi-task encoder in our two-step approach since the multitask learning network scales much better with the number of objects, attributes, and relationships in the perceptual domain.

Original: B. M ULTI - TASK VERSUS R ELATIONSHIP -O NLY V ISUAL R ELATIONSHIP E NCODER
Revised: B. M ULTI - TASK VERSUS R ELATIONSHIP -O NLY V ISUAL R ELATIONSHIP E NCODER

Original: We also performed an ablation study on predicting only the visual relationship from the encoder and comparing it with the full-task of predicting relationship , object , attribute .
Revised: We also performed an ablation study on predicting only the visual relationship from the encoder and comparing it with the full-task of predicting relationship , object , attribute .

Original: We found that predicting only the relationship structure is competitive with a multi-task predictor in the Novel Domain Transfer regime, and since our encoder does not perform generalization across domains or candidate types this should hold for other regimes as well.
Revised: We found that predicting only the relationship structure is competitive with a multi-task predictor in the Novel Domain Transfer regime, and since our encoder does not perform generalization across domains or candidate types this should hold for other regimes as well.

Original: This does not take away from our central contribution of incorporating structure mapping into our model, rather it only highlights the importance of structure extraction as the ﬁrst-step of our model
Revised: This does not take away from our central contribution of incorporating structure mapping into our model, rather it only highlights the importance of structure extraction as the ﬁrst-step of our model

Original: [Table]
Revised: [Table]

Original: Table
Revised: Table

Original: C I MPLEMENTATION D ETAILS
Revised: C I MPLEMENTATION D ETAILS

Original: C
Revised: C

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D S UPPLEMENTARY R ESULTS D. V ISUAL R ELATIONSHIP E NCODER
Revised: D S UPPLEMENTARY R ESULTS D. V ISUAL R ELATIONSHIP E NCODER

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D. A NALOGY I NFERENCE E NGINE
Revised: D. A NALOGY I NFERENCE E NGINE

Original: [Table] [Table]
Revised: [Table] [Table]