Original: Under the ICLR workshop on and

Original: A LIGN - DEFORM - SUBTRACT : A N INTERVENTIONAL FRAMEWORK FOR EXPLAINING OBJECT DIFFERENCES
Revised: A LIGN - DEFORM - SUBTRACT : A N INTERVENTIONAL FRAMEWORK FOR EXPLAINING OBJECT DIFFERENCES

Original: Anonymous authors
Revised: c.eastwood@ed.ac.uk Li Nanbo ∗

Original: Paper under double-blind review
Revised: School of Informatics University of Edinburgh nanbo.li@ed.ac.uk Christopher K. I. Williams ckiw@inf.ed.ac.uk

Original: A BSTRACT
Revised: A BSTRACT

Original: Given two object images, how can we explain their differences in terms of the underlying object properties?
Revised: Given two object images, how can we explain their differences in terms of the underlying object properties?

Original: To address this question, we propose Align-DeformSubtract (ADS)—an interventional framework for explaining object differences.
Revised: To address this question, we propose Align-DeformSubtract (ADS)—an interventional framework for explaining object differences.

Original: By leveraging semantic alignments in image-space as counterfactual interventions on the underlying object properties, ADS iteratively quantiﬁes and removes differences in object properties.
Revised: By leveraging semantic alignments in image-space as counterfactual interventions on the underlying object properties, ADS iteratively quantiﬁes and removes differences in object properties.

Original: The result is a set of “disentangled” error measures that explain object differences in terms of their underlying properties.
Revised: The result is a set of “disentangled” error measures which explain object differences in terms of the underlying properties.

Original: Experiments on real and synthetic data illustrate the efﬁcacy of the framework.
Revised: Experiments on real and synthetic data illustrate the efﬁcacy of the framework.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Given two object images, such as those depicted in Fig.
Revised: Given two object images, such as those depicted in Fig.

Original: 1a, can we explain how the objects in those images differ in terms of their properties?
Revised: 1a, can we explain how the objects in those images differ in terms of their properties?

Original: For example, can we say that the objects depicted in
Revised: For example, can we say that the objects depicted in

Original: Fig.
Revised: Fig.

Original: 1a have similar poses but differ in terms of shape and appearance?
Revised: 1a have similar poses but differ in terms of shape and appearance?

Original: Interpretable explanations of how two object-images differ, in terms of their properties, has important applications in image retrieval (Wan et al., 2014), assessing goodness-of-ﬁt (Belin & Rubin, 1995; Gelman et al., 2004), and learning “disentangled” generative models (Kulkarni et al., 2015; Eastwood & Williams, 2018).
Revised: Interpretable explanations of how two object-images differ, in terms of their properties, has important applications in image retrieval (Wan et al., 2014), assessing goodness-of-ﬁt (Belin & Rubin, 1995; Gelman et al., 2004), and learning “disentangled” generative models (Kulkarni et al., 2015; Eastwood & Williams, 2018).

Original: One simple approach might be to just report a single-number summary of the object differences, such as mean-square-error (MSE).
Revised: One simple approach might be to just report a single-number summary of the object differences, such as the mean squared error (MSE).

Original: However, such summaries are clearly insufﬁcient as they fail to tell us how or why the objects are different.
Revised: However, such summaries are clearly insufﬁcient as they fail to tell us how or why the objects are different.

Original: Another approach would be to manually annotate (tens of) thousands of object-image pairs with their ground-truth property differences.
Revised: Another approach would be to manually annotate (tens of) thousands of object-image pairs with their ground-truth property differences.

Original: However, since 1000 images have P (1000 , 2)
Revised: However, since 1000 images have P (1000 , 2)

Original: =
Revised: =

Original: 999000 possible pairings, this seems prohibitively expensive.
Revised: 999000 possible pairings, this seems prohibitively expensive.

Original: To address this problem, we propose Align-Deform-Subtract (ADS)—an interventional framework for explaining object differences.
Revised: To address this problem, we propose Align-Deform-Subtract (ADS)—an interventional framework for explaining object differences.

Original: ADS takes a causal perspective on the above problem, asking what must we do to object 1 (source) so that it looks like object 2 (target) , or, equivalently, how must we change or intervene upon the properties of the source object such that it looks like the target object?
Revised: ADS takes a causal perspective on the above problem, asking what must we do to object 1 (source) so that it looks like object 2 (target) , or, equivalently, how must we change or intervene upon the properties of the source object such that it looks like the target object?

Original: To intervene without an explicit object model, ADS leverages the image-space transformations of semantic alignment networks (Rocco et al., 2018a) as counterfactual interventions on the underlying object properties.
Revised: To intervene without an explicit object model, ADS leverages the image-space transformations of semantic alignment networks (Rocco et al., 2018a) as counterfactual interventions on the underlying object properties.

Original: By iteratively intervening on the source-object properties, setting them to be those of the target, ADS both quantiﬁes and removes object-property differences.
Revised: By iteratively intervening on the source-object properties, setting them to be those of the target, ADS both quantiﬁes and removes object-property differences.

Original: The result is a set of disentangled error measures explaining how the objects differ in terms of their underlying properties.
Revised: The result is a set of disentangled error measures explaining how the objects differ in terms of their underlying properties.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  B ACKGROUND : S EMANTIC ALIGNMENT NETWORKS
Revised:  B ACKGROUND : S EMANTIC ALIGNMENT NETWORKS

Original: Estimating correspondences between images is an important problem in computer vision (Hartley & Zisserman, 2003).
Revised: Estimating correspondences between images is an important problem in computer vision (Hartley & Zisserman, 2003).

Original: The classical approach has been to combine several different procedures like detecting and matching local features (e.g. SIFT, Lowe 2004), pruning incorrect matches using local geometric constraints (Schmid & Mohr, 1997), and estimating a global geometric transformation (Fischler & Bolles, 1981).
Revised: The classical approach has been to combine several different procedures like detecting and matching local features (e.g. SIFT, Lowe 2004), pruning incorrect matches using local geometric constraints (Schmid & Mohr, 1997), and estimating a global geometric transformation (Fischler & Bolles, 1981).

Original: However, recent works have achieved state-of-the-art results using convolutional neural networks (CNNs) as an end-to-end solution for semantic alignment (Rocco et al., 2018a;b).
Revised: However, recent works have achieved state-of-the-art results using convolutional neural networks (CNNs) as an end-to-end solution for semantic alignment (Rocco et al., 2018a;b).

Original: In particular, Rocco et al.
Revised: In particular, Rocco et al.

Original: (2018a) introduce a CNN architecture which takes as input two images—a source image X s and a target image X t —and outputs the parameters of a geometric transformation which aligns them.
Revised: (2018a) introduce a CNN architecture which takes as input two images—a source image X s and a target image X t —and outputs the parameters of a geometric transformation which aligns them.

Original: To best alignthe source and target images, Rocco et al.
Revised: To best align source and target images, Rocco et al.

Original: use the following two-stage procedure:
Revised: use the following two-stage procedure:

Original: Afﬁne alignment: A CNN-based alignment network f aff estimates the parameters θ aff of the afﬁne transformation T aff 
Revised: Afﬁne alignment: A CNN-based alignment network f aff estimates the parameters θ aff of the afﬁne transformation T aff best-

Original: best-aligning the source and target images.
Revised: aligning the source and target images.

Original: More concretely, θ aff = f aff ( X s , X t ) and T aff =  minτ
Revised: More concretely, θ aff = f aff ( X s , X t ) and T aff = arg min

Original:  ∈T afﬁne D keypoints ( τ ( X s ) , X t ) , with D keypoints being some measure of the “distance” between keypoints.
Revised: τ ∈T afﬁne D keypoints ( τ ( X s ) , X t ) , with D keypoints being some measure of the “distance” between keypoints.

Original: As depicted in the second column of Fig.
Revised: As depicted in the second column of Fig.

Original: 2, T aff is then applied to X s .
Revised: 2, T aff is then applied to X s .

Original: Fig.
Revised: Fig.

Original: 10 of Appendix D depicts these keypoints andtheir reduced distances after alignment.
Revised: 10 of Appendix D depicts keypoint “distances” before and after alignment.

Original: Thin-plate spline (TPS) alignment:
Revised: Thin-plate spline (TPS) alignment:

Original: To reﬁne the rough afﬁne alignment, a second network f
Revised: To reﬁne the rough afﬁne alignment, a second network f

Original: tps is used to estimate the parameters θ tps of the TPS transformation T tps best-aligning the afﬁnealigned source and target images,i.e. T tps =
Revised: tps is used to estimate the parameters θ tps of the TPS transformation T tps best-aligning the afﬁne-aligned source and target images,

Original:  min τ ∈T TPS D keypoints ( τ ( T aff ( X s )) , X t ) , and θ tps=
Revised: i.e. T tps = arg min τ ∈T TPS D keypoints ( τ ( T aff ( X s )) , X t ) , and θ tps

Revised: =

Original: f tps ( T aff ( X s ) , X t ) .
Revised: f tps ( T aff ( X s ) , X t ) .

Original: T tps is then applied to T aff ( X s ) , as depicted in the third column of Fig.
Revised: T tps is then applied to T aff ( X s ) , as depicted in the third column of Fig.

Original: Importantly, this method is fully trainable from synthetic imagery without the need  manual annotations, and generalizes to unseen images.
Revised: Importantly, this method is fully trainable from synthetic transformations without the need for manual annotations, and generalizes to unseen images.

Original: See Rocco et al.
Revised: See Rocco et al.

Original: (2018a)
Revised: (2018a)

Original: for more details.
Revised: for more details.

Original:  I NTERVENTIONAL FRAMEWORK
Revised:  I NTERVENTIONAL FRAMEWORK

Original: We now describe ADS—our interventional framework for explaining object differences.
Revised: We now describe ADS—our interventional framework for explaining object differences.

Original: As depicted in Fig.
Revised: As depicted in Fig.

Original: 2, source-object properties are iteratively aligned with that of the target in a 3-step procedure, with each step both quantifying and removing a particular object-property difference.
Revised: 2, source-object properties are iteratively aligned with that of the target in a 3-step procedure, with each step both quantifying and removing a particular object-property difference.

Original: By viewing these image-space transformations as (counterfactual) interventions on the source-object properties, we can precisely describe our framework and its assumptions using the language of causal inference.
Revised: By viewing these image-space transformations as (counterfactual) interventions on the source-object properties, we can precisely describe our framework and its assumptions using the language of causal inference.

Original: 3. A LIGN
Revised: 3. A LIGN

Original: First, we leverage a pretrained afﬁne alignment network f aff (see § 2) to obtain the parameters of the afﬁne transformation T aff best-aligning our source and target images, i.e. θ aff = f aff ( X s , X t ) .
Revised: First, we leverage a pretrained afﬁne alignment network f aff (see § 2) to obtain the parameters of the afﬁne transformation T aff best-aligning our source and target images, i.e. θ aff = f aff ( X s , X t ) .

Original: We then view T aff ( X s ) —depicted in the second image of Fig.
Revised: We then view T aff ( X s ) —depicted in the second column of Fig.

Original: 2—as a counterfactual intervention on the pose of the source object, setting it to be that of the target object.
Revised: 2—as a counterfactual intervention on the pose of the source object, setting it to be that of the target object.

Original: We denote this counterfactual— “what if the source object had the pose of the target object?” —as X C ; do ( Pose := pose t ) s , with C a structural causal model (SCM, Pearl 2009) consistent with Fig.
Revised: We denote this counterfactual— “what if the source object had the pose of the target object?” —as X C ; do ( Pose := pose t ) s , with C a structural causal model (SCM, Pearl 2009) consistent with Fig.

Original: 1b and X s a slight abuse of notation to denote X =
Revised: 1b and X s a slight abuse of notation to denote X =

Original: x s (i.e. conditioning on the factually-observed source object).
Revised: x s (i.e. conditioning on the factually-observed source object).

Original: Armed with this causal perspective on aligning transforms, we now describe the two key insights which allow us explain object-pose differences using T aff :
Revised: Armed with this causal perspective on aligning transforms, we now describe the two key insights which allow us explain object-pose differences using T aff :

Original: T aff quantiﬁes the pose differences .
Revised: T aff quantiﬁes the pose differences .

Original: Since T aff ( X s ) ≈ X C ; do ( Pose :=
Revised: Since T aff ( X s ) ≈ X C ; do ( Pose :=

Original: pose t ) s , we note that || T aff || ∝ ∆ pose ( X s , X t ) , where || · || is some measure of transformation magnitude.
Revised: pose t ) s , we note that || T aff || ∝ ∆ pose ( X s , X t ) , where || · || is some measure of transformation magnitude and ∆ pose the groundtruth pose difference.

Original: Intuitively, || T aff || can be seen as the “effort” of transforming the source-object pose into that of the target.
Revised: Intuitively, || T aff || is the “effort” of transforming the source-object pose into that of the target.

Original: Furthermore, by decomposing T aff into scale, translation, and rotation components, we can get more ﬁne-grained measures ofthe object differences—namely  scaling  s , translation ∆ t and in-plane rotation∆ θ .
Revised: Furthermore, by decomposing T aff into scale, translation, and rotation components, we can get more ﬁne-grained measures of object differences—namely estimated scaling ˆ s , translation ˆ t and in-plane rotation

Original:  See Appendix A for details,with Table 2 summarising our measures.
Revised: ˆ θ . See Appendix A for details, Table 2 for a summary.

Original: T aff removes the pose differences .
Revised: T aff removes the pose differences .

Original: Since T aff ( X s ) ≈ X C ; do ( Pose :=
Revised: Since T aff ( X s ) ≈ X C ; do ( Pose :=

Original: pose t ) s , it follows that 
Revised: pose t ) s , it follows that ∆

Original: ∆ pose ( T aff ( X s )
Revised: pose ( T aff ( X s )

Original: , X t )
Revised: X t )

Original: ≈ ∆ pose ( X C ; do ( Pose := pose t ) s
Revised: ≈ ∆ pose ( X C ; do ( Pose := pose t ) s

Original: , X t )
Revised: , X t )

Original: = 0 .
Revised: = 0 .

Original: This removal or explaining-away of pose errors is critical for subsequent steps which assume that no pose differences remain.
Revised: This removal or explaining-away of pose errors is critical for subsequent steps which assume that no pose differences remain.

Original: 3. D EFORM
Revised: 3. D EFORM

Original: Next, we leverage a pretrained TPS alignment network f tps (see § 2) to obtain the parameters of the TPS transformation T tps best-aligning our (afﬁne-aligned) source and target images, i.e. θ tps =
Revised: Next, we leverage a pretrained TPS alignment network f tps (see § 2) to obtain the parameters of the TPS transformation T tps best-aligning our (afﬁne-aligned) source and target images, i.e. θ tps =

Original: f tps ( T aff ( X s ) , X t ) .
Revised: f tps ( T aff ( X s ) , X t ) .

Original: We then view T tps ( T aff ( X s )) —depicted in the third image of Fig.
Revised: We then view T tps ( T aff ( X s )) —depicted in the third column of Fig.

Original: 2—as a further (counterfactual) intervention on the source object, setting its shape to be that of the target object.
Revised: 2—as a further (counterfactual) intervention on the source object, setting its shape to be that of the target object.

Original: We denote this counterfactual— “what if the source object also had the shape of the target object?” —as 
Revised: We denote this counterfactual— “what if the source object also had the shape of the target object?” —as X C ; do ( Pose :=

Original: X C ; do ( Pose := pose t , Shape := shape t ) s .
Revised: pose t , Shape := shape t ) s .

Original: As before, this causal perspective on aligning transforms makes clear two key insights which allow us to explain shape differences with T tps :
Revised: As before, this causal perspective on aligning transforms makes clear two key insights which allow us to explain shape differences with T tps :

Original: T tpsquantiﬁes
Revised: T tps

Original:  the shape differences .
Revised: quantiﬁes the shape differences .

Original: Since T tps ( T aff ( X s ))
Revised: Since T tps ( T aff ( X s ))

Original: ≈ X C ; do ( Pose := pose t , Shape := shape t )
Revised: ≈ X C ; do ( Pose := pose t , Shape := shape t )

Original: s , we note that || T
Revised: s , we note that || T

Original: tps || ∝ ∆ shape ( X s , X t ) , where || T tps || is seen as the “effort” of transforming the source-object shape into that of the target (see Appendix A for details).
Revised: tps || ∝ ∆ shape ( X s , X t ) , where || T tps || is seen as the “effort” of transforming the source-object shape into that of the target (see Appendix A for details).

Original: This gives us a measure of  shape or deformation difference ∆ d .
Revised: This gives us an estimate of the shape or deformation difference ˆ d .

Original: Note that, if we had not removed pose differences in the previous step, || T tps || would be an “entangled” measure of both pose and shape differences.
Revised: Note that, if we had not removed pose differences in the previous step, || T tps || would be an “entangled” measure of both pose and shape differences.

Original: T tps removes the shape differences .
Revised: T tps removes the shape differences .

Original: Since T tps ( T aff ( X s ))
Revised: Since T tps ( T aff ( X s ))

Original: ≈ X C ; do ( Pose := pose t , Shape := shape t )
Revised: ≈ X C ; do ( Pose := pose t , Shape := shape t )

Original: s , it follows that ∆ shape ( T tps ( T aff ( X s )) , X t )
Revised: s , it follows that ∆ shape ( T tps ( T aff ( X s )) , X t )

Original: ≈ ∆ shape ( X C ; do ( Pose := pose t , Shape := shape t ) s , X t )
Revised: ≈ ∆ shape ( X C ; do ( Pose := pose t , Shape := shape t ) s , X t )

Original: = 0 .
Revised: = 0 .

Original: This removal or explaining-away of shape errors is critical for the ﬁnal step which assumes no pose or shape differences remain.
Revised: This removal or explaining-away of shape errors is critical for the ﬁnal step which assumes no pose or shape differences remain.

Original: 3. S UBTRACT
Revised: 3. S UBTRACT

Original: Finally, after removing pose and shape differences using T aff and T tps respectively, we seek to quantify appearance differences.
Revised: Finally, after removing pose and shape differences using T aff and T tps respectively, we seek to quantify appearance differences.

Original: Assuming the causal model of Fig.
Revised: Assuming the causal model of Fig.

Original: 1b, only appearance and background differences remain.
Revised: 1b, only appearance and background differences remain.

Original: To remove the latter, we make use of an estimated source-object mask ˜ M s , obtained from a pretrained instance segmentation network (He et al., 2017) or otherwise.
Revised: To remove the latter, we make use of an estimated source-object mask ˜ M s , obtained from a pretrained instance segmentation network (He et al., 2017) or otherwise.

Original: 1 Writing this background removal more formally, we have ∆ backgr.
Revised: 1 Writing this background removal more formally, we have ∆ backgr.

Original: ( ˜ M s (cid:12) T tps ( T aff ( X s )) , ˜ M s (cid:12) X t ))
Revised: ( ˜ M s (cid:12) T tps ( T aff ( X s )) , ˜ M s (cid:12) X t ))

Original: ≈ ∆ backgr.
Revised: ≈ ∆ backgr.

Original: ( X C ; do ( Pose := pose t , Shape := shape t , Backgr.
Revised: ( X C ; do ( Pose := pose t , Shape := shape t , Backgr.

Original: :=0) s , X C ; do ( Backgr.
Revised: :=0) s , X C ; do ( Backgr.

Original: =0) t )
Revised: =0) t )

Original: = 0 , where (cid:12) denotes elementwise multiplication.
Revised: = 0 , where (cid:12) denotes elementwise multiplication.

Original: We can then quantify the appearance difference ∆ a using a masked MSE between the afﬁne- and TPS-aligned source image T tps ( T aff ( X s )) and the target image X t , i.e.
Revised: We can then get an estimated appearance difference ˆ a using a masked MSE between the afﬁne- and TPS-aligned source image T tps ( T aff ( X s )) and the target image X t , i.e.

Original: ∆ a = MSE ( ˜ M s ( cid:12) T tps ( T aff ( X s )) , ˜ M s (cid:12) X t ) .
Revised: ˆ a = MSE ( ˜ M s (cid:12) T tps ( T aff ( X s )) , ˜ M s (cid:12) X t ) .

Original: The penultimate image of Fig.
Revised: The penultimate column of Fig.

Original: 2 provides a visualisation of ∆ a before averaging over unmasked foreground pixels, using a square-error heatmap.
Revised: 2 provides a visualisation of ˆ a before averaging over unmasked foreground pixels, using a square-error heatmap.

Original: Note that the standard MSE—MSE ( X s , X t ) — entangles errors arising from differences in pose, shape, appearance and background, while our measures disentangle these errors through iterative interventions on the source object.
Revised: Note that the standard MSE—MSE ( X s , X t ) — entangles errors arising from differences in pose, shape, appearance and background, while our measures disentangle these errors through iterative interventions on the source object.

Original: This crucial difference is illustrated in Fig.
Revised: This crucial difference is illustrated in Fig.

Original: 7 of Appendix C.1.
Revised: 7 of Appendix C.1.

Original:  E XPERIMENTS 4. R EAL DATA : QUALITATIVE EVALUATION
Revised:  E XPERIMENTS 4. R EAL DATA : QUALITATIVE EVALUATION

Original: We ﬁrst evaluate our measures on real data from the Proposal Flow (PF) dataset (Ham et al., 2017).
Revised: We ﬁrst evaluate our measures on real data from the Proposal Flow (PF) dataset (Ham et al., 2017).

Original: As the ground-truth (GT) property differences are not available, this evaluation is qualitative , i.e. one must decide, by visual inspection, how well our measures explain the object-property differences.
Revised: As the ground-truth (GT) property differences are not available, this evaluation is qualitative , i.e. one must decide, by visual inspection, how well our measures explain the object-property differences

Original: [Figure]
Revised: [Figure]

Original: Results.
Revised: Results.

Original: By comparing the relative magnitude of our measures across the 3 examples in Fig.
Revised: By comparing the relative magnitude of our measures across the 3 examples in Fig.

Original: 3, we see that our measures accurately capture the object-property differences.
Revised: 3, we see that our measures accurately capture the object-property differences.

Original: For example, letting  s ( a ) denote our predicted scaling factor  s in Fig.
Revised: For example, letting ˆ s ( a ) denote our estimated scaling factor ˆ s in Fig.

Original: 3a, we see that  s ( c ) >  s ( a ) >  s ( b ) , which is visually consistent with source-object scalings.
Revised: 3a, we see that ˆ s ( c ) > ˆ s ( a ) > ˆ s ( b ) , which is visually consistent with source-object scalings.

Original: Likewise, ∆ t ( a ) > ∆ t ( c ) > ∆ t ( b ) is consistent with the source-object translations, as ∆ θ ( a )  >
Revised: Likewise, ˆ t ( a ) > ˆ t ( c ) > ˆ t ( b ) is consistent with the source-object translations, as | ˆ θ ( a ) | >

Original: ∆ θ ( b )  > ∆ θ ( c )  is with the rotations.
Revised: | ˆ θ ( b ) | > | ˆ θ ( c ) | is with the rotation magnitudes.

Original: 4. S YNTHETIC DATA : QUANTITATIVE EVALUATION
Revised: 4. S YNTHETIC DATA : QUANTITATIVE EVALUATION

Original: We next evaluate our measures on synthetic data where we have the GT object differences.
Revised: We next evaluate our measures on synthetic data where we have the GT object differences.

Original: To do so, we use the teapot model of Moreno et al.
Revised: To do so, we use the teapot model of Moreno et al.

Original: (2016) to generate 2000 images of teapots, randomly sampling pose, shape, appearance and background parameters.
Revised: (2016) to generate 2000 images of teapots, randomly sampling pose, shape, appearance and background parameters.

Original: We then randomly pair these images, resulting in 1000 source-target pairs, and calculate the GT differences in pose, shape and appearance.
Revised: We then randomly pair these images, resulting in 1000 source-target pairs, and calculate the GT differences in pose, shape and appearance.

Original: Appendix B details the generation process and subsequent calculation of GT “distances” between teapot properties, while also depicting some example images (see Fig. 5).
Revised: Appendix B details the generation process and subsequent calculation of GT “distances” between teapot properties, while also depicting some example images (see Fig. 5).

Original: Next, we pass all sourcetarget pairs { ( X ( i ) s , X ( i ) t ) } 1000 i =1 through afﬁne and TPS alignment networks—pretrained on PF—to get θ ( i ) aff
Revised: Next, we pass all sourcetarget pairs { ( X ( i ) s , X ( i ) t ) } 1000 i =1 through afﬁne and TPS alignment networks—pretrained on PF—to get θ ( i ) aff

Original: = f aff ( X ( i ) s , X ( i ) t ) and θ ( i ) tps
Revised: = f aff ( X ( i ) s , X ( i ) t ) and θ ( i ) tps

Original: = f aff ( T ( i ) aff ( X ( i ) s ) , X ( i ) t ) (see Fig.
Revised: = f aff ( T ( i ) aff ( X ( i ) s ) , X ( i ) t ) (see Fig.

Original: 8 of Appendix C.2 for example transformations).
Revised: 8 of Appendix C.2 for example transformations).

Original: Finally, we use θ ( i ) aff and θ ( i ) tps to calculate our pose (  s x ,  s y , ∆ t x , ∆ t y , ∆ θ ), shape ( ∆ d ), and appearance ( ∆ a ) measures for each source-target pair (see Appendix A).
Revised: Finally, we use θ ( i ) aff and θ ( i ) tps to calculate our pose ( ˆ s x , ˆ s y , ˆ t x , ˆ t y , ˆ θ ), shape ( ˆ d ), and appearance ( ˆ a ) measures for each source-target pair (see Appendix A).

Original: Results.
Revised: Results.

Original: Each panel in Fig.
Revised: Each panel in Fig.

Original: 4 comprises two plots.
Revised: 4 comprises two plots.

Original: The blue dots show our predicted object-property difference (y-axis) against the GT object-property difference (x-axis).
Revised: The blue dots show our predicted object-property difference (y-axis) against the GT object-property difference (x-axis).

Original: The yellow dots show the
Revised: The yellow dots show the

Original: MSE for a particular object-property difference—as MSE is also affected by differences in all other object properties, it is poorly correlated with the particular object-property difference against which it is plotted.
Revised: MSE for a particular object-property difference—as MSE is also affected by differences in all other object properties, it is poorly correlated with the particular object-property difference against which it is plotted.

Original: For example, the bottom-left panel plots the GT difference in x -position,∆ t GTx , against our prediction ∆ t x (blue dots) and the MSE (yellow dots).
Revised: For example, the bottom-left panel plots the GT difference in x -position, t x , against our prediction ˆ t x (blue dots) and the MSE (yellow dots).

Original: Evidently, the GT difference has a much stronger correlation with our prediction than it does with the MSE—as made clear by the correlation coefﬁcients in Table 1.
Revised: Evidently, the GT difference has a much stronger correlation with our prediction than it does with the MSE—as made clear by the correlation coefﬁcients in Table 1.

Original: Importantly, the GT appearance difference∆ aGT has a much stronger correlation with our prediction ∆ a than it does with the MSE.
Revised: Importantly, the GT appearance difference a has a much stronger correlation with our prediction ˆ a than it does with the MSE.

Original: This improvement is explained by Fig. 9—by ﬁrst intervening on the pose and shape of the source object such that they match those of the target, our measure gains invariance to pose and shape differences.
Revised: This improvement is explained by Fig. 9—by ﬁrst intervening on the pose and shape of the source object such that they match those of the target, our measure gains invariance to pose and shape differences.

Original: Overall, most of our measures show a strong correlation with the corresponding GT object-property difference, indicating that they accurately capture the mismatch for a particular object property while remaining relatively invariant to mismatches in other object properties.
Revised: Overall, most of our measures show a strong correlation with the corresponding GT object-property difference, indicating that they accurately capture the mismatch for a particular object property while remaining relatively invariant to mismatches in other object properties.

Original: This conﬁrms that our ADS framework can indeed disentangle errors arising from different object-property mismatches, and, as a result, provide a good explanation of how the source and target objects differ.
Revised: This conﬁrms that our ADS framework can indeed disentangle errors arising from different object-property mismatches, and, as a result, provide a good explanation of how the source and target objects differ.

Original: One notable exception is shape (∆ d ).
Revised: One notable exception is shape ( d ).

Original: We attribute this to the TPS network of Rocco et al.
Revised: We attribute this to the TPS network of Rocco et al.

Original: (2018a), which often performs quite poorly.
Revised: (2018a), which often performs quite poorly.

Original: We hope that this will be resolved by future improvements in semantic alignment networks.
Revised: We hope that this will be resolved by future improvements in semantic alignment networks.

Original: Under review at the ICLR 2022 workshop on Objects, Structure and Causality 
Revised: Published at the ICLR 2022 workshop on Objects, Structure and Causality 0. 1.0 d 0. M e a s u r e V a l u e

Revised: Shape ideal MSE

Revised: Ours ( d ) 0.

Original:  [Figure]
Revised: 1.0 s x M e a s u r e V a l u e [Figure]

Revised: MSE

Revised: [Figure]

Revised: ( s

Revised: [Figure]

Revised: 1.0 s y

Revised: M e a s u r e

Revised: V a l u e

Revised: [Figure]

Revised: _y

Revised: [Figure]

Revised: MSE

Revised: [Figure]

Revised: ( s

Revised: [Figure]

Revised: M e a s u r e

Revised: V a l u e

Revised: Rotation ideal MSE

Revised: Ours ( ) 0.

Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Use cases.
Revised: Related work. Belongie et al. quantiﬁed object shape differences using the magnitude of the aligning transform along with the sum of the matching errors between correspondence points. In self-supervised learning, Von K ¨ ugelgen et al.

Original:  Potential use cases for interpretable, ﬁne-grained explanations of object differences include: (i) Image retrieval: our measures could permit the retrieval of object images that are most similar in pose, shape or appearance; (ii) Assessing
Revised: (2021) and Mitrovic et al. (2021) recently viewed image transformations or augmentations as counterfactual interventions on underlying properties. Use cases. Potential use cases for interpretable, ﬁne-grained explanations of object differences include: (i) Image retrieval: our measures could permit the retrieval of object images that are most similar in pose, shape or appearance; (ii) Assessing

Original: goodness-of-ﬁt: If the source image is the output or rendering of an object model, then our measures elucidate how and why the model is wrong.
Revised: goodness-of-ﬁt: If the source image is the output or rendering of an object model, then our measures elucidate how and why the model is wrong.

Original: While “all models are wrong” (Box, 1976) in practice, these measures at least allow us to determine if the model is useful (Belin & Rubin, 1995, pg. 165), e.g. manufacturing applications may care exclusively about object shape; and (iii) Learning disentangled generative models : Disentangled errors may aid the learning of disentangled generative models (Kulkarni et al., 2015; Eastwood & Williams, 2018) in which pose, shape and appearance are represented separately.
Revised: While “all models are wrong” (Box, 1976) in practice, these measures at least allow us to determine if the model is useful (Belin & Rubin, 1995, pg. 165), e.g. manufacturing applications may care exclusively about object shape; and (iii) Learning disentangled generative models : Disentangled errors may aid the learning of disentangled generative models (Kulkarni et al., 2015) in which pose, shape and appearance are represented separately.

Original: Limitations.
Revised: Limitations.

Original: We make several simplifying assumptions that may be seen as limitations: (i) Simple causal model:
Revised: We make several simplifying assumptions that may be seen as limitations: (i) Simple causal model:

Original: In reality, Fig.
Revised: In reality, Fig.

Original: 1b may not always be correct, e.g. there could be other object properties, or multiple objects in one image; (ii) Alignment networks work well: Our framework and its constituent measures rely on good afﬁne and TPS alignment.
Revised: 1b may not always be correct, e.g. there could be other object properties, or multiple objects in one image; (ii) Alignment networks work well: Our framework and its constituent measures rely on good (rigid and non-rigid) alignment.

Original: While this may not be true for all datasets, such networks will only improve in the future (semantic alignment is a popular topic in the computer vision community); (iii) Pose primarily differs in the image plane:
Revised: While this may not be true for all datasets, such networks will only improve in the future (semantic alignment is a popular topic in the computer vision community); (iii) Pose primarily differs in the image plane:

Original: To handle large 3D pose errors about any axis, future work may look to incorporate explicit object models.
Revised: To handle large 3D pose errors about any axis, future work may look to incorporate explicit object models.

Original: Conclusion.
Revised: Conclusion.

Original: We have presented ADS—an interventional framework for explaining object differences.
Revised: We have presented ADS—an interventional framework for explaining object differences.

Original: ADS leverages the image-space transformations of semantic alignment networks to align and deform the source-object image such that it matches the target, and casts these image-space alignments as counterfactual interventions on the underlying object properties.
Revised: ADS leverages the image-space transformations of semantic alignment networks to align and deform the source-object image such that it matches the target, and casts these image-space alignments as counterfactual interventions on the underlying object properties.

Original: This allows ADS to disentangle errors arising from different object-property mismatches, ultimately providing ﬁnegrained explanations of object differences in terms of their underlying properties.
Revised: This allows ADS to disentangle errors arising from different object-property mismatches, ultimately providing ﬁnegrained explanations of object differences in terms of their underlying properties.

Original: We believe that
Revised: We believe that

Original: ADS constitutes an interesting and novel framework, even if some components need improvement before it can be deployed in the real-world. 
Revised: ADS constitutes an interesting and novel framework, even if some components need improvement before it can be deployed in the real-world. A CKNOWLEDGEMENTS

Revised: We thank Robert Fisher, Julius von K¨ugelgen and Andrew Zisserman for helpful discussions and comments.

Revised: We also thank Pol Moreno for help with the generation of the teapot dataset.

Original: A C ALCULATION OF OUR MEASURES
Revised: A C ALCULATION OF OUR MEASURES

Original: We now detail the calculation of our measures.
Revised: We now detail the calculation of our measures.

Original: Similar to Belongie et al.
Revised: Similar to Belongie et al.

Original: (2002), we leverage the magnitude of the aligning transform to quantify object-property dissimilarity, viewing this magnitude as the “effort” of transforming the source-object property into that of the target.
Revised: (2002), we leverage the magnitude of the aligning transform to quantify object-property dissimilarity, viewing this magnitude as the “effort” of transforming the source-object property into that of the target.

Original: Afﬁne transform.
Revised: Afﬁne transform.

Original: For the afﬁne transform, we have:
Revised: For the afﬁne transform, we have:

Original: [Equation]
Revised: [Equation]

Original: where X grid s , X gridaff ∈ R 2 × N ( N is the number of pixels) denote the grid coordinates of the source and transformed-source images respectively.
Revised: where X grid s , X gridaff ∈ R 2 × N ( N is the number of pixels) denote the grid coordinates of the source and transformed-source images respectively.

Original: Following Rocco et al.
Revised: Following Rocco et al.

Original: (2018a), the six afﬁne parameters θ aff =
Revised: (2018a), the six afﬁne parameters θ aff =

Original: [ r 11 , r 12 , r 21 , r 22 , ∆ t x , ∆ t y ] are the output of a pretrained afﬁne-alignment network f aff (see § 2).
Revised: [ r 11 , r 12 , r 21 , r 22 , ˆ t x , ˆ t y ] are the output of a pretrained afﬁne-alignment network f aff (see § 2).

Original: To get more ﬁne-grained measures, we further decompose this afﬁne transform as:
Revised: To get more ﬁne-grained measures, we further decompose this afﬁne transform as:

Original: [Equation]
Revised: [Equation]

Original: where ∆ θ is a measure of the in-plane rotation difference between the source and target objects, ( s x ,  s y ) are measures of the scaling factors , (∆ t x , ∆ t y ) are measures of the translation differences, and b is a measure of the shear differences.
Revised: where ˆ θ is a measure of the in-plane rotation difference between the source and target objects, (ˆ s x , ˆ s y ) are measures of the scaling factors , (ˆ t x , ˆ t y ) are measures of the translation differences along the x (horizontal) and y (vertical) axis respectively, and b is a measure of the shear differences.

Original: To ease interpretation, we often a report single measure for the scaling factor  s =  s x  s y , which represents the factor by which the source-object area is scaled.
Revised: To ease interpretation, we often a report single measure for the scaling factor ˆ s = ˆ s x ˆ s y , which represents the factor by which the source-object area is scaled.

Original: We also report a single translation difference ∆ t = (cid:112) (∆ tx ) 2  + (∆ ty ) 2  .
Revised: We also report a single translation difference ˆ t = (cid:113) ˆ t 2 x + ˆ t 2 y .

Original: TPS deformation.
Revised: TPS deformation.

Original: For the TPS deformation, we again follow Rocco et al.
Revised: For the TPS deformation, we again follow Rocco et al.

Original: (2018a), with the pretrained TPS-alignment network outputting the parameters of a TPS deformation on a 3 × 3 grid (a set of 9 points).
Revised: (2018a), with the pretrained TPS-alignment network outputting the parameters of a TPS deformation on a 3 × 3 grid (a set of 9 points).

Original: That is, θ tps ∈ R 2 × 9 is a matrix containing the 2D coordinates of each of the 9 grid points.
Revised: That is, θ tps ∈ R 2 × 9 is a matrix containing the 2D coordinates of each of the 9 grid points.

Original: We denote the TPS-transformed image grid as X gridtps ∈ R 2 × N ( N is the number of pixels) and the magnitude of this transformation or deformation ∆ d (which we use as our shape measure).
Revised: We denote the TPS-transformed image grid as X gridtps ∈ R 2 × N ( N is the number of pixels) and the magnitude of this transformation or deformation ˆ d (which we use as our shape measure).

Original: Speciﬁcally, ∆ d is the average distance moved by X gridtps after normalizing with the diagonal imagegrid size or “distance” ( 2 √ [Equation]
Revised: Speciﬁcally, ˆ d is the average distance moved by X gridtps after normalising with the diagonal image-grid size or “distance” ( 2 √ 2 ):

Original: [Equation]
Revised: [Equation]

Original: (3) where || · || 2 , 1 represents the L 2 , 1 norm of a matrix.
Revised: where || · || 2 , 1 represents the L 2 , 1 norm of a matrix.

Original: Appearance.
Revised: Appearance.

Original: Finally, we measure the appearance difference ∆ a using a source-masked MSE, i.e. ∆ a = MSE ( ˜ M s (cid:12) T aff ( X s ) , ˜ M s (cid:12) X t ) , where ˜ M s denotes the source-object mask.
Revised: Finally, we measure the appearance difference ˆ a using a source-masked MSE, i.e. ˆ a = MSE ( ˜ M s (cid:12) T aff ( X s ) , ˜ M s (cid:12) X t ) , where ˜ M s denotes the source-object mask.

Original: Note that ∆ a ∈ [0 , 1] since the RGB pixel values are in [0 , 1] 3 .
Revised: Note that ˆ a ∈ [0 , 1] since the RGB pixel values are in [0 , 1] 3 .

Original: [Table]
Revised: [Table]

Original: B T EAPOT DATASET
Revised: B T EAPOT DATASET

Original: Overview The teapot dataset contains 1000 data samples, where each sample consists of a source image (with its associated mask) and a target image—see Fig.
Revised: Overview The teapot dataset contains 1000 data samples, where each sample consists of a source image (with its associated mask) and a target image—see Fig.

Original: 5 below.
Revised: 5 below.

Revised: [Figure]

Original: Generation.
Revised: Generation.

Original: The generation of each teapot is controlled by a unique 7D vector, with each dimension describing a different property.
Revised: The generation of each teapot is controlled by a unique 7D vector, with each dimension describing a different property.

Original: Note that subscripts x and y used to indicate an item measured along the x (horizontal) and y (vertical) axis respectively.
Revised: Note that subscripts x and y used to indicate an item measured along the x (horizontal) and y (vertical) axis respectively.

Original: Thus, we store the 2000 7D vectors as object descriptors for the 2000 simulated teapots (1000 source + 1000 target).[Figure]
Revised: Thus, we store the 2000 7D vectors as object descriptors for the 2000 simulated teapots (1000 source + 1000 target).

Original: Simulated shape deformations.
Revised: Simulated shape deformations.

Original: To simulate shape deformations while retaining access to a GT shape difference, we ﬁt a PCA model to a set of different teapot mesh models, each consisting of 8163 vertices.
Revised: To simulate shape deformations while retaining access to a GT shape difference, we ﬁt a PCA model to a set of different teapot mesh models, each consisting of 8163 vertices.

Original: We then use the top 10 principal components to describe a teapot mesh, which allows us to deform the shape by manipulating this 10D latent code.
Revised: We then use the top 10 principal components to describe a teapot mesh, which allows us to deform the shape by manipulating this 10D latent code.

Original: To simplify our experiments, we select a single dimension to manipulate while keeping ﬁxed the other 9.
Revised: To simplify our experiments, we select a single dimension to manipulate while keeping ﬁxed the other 9.

Original: As a result, the shape deformation of a teapot can be described using a single scalar d ∈ R .
Revised: As a result, the shape deformation of a teapot can be described using a single scalar d ∈ R .

Original: We thus generate 2000 different teapots by randomly sampling 2000 latent values d ∼ U ( − 4 ., 4 . ) .
Revised: We thus generate 2000 different teapots by randomly sampling 2000 latent values d ∼ U ( − 4 ., 4 . ) .

Original: Simulated geometric variations.
Revised: Simulated geometric variations.

Original: As we set up our problem in the image domain where 3D spatial information is generally incomplete, we describe an object’s geometric properties, speciﬁcally pose and scale, as 2D properties in the image plane.
Revised: As we set up our problem in the image domain where 3D spatial information is generally incomplete, we describe an object’s geometric properties, speciﬁcally pose and scale, as 2D properties in the image plane.

Original: To simulate geometric variations (i.e. poses and scales) while retaining access to a GT difference, we describe the pose of a teapot using translations along the x and y axes, t x ∈ R and t y ∈ R respectively, and an in-plane rotation angle θ ∈ R (clockwise).
Revised: To simulate geometric variations (i.e. poses and scales) while retaining access to a GT difference, we describe the pose of a teapot using translations along the x and y axes, t x ∈ R and t y ∈ R respectively, and an in-plane rotation angle θ ∈ R (clockwise).

Original: Similarly, we describe the scale of a teapot along the x and y axes as s x ∈ R and s y ∈ R .
Revised: Similarly, we describe the scale of a teapot along the x and y axes as s x ∈ R and s y ∈ R .

Original: During simulation, we noticed that the ratio s y s x can implicitly explain-away some of the shape variance.
Revised: During simulation, we noticed that the ratio s y s x can implicitly explain-away some of the shape variance.

Original: As a result, we ﬁx s y =
Revised: As a result, we ﬁx s y =

Original: s x when simulating the data.
Revised: s x when simulating the data.

Original: This allows us to interpret changes in scale as the effect of a change in distance between the object and camera.
Revised: This allows us to interpret changes in scale as the effect of a change in distance between the object and camera.

Original: In sum, we randomly simulate 2000 pose parameters by sampling: θ ∼ U ( − 18 ◦ , 18 ◦ ) , t x ∼ U ( − 0 .
Revised: In sum, we randomly simulate 2000 pose parameters by sampling: θ ∼ U ( − 18 ◦ , 18 ◦ ) , t x ∼ U ( − 0 .

Original: t y ∼ U ( − 0 .
Revised: t y ∼ U ( − 0 .

Original: 4 , 0 . 4) , and 2000 scale parameters by sampling: s x ∼ U (0 .
Revised: 4 , 0 . 4) , and 2000 scale parameters by sampling: s x ∼ U (0 .

Original: Simulated appearance/texture changes.
Revised: Simulated appearance/texture changes.

Original: To simulate appearance changes while retaining access to a GT value, we interpolate RGB values between two randomly-selected texture maps using an interpolation ratio a ∼ U (0 , 1) .
Revised: To simulate appearance changes while retaining access to a GT value, we interpolate RGB values between two randomly-selected texture maps using an interpolation ratio a ∼ U (0 , 1) .

Original: For example, given two texture maps A 0 and A 1 , a simulated texture is a A 1 + (1 − a )
Revised: For example, given two texture maps A 0 and A 1 , a simulated texture is a A 1 + (1 − a )

Original: A 0 .
Revised: A 0 .

Original: For simplicity, we only use texture maps with solid colours.
Revised: For simplicity, we only use texture maps with solid colours.

Original: Ground-truth “distance” measures.
Revised: Ground-truth “distance” measures.

Original: We compute the GT property differences for a source-target image-pair using the saved 7D object descriptors.
Revised: We compute the GT property differences for a source-target image-pair using the saved 7D object descriptors.

Original: Below we detail the calculation of these differences, using the subscripts “ src ” and “ tar ” to indicate properties of the source and target teapots:
Revised: Below we detail the calculation of these differences, using the subscripts “ src ” and “ tar ” to indicate properties of the source and target teapots:

Original: • Shape:∆ dGT = | d tar − d src | • Scale (2D): s GTx = s tarx /s srcx and s GTy =
Revised: • Shape: d = | d tar − d src | • Scale (2D): s x = s tarx /s srcx and s y

Original:  s tary /s srcy • Rotation:∆ θGT = θ tar − θ src (rotations deﬁned within the image plane) • Translation (2D):∆ t GTx = t tarx − t srcx and∆ t GTy =
Revised: = s tary /s srcy • Rotation: θ = θ tar − θ src (rotations deﬁned within the image plane) • Translation (2D): t x = t tar x − t src x and t y =

Original: t tary − t srcy • Texture:∆ aGT = | a tar − a src |
Revised: t tar y − t src y • Texture: a = | a tar − a src |

Original: C E XAMPLES C. P ROPOSAL F LOW (PF)
Revised: C E XAMPLES C. P ROPOSAL F LOW (PF)

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: MSE: initial 2.

Original: MSE: appearances 2.
Revised: : appearances 2.

Original: source: X T tps Taff ( X )
Revised: [Figure] T

Original: [Figure]
Revised: [Figure]

Original: MSE
Revised: ( X )

Original: [Figure] appearances 2.
Revised: [Figure] [Figure] [Figure]

Original: D S EMANTIC ALIGNMENT NETWORKS
Revised: D S EMANTIC ALIGNMENT NETWORKS

Original: [Figure]
Revised: [Figure]