Original: S HAPE -T AILORED D EEP N EURAL N ETWORKS
Revised: S HAPE -T AILORED D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Shape-Tailored Deep Neural Networks (ST-DNN).
Revised: We present Shape-Tailored Deep Neural Networks (ST-DNN).

Original: ST-DNN extend convolutional networks (CNN), which aggregate data from ﬁxed shape (square) neighborhoods, to compute descriptors deﬁned on arbitrarily shaped regions.
Revised: ST-DNN extend convolutional networks (CNN), which aggregate data from ﬁxed shape (square) neighborhoods, to compute descriptors deﬁned on arbitrarily shaped regions.

Original: This is natural for segmentation, where descriptors should describe regions (e.g., of objects) that have diverse shape.
Revised: This is natural for segmentation, where descriptors should describe regions (e.g., of objects) that have diverse shape.

Original: We formulate these descriptors through the Poisson partial differential equation (PDE), which can be used to generalize convolution to arbitrary regions.
Revised: We formulate these descriptors through the Poisson partial differential equation (PDE), which can be used to generalize convolution to arbitrary regions.

Original: We stack multiple PDE layers to generalize a deep CNN to arbitrary regions, and apply it to segmentation.
Revised: We stack multiple PDE layers to generalize a deep CNN to arbitrary regions, and apply it to segmentation.

Original: We show that ST-DNN are covariant to translations and rotations and robust to domain deformations, natural for segmentation, which existing CNN based methods lack.
Revised: We show that ST-DNN are covariant to translations and rotations and robust to domain deformations, natural for segmentation, which existing CNN based methods lack.

Original: ST-DNN are 3-4 orders of magnitude smaller then CNNs used for segmentation.
Revised: ST-DNN are 3-4 orders of magnitude smaller then CNNs used for segmentation.

Original: We show that they exceed segmentation performance compared to state-of-the-art CNN-based descriptors using 2-3 orders smaller training sets on the texture segmentation problem.
Revised: We show that they exceed segmentation performance compared to state-of-the-art CNN-based descriptors using 2-3 orders smaller training sets on the texture segmentation problem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional neural networks (CNNs) have been used extensively for segmentation problems in computer vision
Revised: Convolutional neural networks (CNNs) have been used extensively for segmentation problems in computer vision

Original: He et al.
Revised: He et al.

Original: ; He et al.
Revised: ; He et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Xie & Tu (2015).
Revised: ; Xie & Tu (2015).

Original: CNNs provide a framework for learning descriptors that are able to discriminate different textured or semantic regions within images.
Revised: CNNs provide a framework for learning descriptors that are able to discriminate different textured or semantic regions within images.

Original: Much progress has been made in segmentation with CNNs but results are still far from human performance.
Revised: Much progress has been made in segmentation with CNNs but results are still far from human performance.

Original: Also, signiﬁcant engineering must be performed to adapt CNNs to segmentation problems.
Revised: Also, signiﬁcant engineering must be performed to adapt CNNs to segmentation problems.

Original: A basic component in the architecture for segmentation problems involves labeling or grouping dense descriptors returned by a backbone CNN.
Revised: A basic component in the architecture for segmentation problems involves labeling or grouping dense descriptors returned by a backbone CNN.

Original: A difﬁculty in grouping these descriptors arises, especially near the boundaries of segmentation regions, as CNN descriptors aggregate data from ﬁxed shape (square neighborhoods) at each pixel and may thus aggregate data from different regions.
Revised: A difﬁculty in grouping these descriptors arises, especially near the boundaries of segmentation regions, as CNN descriptors aggregate data from ﬁxed shape (square neighborhoods) at each pixel and may thus aggregate data from different regions.

Original: This makes grouping these descriptors into a unique region difﬁcult, which often results in errors in the grouping.
Revised: This makes grouping these descriptors into a unique region difﬁcult, which often results in errors in the grouping.

Original: In segmentation problems (e.g., semantic segmentation), current methods attempt to mitigate these errors by adding post-processing layers that aim to group simultaneously the (coarse-scale) descriptors from the CNN backbone and the ﬁne-level pixel data.
Revised: In segmentation problems (e.g., semantic segmentation), current methods attempt to mitigate these errors by adding post-processing layers that aim to group simultaneously the (coarse-scale) descriptors from the CNN backbone and the ﬁne-level pixel data.

Original: However, the errors introduced might not always be ﬁxed.
Revised: However, the errors introduced might not always be ﬁxed.

Original: A more natural approach to avoid this problem is to consider the coarse and ﬁne structure together, avoiding aggregation across boundaries, to prevent errors at the outset.
Revised: A more natural approach to avoid this problem is to consider the coarse and ﬁne structure together, avoiding aggregation across boundaries, to prevent errors at the outset.

Original: To avoid such errors, one could design descriptors that aggregate data only within boundaries.
Revised: To avoid such errors, one could design descriptors that aggregate data only within boundaries.

Original: To this end, Khan et al.
Revised: To this end, Khan et al.

Original: (2015) introduced “shape-tailored” descriptors that aggregate data within a region of interest, and used these descriptors for segmentation.
Revised: (2015) introduced “shape-tailored” descriptors that aggregate data within a region of interest, and used these descriptors for segmentation.

Original: However, these descriptors are hand-crafted and do not perform on-par with learned approaches.
Revised: However, these descriptors are hand-crafted and do not perform on-par with learned approaches.

Original: Khan & Sundaramoorthi (2018) introduced learned shape-tailored descriptors by learning a neural network operating on the input channel dimension of input hand-crafted shape-tailored descriptors for segmentation.
Revised: Khan & Sundaramoorthi (2018) introduced learned shape-tailored descriptors by learning a neural network operating on the input channel dimension of input hand-crafted shape-tailored descriptors for segmentation.

Original: However, these networks, though deep in the channel dimension, did not ﬁlter data spatially within layers.
Revised: However, these networks, though deep in the channel dimension, did not ﬁlter data spatially within layers.

Original: Since an advantage of
Revised: Since an advantage of

Original: CNNs comes from exploiting spatial ﬁltering at each depth of the network, in this work, we design shape-tailored networks that are deep and perform shape-tailored ﬁltering in space at each layer using solutions of the Poisson PDE.
Revised: CNNs comes from exploiting spatial ﬁltering at each depth of the network, in this work, we design shape-tailored networks that are deep and perform shape-tailored ﬁltering in space at each layer using solutions of the Poisson PDE.

Original: This results in shape-tailored networks that provide more discriminative descriptors than a single shape-tailored kernel.
Revised: This results in shape-tailored networks that provide more discriminative descriptors than a single shape-tailored kernel.

Original: This extension requires development of techniques to back-propagate through PDEs, which we derive in this work.
Revised: This extension requires development of techniques to back-propagate through PDEs, which we derive in this work.

Original: Our contributions are speciﬁcally:
Revised: Our contributions are speciﬁcally:

Original: We construct and show how to train ST-DNN, deep networks that perform shape-tailored spatial ﬁltering via the Poisson PDE at each depth so as to generalize a CNN to arbitrarily shaped regions.
Revised: We construct and show how to train ST-DNN, deep networks that perform shape-tailored spatial ﬁltering via the Poisson PDE at each depth so as to generalize a CNN to arbitrarily shaped regions.

Original: We show analytically and empirically that ST-DNNs are covariant to translations and rotations as they inherit this property from the Poisson PDE.
Revised: We show analytically and empirically that ST-DNNs are covariant to translations and rotations as they inherit this property from the Poisson PDE.

Original: In segmentation, covariance (a.k.a., equivariance)
Revised: In segmentation, covariance (a.k.a., equivariance)

Original: to translation and rotation is a desired property: if a segment in an image is found, then the corresponding segment should be found in the translated / rotated image (or object).
Revised: to translation and rotation is a desired property: if a segment in an image is found, then the corresponding segment should be found in the translated / rotated image (or object).

Original: This property is not generally present with existing CNN-based segmentation methods even when trained with augmented translated and rotated images Azulay & Weiss (2019), and requires special consideration.
Revised: This property is not generally present with existing CNN-based segmentation methods even when trained with augmented translated and rotated images Azulay & Weiss (2019), and requires special consideration.

Original: We show analytically and empirically that ST-DNNs are robust to domain deformations.
Revised: We show analytically and empirically that ST-DNNs are robust to domain deformations.

Original: These
Revised: These

Original: result
Revised: result

Original: from viewpoint change or object articulation, and so they should not affect the descriptor.
Revised: from viewpoint change or object articulation, and so they should not affect the descriptor.

Original: To demonstrate ST-DNN and the properties above, we validate them on the task of segmentation
Revised: To demonstrate ST-DNN and the properties above, we validate them on the task of segmentation

Original: , an important problem in low-
Revised: , an important problem in low-

Original: level vision
Revised: level vision

Original: Malik & Perona (1990)
Revised: Malik & Perona (1990)

Original: Arbelaez et al.
Revised: Arbelaez et al.

Original: (2011b).
Revised: (2011b).

Original: Because of properties of the PDE, ST-DNN also have desirable generalization properties.
Revised: Because of properties of the PDE, ST-DNN also have desirable generalization properties.

Original: This is because: a) The robustness and covariance properties are built into our descriptors and do not need to be learned from data, b) The PDE solutions, generalizations of Gabor-like ﬁlters Olshausen & Field (1996); Zador (2019), have natural image structure inherent in their solutions and so this does not need to be learned from data, and c)
Revised: This is because: a) The robustness and covariance properties are built into our descriptors and do not need to be learned from data, b) The PDE solutions, generalizations of Gabor-like ﬁlters Olshausen & Field (1996); Zador (2019), have natural image structure inherent in their solutions and so this does not need to be learned from data, and c)

Original: Our networks have fewer parameters compared to existing networks in segmentation.
Revised: Our networks have fewer parameters compared to existing networks in segmentation.

Original: This is because the PDE solutions form a basis and only linear combinations of a few basis elements are needed to learn discriminative descriptors for segmentation.
Revised: This is because the PDE solutions form a basis and only linear combinations of a few basis elements are needed to learn discriminative descriptors for segmentation.

Original: In contrast,
Revised: In contrast,

Original: CNNs spend a lot of parameters to learn this structure.
Revised: CNNs spend a lot of parameters to learn this structure.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Traditional approaches to segmentation rely on hand-crafted features, e.g., through a ﬁlter bank
Revised: Traditional approaches to segmentation rely on hand-crafted features, e.g., through a ﬁlter bank

Original: Haralick & Shapiro (1985).
Revised: Haralick & Shapiro (1985).

Original: These features are ambiguous near the boundaries of objects.
Revised: These features are ambiguous near the boundaries of objects.

Original: In Khan et al.
Revised: In Khan et al.

Original: (2015) hand-crafted descriptors that aggregate data within object boundaries are constructed to avoid this, but lack sufﬁcient capacity to capture the diversity of textures or be invariant to nuisances.
Revised: (2015) hand-crafted descriptors that aggregate data within object boundaries are constructed to avoid this, but lack sufﬁcient capacity to capture the diversity of textures or be invariant to nuisances.

Original: Deep-learning based approaches have showed state-of-the-art results in edge-based methods Xie &
Revised: Deep-learning based approaches have showed state-of-the-art results in edge-based methods Xie &

Original: Tu (2017); He et al.
Revised: Tu (2017); He et al.

Original: ; Deng et al.
Revised: ; Deng et al.

Original: Watershed is applied on edge-maps to obtain the segmentation.
Revised: Watershed is applied on edge-maps to obtain the segmentation.

Original: The main drawback of these methods is it is often difﬁcult to form segmentations due to extraneous or faint edges, particularly when "textons" in textures are large.
Revised: The main drawback of these methods is it is often difﬁcult to form segmentations due to extraneous or faint edges, particularly when "textons" in textures are large.

Original: CNNs have been applied to compute descriptors for semantic segmentation, where pixels in an image are classiﬁed into certain semantic object classes Li et al.
Revised: CNNs have been applied to compute descriptors for semantic segmentation, where pixels in an image are classiﬁed into certain semantic object classes Li et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: Du et al.
Revised: Du et al.

Original: ; Pang et al.
Revised: ; Pang et al.

Original: Zhu et al.
Revised: Zhu et al.

Original: Liu et al.
Revised: Liu et al.

Original: Usually these classes are limited to a few object classes and do not tackle general textures, where the number of classes may be far greater, and thus such approaches are not directly applicable to texture segmentation.
Revised: Usually these classes are limited to a few object classes and do not tackle general textures, where the number of classes may be far greater, and thus such approaches are not directly applicable to texture segmentation.

Original: But semantic segmentation approaches may eventually beneﬁt from our methodology as descriptors aggregating data only within objects or regions are also relevant to these problems.
Revised: But semantic segmentation approaches may eventually beneﬁt from our methodology as descriptors aggregating data only within objects or regions are also relevant to these problems.

Original: A learned shape-tailored descriptor Khan & Sundaramoorthi (2018) is constructed with a Siamese network on hand-crafted shape-tailored descriptors.
Revised: A learned shape-tailored descriptor Khan & Sundaramoorthi (2018) is constructed with a Siamese network on hand-crafted shape-tailored descriptors.

Original: However, Khan & Sundaramoorthi (2018) only does shape-tailored ﬁltering in pre-processing as layering these requires new methods to train.
Revised: However, Khan & Sundaramoorthi (2018) only does shape-tailored ﬁltering in pre-processing as layering these requires new methods to train.

Original: We further examine covariance and robustness, not examined in Khan & Sundaramoorthi (2018).
Revised: We further examine covariance and robustness, not examined in Khan & Sundaramoorthi (2018).

Original: Covariance to rotation in CNNs has been examined in recent works, e.g., Weiler et al.
Revised: Covariance to rotation in CNNs has been examined in recent works, e.g., Weiler et al.

Original: ; Yin et al.
Revised: ; Yin et al.

Original: ; Anderson et al.
Revised: ; Anderson et al.

Original: They, however, are not shape-tailored so do not aggregate data only within shaped regions.
Revised: They, however, are not shape-tailored so do not aggregate data only within shaped regions.

Original: Lack of robustness to deformation (and translation) in CNNs is examined in Azulay & Weiss (2019) and theoretically in Bietti & Mairal (2017).
Revised: Lack of robustness to deformation (and translation) in CNNs is examined in Azulay & Weiss (2019) and theoretically in Bietti & Mairal (2017).

Original: Sifre & Mallat (2013) constructs deformation robust descriptors inspired by CNNs, but are hand-crafted.
Revised: Sifre & Mallat (2013) constructs deformation robust descriptors inspired by CNNs, but are hand-crafted.

Original:  C ONSTRUCTION OF S HAPE - TAILORED DNN AND P ROPERTIES
Revised:  C ONSTRUCTION OF S HAPE - TAILORED DNN AND P ROPERTIES

Original: In this section, we design a deep neural network that outputs descriptors at each pixel within an arbitrary shaped region of interest and aggregates data only from within the region.
Revised: In this section, we design a deep neural network that outputs descriptors at each pixel within an arbitrary shaped region of interest and aggregates data only from within the region.

Original: We want the descriptors to be discriminative of different texture, yet robust to nuisances within the region (e.g., local photometric and geometric variability) to be useful for segmentation.
Revised: We want the descriptors to be discriminative of different texture, yet robust to nuisances within the region (e.g., local photometric and geometric variability) to be useful for segmentation.

Original: Our construction uses a
Revised: Our construction uses a

Original: Poisson PDE, which naturally smooths data only within a region of interest.
Revised: Poisson PDE, which naturally smooths data only within a region of interest.

Original: Smoothing naturally yields robustness to geometric nuisances (domain deformations).
Revised: Smoothing naturally yields robustness to geometric nuisances (domain deformations).

Original: By taking linear combinations of derivatives of the output of the PDE, we can approximate the effect of general convolutional kernels but avoid mixing data across the boundary of region of interest.
Revised: By taking linear combinations of derivatives of the output of the PDE, we can approximate the effect of general convolutional kernels but avoid mixing data across the boundary of region of interest.

Original: ST-DNN is also covariant to translations and rotations, inheriting it from the Poisson equation, which leads to the segmentation algorithm being covariant to such transformations.
Revised: ST-DNN is also covariant to translations and rotations, inheriting it from the Poisson equation, which leads to the segmentation algorithm being covariant to such transformations.

Original: 2. S HAPE -T AILORED DNN D ESCRIPTORS THROUGH P OISSON PDE
Revised: 2. S HAPE -T AILORED DNN D ESCRIPTORS THROUGH P OISSON PDE

Original: Shape-tailored Smoothing via Poisson PDE : To construct a shape-tailored deep network, we ﬁrst smooth the input to a layer using the Poisson PDE so as to aggregate data only within the region of interest, similar to what is done in Khan et al.
Revised: Shape-tailored Smoothing via Poisson PDE : To construct a shape-tailored deep network, we ﬁrst smooth the input to a layer using the Poisson PDE so as to aggregate data only within the region of interest, similar to what is done in Khan et al.

Original: for just the ﬁrst layer.
Revised: for just the ﬁrst layer.

Original: Let R ⊂ Ω ⊂ R 2 be the region of interest, where Ω is the domain of the input image
Revised: Let R ⊂ Ω ⊂ R 2 be the region of interest, where Ω is the domain of the input image

Original: I : Ω → R k and k is the number of input channels to the layer.
Revised: I : Ω → R k and k is the number of input channels to the layer.

Original: Let u : R → R M ( M is the number of output channels) be the result of the smoothing; the components u of u solve the PDE within R :
Revised: Let u : R → R M ( M is the number of output channels) be the result of the smoothing; the components u of u solve the PDE within R :

Original: [Equation]
Revised: [Equation]

Original: where I is a channel of I , ∂R is the boundary of R , N is normal to ∂R , α is the scale of smoothing and ∆ / ∇ are the Laplacian and the gradient respectively.
Revised: where I is a channel of I , ∂R is the boundary of R , N is normal to ∂R , α is the scale of smoothing and ∆ / ∇ are the Laplacian and the gradient respectively.

Original: It can be shown that the smoothing can be written in the form u ( x )
Revised: It can be shown that the smoothing can be written in the form u ( x )

Original: = (cid:82) R K ( x, y )
Revised: = (cid:82) R K ( x, y )

Original: I ( y ) dy where K ( ., . )
Revised: I ( y ) dy where K ( ., . )

Original: is the Green’s function of the PDE, a smoothing kernel, which further shows that the PDE aggregates data only within R .
Revised: is the Green’s function of the PDE, a smoothing kernel, which further shows that the PDE aggregates data only within R .

Original: Shape-tailored Deep Network : We can now generalize the operation of convolution tailored to the region of interest by taking linear combinations of partial derivatives of the output of the PDE equation 1.
Revised: Shape-tailored Deep Network : We can now generalize the operation of convolution tailored to the region of interest by taking linear combinations of partial derivatives of the output of the PDE equation 1.

Original: This is motivated by the fact that in R = R 2 , linear combinations of derivatives of
Revised: This is motivated by the fact that in R = R 2 , linear combinations of derivatives of

Original: Gaussians can approximate any kernel arbitrarily well.
Revised: Gaussians can approximate any kernel arbitrarily well.

Original: Gaussian ﬁlters are the solution of the heat equation, and the PDE equation 1 relates to the heat equation, i.e., equation 1 is the steady state solution of a heat equation.
Revised: Gaussian ﬁlters are the solution of the heat equation, and the PDE equation 1 relates to the heat equation, i.e., equation 1 is the steady state solution of a heat equation.

Original: Thus, linear combinations of derivatives of equation 1 generalize convolution to an arbitrary shape R ; in experiments, a few ﬁrst order directional derivatives are sufﬁcient for our segmentation tasks (see Section 5 for details).
Revised: Thus, linear combinations of derivatives of equation 1 generalize convolution to an arbitrary shape R ; in experiments, a few ﬁrst order directional derivatives are sufﬁcient for our segmentation tasks (see Section 5 for details).

Original: A layer of the ST-DNN takes such linear combinations and rectiﬁes it as follows:
Revised: A layer of the ST-DNN takes such linear combinations and rectiﬁes it as follows:

Original: [Equation]
Revised: [Equation]

Original: where I : R → R k is the input to the layer, T is an operator that outputs derivatives of the solution of the Poisson PDE equation 1, L i ( y ) =
Revised: where I : R → R k is the input to the layer, T is an operator that outputs derivatives of the solution of the Poisson PDE equation 1, L i ( y ) =

Original: w i y
Revised: w i y

Original: + b i is a point-wise linear function (i.e., a 1 × 1 convolution applied to combine different channels), r is the rectiﬁed linear function, and i indexes the layer of the network.
Revised: + b i is a point-wise linear function (i.e., a 1 × 1 convolution applied to combine different channels), r is the rectiﬁed linear function, and i indexes the layer of the network.

Original: Notice that since r and L i are pointwise operations, they preserve the property of T that it aggregates data only within the region R .
Revised: Notice that since r and L i are pointwise operations, they preserve the property of T that it aggregates data only within the region R .

Original: We now compose layers to construct a ST-DNN as follows:
Revised: We now compose layers to construct a ST-DNN as follows:

Original: [Equation]
Revised: [Equation]

Original: where F [ I ]( x ) is the output of the ST-DNN, f 0 , ..., f m are the m + 1 layers of the network, I is the input image, and s represents the soft-max operation (to bound the output values).
Revised: where F [ I ]( x ) is the output of the ST-DNN, f 0 , ..., f m are the m + 1 layers of the network, I is the input image, and s represents the soft-max operation (to bound the output values).

Original: ST-DNN does not have a pooling layer because the PDE already aggregates data from a neighborhood by smoothing; further, the lack of reduction in spatial dimension allows for more accurate shape estimation in our subsequent segmentation, and avoids the need for up-sampling layers.
Revised: ST-DNN does not have a pooling layer because the PDE already aggregates data from a neighborhood by smoothing; further, the lack of reduction in spatial dimension allows for more accurate shape estimation in our subsequent segmentation, and avoids the need for up-sampling layers.

Original: We will show that we can retain efﬁciency in training and inference.
Revised: We will show that we can retain efﬁciency in training and inference.

Original: 2. C OVARIANCE AND R OBUSTNESS OF ST-DNN
Revised: 2. C OVARIANCE AND R OBUSTNESS OF ST-DNN

Original: In addition to ST-DNN generalizing CNNs to arbitrary shaped regions, the ST-DNN is also covariant to in-plane translation and rotation, and robustness to domain deformations due to properties of the
Revised: In addition to ST-DNN generalizing CNNs to arbitrary shaped regions, the ST-DNN is also covariant to in-plane translation and rotation, and robustness to domain deformations due to properties of the

Original: Poisson PDE.
Revised: Poisson PDE.

Original: This means covariance also extends to our segmentation method, which is important as any object segmented in an image will also be segmented if the camera undergoes these transformations.
Revised: This means covariance also extends to our segmentation method, which is important as any object segmented in an image will also be segmented if the camera undergoes these transformations.

Original: Robustness to deformations is important as this means that small geometric variability (e.g., shape variations in textons, small viewpoint change, object deformation) will not affect the descriptors and the segmentation much.
Revised: Robustness to deformations is important as this means that small geometric variability (e.g., shape variations in textons, small viewpoint change, object deformation) will not affect the descriptors and the segmentation much.

Original: We make these properties more precise, and give intuition for proofs, leaving details to Appendix B.
Revised: We make these properties more precise, and give intuition for proofs, leaving details to Appendix B.

Original: A covariant operator commutes with a set of transformations:
Revised: A covariant operator commutes with a set of transformations:

Original: Deﬁnition 1 An operator S : I → I (from the set of images I to itself) is covariant to a class W of transformations if S [ I ◦ w ] =
Revised: Deﬁnition 1 An operator S : I → I (from the set of images I to itself) is covariant to a class W of transformations if S [ I ◦ w ] =

Original: [ SI ] ◦ w for every I ∈ I and w ∈ W .
Revised: [ SI ] ◦ w for every I ∈ I and w ∈ W .

Original: ST-DNN is covariant to W , the set of in-plane rotations and translations:
Revised: ST-DNN is covariant to W , the set of in-plane rotations and translations:

Original: Theorem 1 The ST-DNN equation 3 is covariant to the set of translations and rotations, i.e., x →
Revised: Theorem 1 The ST-DNN equation 3 is covariant to the set of translations and rotations, i.e., x →

Original: R x + T where R is a 2 × 2 rotation matrix and T ∈ R 2 .
Revised: R x + T where R is a 2 × 2 rotation matrix and T ∈ R 2 .

Original: This follows from the covariance of the Laplacian and point-wise operations (rectiﬁcation, 1 ×convolution), and lack of sub-sampling.
Revised: This follows from the covariance of the Laplacian and point-wise operations (rectiﬁcation, 1 ×convolution), and lack of sub-sampling.

Original: We now make precise the robustness of the ST-DNN to domain deformations:
Revised: We now make precise the robustness of the ST-DNN to domain deformations:

Original: Theorem 2 The ST-DNN equation 3 is insensitive to deformations, i.e.,
Revised: Theorem 2 The ST-DNN equation 3 is insensitive to deformations, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where w : Ω → Ω is a domain deformation, id is the identity map, H 1 is the Sobolev norm (measures both the amount and smoothness of the deformation), and C is a constant independent of w, I .
Revised: where w : Ω → Ω is a domain deformation, id is the identity map, H 1 is the Sobolev norm (measures both the amount and smoothness of the deformation), and C is a constant independent of w, I .

Original: Intuitively, this follows from the fact that the Poisson PDE locally averages input data, and local averages are robust to translation and hence deformations, which are locally translations.
Revised: Intuitively, this follows from the fact that the Poisson PDE locally averages input data, and local averages are robust to translation and hence deformations, which are locally translations.

Original:  T RAINING OF THE N ETWORK AND B ACK -P ROPAGATION
Revised:  T RAINING OF THE N ETWORK AND B ACK -P ROPAGATION

Original: In this section, we describe the training of ST-DNN by introducing a loss function, how the weights can be learned, and the implementation.
Revised: In this section, we describe the training of ST-DNN by introducing a loss function, how the weights can be learned, and the implementation.

Original: As the network layers solutions to PDEs, one needs to differentiate through such layers, which we describe.
Revised: As the network layers solutions to PDEs, one needs to differentiate through such layers, which we describe.

Original: 3. L OSS F UNCTION FOR T RAINING
Revised: 3. L OSS F UNCTION FOR T RAINING

Original: Given the ST-DNN of Section 2.1, the loss function to train such descriptors from ground truth segmentation masks (motivated by consistency to the segmentation algorithm in Section 4 that is based on classical energies Chan & Vese (2001); Yezzi Jr et al. (1999) from computer vision) is deﬁned as:
Revised: Given the ST-DNN of Section 2.1, the loss function to train such descriptors from ground truth segmentation masks (motivated by consistency to the segmentation algorithm in Section 4 that is based on classical energies Chan & Vese (2001); Yezzi Jr et al. (1999) from computer vision) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: i (cid:88) j (cid:54) =
Revised: i (cid:88) j (cid:54) =

Original: i || a
Revised: i || a

Original: i − a j ||(5) where i, j ∈ { 1 , 2 , ..., N } are the indices for the regions in the ground truth segmentation, F W ( x ) is the output of the ST-DNN, W are the weights of the network (i.e., weights on derivatives of the Poisson PDE solution), | R i | is the area of region i , and a i is the average descriptor within the i th region, i.e., a i = 1 | R i | (cid:82) R i F W ( x ) dx .
Revised: i − a j ||(5) where i, j ∈ { 1 , 2 , ..., N } are the indices for the regions in the ground truth segmentation, F W ( x ) is the output of the ST-DNN, W are the weights of the network (i.e., weights on derivatives of the Poisson PDE solution), | R i | is the area of region i , and a i is the average descriptor within the i th region, i.e., a i = 1 | R i | (cid:82) R i F W ( x ) dx .

Original: The loss function is comprised of two terms.
Revised: The loss function is comprised of two terms.

Original: The ﬁrst component of the loss is minimized when the learned descriptor is constant within regions
Revised: The ﬁrst component of the loss is minimized when the learned descriptor is constant within regions

Original: R i so that each region consists of parts of the image with uniform descriptor.
Revised: R i so that each region consists of parts of the image with uniform descriptor.

Original: The second term forces the learned descriptor of different regions to be different to discriminate different textures.
Revised: The second term forces the learned descriptor of different regions to be different to discriminate different textures.

Original: 3. C OMPUTING G RADIENTS OF THE L OSS AND T RAINING
Revised: 3. C OMPUTING G RADIENTS OF THE L OSS AND T RAINING

Original: Computing gradients of the loss function for training requires consideration as it involves differentiating through PDEs.
Revised: Computing gradients of the loss function for training requires consideration as it involves differentiating through PDEs.

Original: The most straightforward way to do this involves discretizing the PDE, so the solution is a linear matrix system as we do below.
Revised: The most straightforward way to do this involves discretizing the PDE, so the solution is a linear matrix system as we do below.

Original: This allows the use of existing deep learning packages to perform back-propagation by storing the matrix in memory.
Revised: This allows the use of existing deep learning packages to perform back-propagation by storing the matrix in memory.

Original: However, this can lead to large memory consumption as the matrix can be large and is only feasible for small images.
Revised: However, this can lead to large memory consumption as the matrix can be large and is only feasible for small images.

Original: Fortunately, our PDEs involve a scale parameter α so we can train using down-sampled images, facilitating the use of existing packages for back-prop, and then infer on native resolution images by simply scaling α by the down-sampling factor, which we do in experiments.
Revised: Fortunately, our PDEs involve a scale parameter α so we can train using down-sampled images, facilitating the use of existing packages for back-prop, and then infer on native resolution images by simply scaling α by the down-sampling factor, which we do in experiments.

Original: The more accurate method, though more difﬁcult to implement, is to avoid storing the matrix and instead compute the PDE solution by an iterative numerical PDE method that does not require storage of matrices.
Revised: The more accurate method, though more difﬁcult to implement, is to avoid storing the matrix and instead compute the PDE solution by an iterative numerical PDE method that does not require storage of matrices.

Original: This requires formulating a variant of the back-propagation algorithm, that is similar but involves forward propagation of layer derivatives with respect to the weights through the PDE solutions.
Revised: This requires formulating a variant of the back-propagation algorithm, that is similar but involves forward propagation of layer derivatives with respect to the weights through the PDE solutions.

Original: This is unfortunately not available in standard deep learning packages.
Revised: This is unfortunately not available in standard deep learning packages.

Original: For completeness, we provide the mathematical formulation of this approach in Appendix C,for which
Revised: For completeness, we provide the mathematical formulation of this approach in Appendix C,for which

Original: [Figure]
Revised: [Figure]

Original: we have performed experiments for few layer cases.
Revised: we have performed experiments for few layer cases.

Original: This did not give an appreciable performance increase given the complexity of implementation, but could be useful in other applications.
Revised: This did not give an appreciable performance increase given the complexity of implementation, but could be useful in other applications.

Original: Implementation : Using the ﬁrst method above, we discretize equation 1 as:
Revised: Implementation : Using the ﬁrst method above, we discretize equation 1 as:

Original: [Equation]
Revised: [Equation]

Original: where N ( i, j ) represents the 4-pixel neighborhood of pixel ( i, j ) , which represents the i th row and j th column, and intersection means that only neighbors in the region are considered as implied by the Neumann boundary condition in the PDE, avoiding aggregation outside R .
Revised: where N ( i, j ) represents the 4-pixel neighborhood of pixel ( i, j ) , which represents the i th row and j th column, and intersection means that only neighbors in the region are considered as implied by the Neumann boundary condition in the PDE, avoiding aggregation outside R .

Original: The discretization approximately preserves the rotation covariance, and any errors vanish with increasing resolution.
Revised: The discretization approximately preserves the rotation covariance, and any errors vanish with increasing resolution.

Original: Note that more accurate discretizations exist, but our experiments demonstrate the sufﬁciency of this scheme.
Revised: Note that more accurate discretizations exist, but our experiments demonstrate the sufﬁciency of this scheme.

Original: We can vectorize u and I and write equation 6 as:
Revised: We can vectorize u and I and write equation 6 as:

Original: [Equation]
Revised: [Equation]

Original: The above is a linear transformation from I to u .
Revised: The above is a linear transformation from I to u .

Original: The size of A is ( mn × mn ) , where m and n are the number of rows and columns in I .
Revised: The size of A is ( mn × mn ) , where m and n are the number of rows and columns in I .

Original: With the PDE layers deﬁned through this matrix multiplication, we can use the usual back-propagation method to compute derivatives with respect to weights (see
Revised: With the PDE layers deﬁned through this matrix multiplication, we can use the usual back-propagation method to compute derivatives with respect to weights (see

Original: Appendix D for more details).
Revised: Appendix D for more details).

Original: In experiments, we downsampled images to 32 × 32 for training.
Revised: In experiments, we downsampled images to 32 × 32 for training.

Original:  A PPLICATION TO S EGMENTATION
Revised:  A PPLICATION TO S EGMENTATION

Original: In this section, we describe the procedure for segmentation using the trained ST-DNN.
Revised: In this section, we describe the procedure for segmentation using the trained ST-DNN.

Original: During inference time, the regions of segmentation are estimated iteratively together with updates of the
Revised: During inference time, the regions of segmentation are estimated iteratively together with updates of the

Original: ST-DNN for each of the regions as they evolve.
Revised: ST-DNN for each of the regions as they evolve.

Original: The evolution of the regions to determine the segmentation is obtained by optimizing the following energy (based on the classical energy Chan &
Revised: The evolution of the regions to determine the segmentation is obtained by optimizing the following energy (based on the classical energy Chan &

Original: Vese (2001)):
Revised: Vese (2001)):

Original: [Equation]
Revised: [Equation]

Original: where R = { R 1 , R 2 , ..., R N } are regions in segmentation, F R i is the shape-tailored descriptor from the DNN given the learned weights W and within R i , and β > 0 is the arclength regularization parameter.
Revised: where R = { R 1 , R 2 , ..., R N } are regions in segmentation, F R i is the shape-tailored descriptor from the DNN given the learned weights W and within R i , and β > 0 is the arclength regularization parameter.

Original: Note that this energy differs from the loss function used for training in two ways.
Revised: Note that this energy differs from the loss function used for training in two ways.

Original: First, the second term in equation 5 is omitted as it is used in training to avoid the descriptor from learning to be uniform across different textured regions; during inference, the network is already trained to be different across different regions.
Revised: First, the second term in equation 5 is omitted as it is used in training to avoid the descriptor from learning to be uniform across different textured regions; during inference, the network is already trained to be different across different regions.

Original: Second, we add regularization to keep the region boundaries smooth; it is not needed in training since we do not solve for the regions as ground truth is available.
Revised: Second, we add regularization to keep the region boundaries smooth; it is not needed in training since we do not solve for the regions as ground truth is available.

Original: To minimize the (non-convex) energy with respect to the region, we use gradient decent.
Revised: To minimize the (non-convex) energy with respect to the region, we use gradient decent.

Original: The gradient with respect to the region R i is approximately given by [ (cid:107) F R i ( x ; W ) − a i (cid:107) 22 − (cid:107) F R j ( x ; W ) − a j (cid:107) 

Revised: The gradient with respect to the region R i is approximately given by [ (cid:107) F R i ( x ; W ) − a i (cid:107) 22 − (cid:107) F R j ( x ; W ) − a j (cid:107)

Original: + βκ i ] N i where N i is the unit outward normal to the region, and κ i is its curvature.
Revised: + βκ i ] N i where N i is the unit outward normal to the region, and κ i is its curvature.

Original: The curve (boundary of regions) evolution to determine the regions is implemented with a method analogous to level set methods Osher & Sethian (1988) by evolving smooth indicator functions of regions for convenient implementation, details of the algorithm and implementation are shown in Algorithmin Appendix D.
Revised: The curve (boundary of regions) evolution to determine the regions is implemented with a method analogous to level set methods Osher & Sethian (1988) by evolving smooth indicator functions of regions for convenient implementation, details of the algorithm and implementation are shown in Algorithmin Appendix D.

Original: The method involves joint updates of the regions and the shape-tailored descriptors within the evolving regions (see Figure 1).
Revised: The method involves joint updates of the regions and the shape-tailored descriptors within the evolving regions (see Figure 1).

Original: We used a box tessellation to initialise the regions, typical of level set methods for segmentation.
Revised: We used a box tessellation to initialise the regions, typical of level set methods for segmentation.

Original: Our method typically takes a few iterations (approx. 20) to converge in our experiments.
Revised: Our method typically takes a few iterations (approx. 20) to converge in our experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Network Architecture: We use a 4 layer ST-DNN, which is optimal for the datasets used: fewer layers lead to less accuracy and more layers lead to overﬁtting (see Appendix E).
Revised: Network Architecture: We use a 4 layer ST-DNN, which is optimal for the datasets used: fewer layers lead to less accuracy and more layers lead to overﬁtting (see Appendix E).

Original: The layer f 0 outputs a 40 dimensional descriptor, with 3 color channels, a gray scale channel and oriented gradients at angles { 0 , π/ 4 , π/ 2 , 3 π/ 4 } over 5 shape-tailored smoothing levels (scales)
Revised: The layer f 0 outputs a 40 dimensional descriptor, with 3 color channels, a gray scale channel and oriented gradients at angles { 0 , π/ 4 , π/ 2 , 3 π/ 4 } over 5 shape-tailored smoothing levels (scales)

Original: α = { 5 , 10 , 15 , 20 , 25 } .
Revised: α = { 5 , 10 , 15 , 20 , 25 } .

Original: The four fully-connected layers have 100, 40, 20, 5 hidden units respectively.
Revised: The four fully-connected layers have 100, 40, 20, 5 hidden units respectively.

Original: The smoothing parameter α for all subsequent layers is set to 5.
Revised: The smoothing parameter α for all subsequent layers is set to 5.

Original: The training on the datasets (below) takes less than 2 hours on Nvidia Quadro RTX 6000 GPU and Intel Xeon 2.60GHz CPU.
Revised: The training on the datasets (below) takes less than 2 hours on Nvidia Quadro RTX 6000 GPU and Intel Xeon 2.60GHz CPU.

Original: The inference time (joint segmentation) is 2 seconds on images of size 256 × 256 .
Revised: The inference time (joint segmentation) is 2 seconds on images of size 256 × 256 .

Original: Datasets: We apply ST-DNN to texture segmentation (since covariance and robustness properties are important for texture descriptors Julesz (1981); Sifre & Mallat (2013)).
Revised: Datasets: We apply ST-DNN to texture segmentation (since covariance and robustness properties are important for texture descriptors Julesz (1981); Sifre & Mallat (2013)).

Original: We evaluate on two challenging texture segmentation datasets Khan et al.
Revised: We evaluate on two challenging texture segmentation datasets Khan et al.

Original: (2015) - the Real-World Texture Segmentation dataset (RWTSD) - 256 complex real-world images (128 training and testing images) and the
Revised: (2015) - the Real-World Texture Segmentation dataset (RWTSD) - 256 complex real-world images (128 training and testing images) and the

Original: Synthetic Texture Dataset consists of 200 test images and 300 training images generated from the
Revised: Synthetic Texture Dataset consists of 200 test images and 300 training images generated from the

Original: Brodatz dataset.
Revised: Brodatz dataset.

Original: We have also tested on multi-region segmentation BSDS500 and Synthetic Texture datasets, details are provided below.
Revised: We have also tested on multi-region segmentation BSDS500 and Synthetic Texture datasets, details are provided below.

Original: Methods: We compare our method against popular deep learning architectures in computer vision - DeepLab-v3 Chen et al.
Revised: Methods: We compare our method against popular deep learning architectures in computer vision - DeepLab-v3 Chen et al.

Original: (2017), and
Revised: (2017), and

Original: FCN-ResNet101 He et al.
Revised: FCN-ResNet101 He et al.

Original: In our notation resnet/deeplabv3-x-y, resnet and deeplabv3 represents FCN-Resnet101 and DeepLab-v3 respectively, x denotes the data used in training (subsequently ﬁne-tuned on the texture segmentation dataset).
Revised: In our notation resnet/deeplabv3-x-y, resnet and deeplabv3 represents FCN-Resnet101 and DeepLab-v3 respectively, x denotes the data used in training (subsequently ﬁne-tuned on the texture segmentation dataset).

Original: ’x’ can be ’m’ for MSRA dataset, ’d’ for DUTS dataset, ’all’ for a combination of all datasets (see
Revised: ’x’ can be ’m’ for MSRA dataset, ’d’ for DUTS dataset, ’all’ for a combination of all datasets (see

Original: Appendix E for more details), and ’TD’ for RWTSD (augmented with 8 rotations and 5 scales). ’
Revised: Appendix E for more details), and ’TD’ for RWTSD (augmented with 8 rotations and 5 scales). ’

Original: y’ denotes the loss function, ’ce’ represents cross-entropy and ’ours’ represents the loss introduced in this paper.
Revised: y’ denotes the loss function, ’ce’ represents cross-entropy and ’ours’ represents the loss introduced in this paper.

Original: Segmentation for all methods is done by minimizing equation 8.
Revised: Segmentation for all methods is done by minimizing equation 8.

Original: We also compare our methods against the state-of-the-art methods for texture segmentation, which contain both the classical Arbelaez et al.
Revised: We also compare our methods against the state-of-the-art methods for texture segmentation, which contain both the classical Arbelaez et al.

Original: (2011b)
Revised: (2011b)

Original: ; Arbeláez et al.
Revised: ; Arbeláez et al.

Original: (2014); Isola et al.
Revised: (2014); Isola et al.

Original: (2014); Kokkinos (2015);
Revised: (2014); Kokkinos (2015);

Original: Khan et al.
Revised: Khan et al.

Original: and deep-learning methods Khan & Sundaramoorthi (2018).
Revised: and deep-learning methods Khan & Sundaramoorthi (2018).

Original: ST-DNN is trained only on texture datasets.
Revised: ST-DNN is trained only on texture datasets.

Original: Evaluation Metrics : We compare on evaluation metrics from Arbelaez et al.
Revised: Evaluation Metrics : We compare on evaluation metrics from Arbelaez et al.

Original: (2011a).
Revised: (2011a).

Original: Ground truth covering (GT-Cov), Random Index (RI) and Variation of Information (VOI) measures region accuracy (higher GT-Cov, RI and lower VOI are more accurate), and F-measure (higher is better) measures boundary accuracy.
Revised: Ground truth covering (GT-Cov), Random Index (RI) and Variation of Information (VOI) measures region accuracy (higher GT-Cov, RI and lower VOI are more accurate), and F-measure (higher is better) measures boundary accuracy.

Original: Ablation Studies : We have performed ablations studies that are summarized in Appendix E Table 4.
Revised: Ablation Studies : We have performed ablations studies that are summarized in Appendix E Table 4.

Original: They show that more PDE ﬁltering layers give higher accuracy (up to a point of overﬁtting atlayers), and that shape-tailored descriptors updated as the region updates outperforms non-shape tailored descriptor (ST-DNN computed on the whole image and not updated as the region evolves).
Revised: They show that more PDE ﬁltering layers give higher accuracy (up to a point of overﬁtting atlayers), and that shape-tailored descriptors updated as the region updates outperforms non-shape tailored descriptor (ST-DNN computed on the whole image and not updated as the region evolves).

Original: Testing Covariance: To demonstrate the covariance of ST-DNN to translation and rotation, we performed an experiment on Real-world Texture Segmentation dataset.
Revised: Testing Covariance: To demonstrate the covariance of ST-DNN to translation and rotation, we performed an experiment on Real-world Texture Segmentation dataset.

Original: Each image in the test set was randomly rotated with θ ∈ { π/ 6 , 2 π/ 6 , 3 π/ 2 , 4 π/ 6 , 5 π/ 6 , π } and cropped to a rectangle at random positions (to simulate translation) in the rotated image; we ensure the rectangle only contains data from the original image.
Revised: Each image in the test set was randomly rotated with θ ∈ { π/ 6 , 2 π/ 6 , 3 π/ 2 , 4 π/ 6 , 5 π/ 6 , π } and cropped to a rectangle at random positions (to simulate translation) in the rotated image; we ensure the rectangle only contains data from the original image.

Original: We segment the original and the transformed image, denoted S
Revised: We segment the original and the transformed image, denoted S

Original: [ I ] and
Revised: [ I ] and

Original: S [ I ◦ w ] , respectively, where w is the transformation used to produce the translated/rotated image.
Revised: S [ I ◦ w ] , respectively, where w is the transformation used to produce the translated/rotated image.

Original: We then measure the difference between S
Revised: We then measure the difference between S

Original: [ I ] ◦ w and S
Revised: [ I ] ◦ w and S

Original: [ I ◦ w ] through GT-covering; both should be equal if the descriptor is covariant.
Revised: [ I ◦ w ] through GT-covering; both should be equal if the descriptor is covariant.

Original: Results are summarized in Figure 2.
Revised: Results are summarized in Figure 2.

Original: ST-DNN outperforms resnet101-all-ce by a margin of almost 25%.
Revised: ST-DNN outperforms resnet101-all-ce by a margin of almost 25%.

Original: Note ST-DNN uses no data augmentation, whereas the competing networks are augmented with translated and rotated images from RWTSD.
Revised: Note ST-DNN uses no data augmentation, whereas the competing networks are augmented with translated and rotated images from RWTSD.

Original: Perfect covariance may not be achieved as translation/rotation also necessarily include occlusion.
Revised: Perfect covariance may not be achieved as translation/rotation also necessarily include occlusion.

Original: [Figure]
Revised: [Figure]

Original: Testing Deformation Robustness: To demonstrate robustness to deformation, we apply the trained networks above to randomly deformed versions of the RWTSD test set.
Revised: Testing Deformation Robustness: To demonstrate robustness to deformation, we apply the trained networks above to randomly deformed versions of the RWTSD test set.

Original: We generate random smooth deformations using truncated Fourier series: v ( x ) = (cid:80)
Revised: We generate random smooth deformations using truncated Fourier series: v ( x ) = (cid:80)

Original: Nk
Revised: Nk

Original: =
Revised: =

Original: − N a k exp( i 2 πk · x ) where x ∈ [0 , 1] 2 , k = ( k 1 , k 2 ) , w ( x ) = x
Revised: − N a k exp( i 2 πk · x ) where x ∈ [0 , 1] 2 , k = ( k 1 , k 2 ) , w ( x ) = x

Original: + v ( x ) is the deformation, a k is randomly generated, and
Revised: + v ( x ) is the deformation, a k is randomly generated, and

Original: N = 10 (appropriate for the resolution).
Revised: N = 10 (appropriate for the resolution).

Original: The Soboev norm is (cid:107) v (cid:107) 2 = | a 0 | 2 + (cid:80) Nk
Revised: The Soboev norm is (cid:107) v (cid:107) 2 = | a 0 | 2 + (cid:80) Nk

Original: = − N | k | 2 | a k
Revised: = − N | k | 2 | a k

Original: | 2 .
Revised: | 2 .

Original: For each image in the dataset we generate 8 random deformations of varying norm (cid:107) v (cid:107) 2 from 10 to 80 in steps of 10.
Revised: For each image in the dataset we generate 8 random deformations of varying norm (cid:107) v (cid:107) 2 from 10 to 80 in steps of 10.

Original: We examine the robustness of descriptors to deformations of increasing norm by comparing the segmentation of the original and deformed images similar to the previous experiment.
Revised: We examine the robustness of descriptors to deformations of increasing norm by comparing the segmentation of the original and deformed images similar to the previous experiment.

Original: Results and qualitative samples are in Figure 3, which show that ST-DNN is more robust by large margins than competing descriptors, and the robustness over competing methods increases with increasing norm.
Revised: Results and qualitative samples are in Figure 3, which show that ST-DNN is more robust by large margins than competing descriptors, and the robustness over competing methods increases with increasing norm.

Original: Note a descriptor could be robust, but not accurate, but this is not the case for
Revised: Note a descriptor could be robust, but not accurate, but this is not the case for

Original: ST-DNN (next experiments).
Revised: ST-DNN (next experiments).

Original: [Figure]
Revised: [Figure]

Original: Comparison of ST-DNN to Standard DNNs : We segment descriptors (ST-DNN and common deep network backbones) by minimizing equation 8.
Revised: Comparison of ST-DNN to Standard DNNs : We segment descriptors (ST-DNN and common deep network backbones) by minimizing equation 8.

Original: Quantitative results are in Table 1 and qualitative samples are shown in Figure 5.
Revised: Quantitative results are in Table 1 and qualitative samples are shown in Figure 5.

Original: ST-DNN outperforms all other descriptors.
Revised: ST-DNN outperforms all other descriptors.

Original: ST-DNN hasparameters
Revised: ST-DNN hasparameters

Original: and is trained with only 128 training images of real world texture dataset.
Revised: and is trained with only 128 training images of real world texture dataset.

Original: ST-DNN is around 3 orders of magnitude smaller than standard deep networks and takes around 2 orders of magnitude less training data (e.g., FCN-ResNet101-all-ours uses 50,000 images plus augmented data and has 45 million parameters), but still outperforms these networks (see Figure 4).
Revised: ST-DNN is around 3 orders of magnitude smaller than standard deep networks and takes around 2 orders of magnitude less training data (e.g., FCN-ResNet101-all-ours uses 50,000 images plus augmented data and has 45 million parameters), but still outperforms these networks (see Figure 4).

Original: Comparison to Texture Segmentation Methods
Revised: Comparison to Texture Segmentation Methods

Original: on Real
Revised: on Real

Original: -World & Brodatz Synthetic Texture
Revised: -World & Brodatz Synthetic Texture

Original: Datasets: We compare to state-of-the-art texture segmentation methods: edge-based (gPb, MCG,
Revised: Datasets: We compare to state-of-the-art texture segmentation methods: edge-based (gPb, MCG,

Original: Kok, CB) Arbelaez et al.
Revised: Kok, CB) Arbelaez et al.

Original: (2011b)
Revised: (2011b)

Original: ; Arbeláez et al.
Revised: ; Arbeláez et al.

Original: (2014); Isola et al.
Revised: (2014); Isola et al.

Original: (2014); Kokkinos (2015)) and region based (STLD Khan et al.
Revised: (2014); Kokkinos (2015)) and region based (STLD Khan et al.

Original: and Siamese Khan & Sundaramoorthi (2018)).
Revised: and Siamese Khan & Sundaramoorthi (2018)).

Original: STLD is a shape-tailored (but hand crafted) approach using PDEs, and non-STLD uses the same PDE as
Revised: STLD is a shape-tailored (but hand crafted) approach using PDEs, and non-STLD uses the same PDE as

Original: STLD but on the whole image so is not shape-tailored.
Revised: STLD but on the whole image so is not shape-tailored.

Original: Siamese uses a neural network on the channel dimension returned by STLD, but does not layer PDE solutions as our approach.
Revised: Siamese uses a neural network on the channel dimension returned by STLD, but does not layer PDE solutions as our approach.

Original: Quantitative results for both texture datasets are in Table 2 and a few qualitative samples of the results are in Figure 5.
Revised: Quantitative results for both texture datasets are in Table 2 and a few qualitative samples of the results are in Figure 5.

Original: Our method out-performs all others by signiﬁcant margins on all regions metrics and achieves close
Revised: Our method out-performs all others by signiﬁcant margins on all regions metrics and achieves close

Original: [Figure]
Revised: [Figure]

Original: to the best result on the contour metric on RWTSD, while on the Synthetic Brodatz, our method out performs all methods on all metrics.
Revised: to the best result on the contour metric on RWTSD, while on the Synthetic Brodatz, our method out performs all methods on all metrics.

Original: More visual results and comparison to more methods is in
Revised: More visual results and comparison to more methods is in

Original: Appendix E
Revised: Appendix E

Original: [Figure]
Revised: [Figure]

Original: Higher Fmeasure for the contour metric, ground truth covering (GT-cov), and rand index indicate better ﬁt, and lower variation of information (Var.
Revised: Higher Fmeasure for the contour metric, ground truth covering (GT-cov), and rand index indicate better ﬁt, and lower variation of information (Var.

Original: Info) indicates a better ﬁt to ground truth.
Revised: Info) indicates a better ﬁt to ground truth.

Original: Comparison on Multi-Region Segmentation: We have also tested our method on two multi-region segmentation datasets, 1) BSDS500 2)
Revised: Comparison on Multi-Region Segmentation: We have also tested our method on two multi-region segmentation datasets, 1) BSDS500 2)

Original: Synthetic Texture dataset.
Revised: Synthetic Texture dataset.

Original: For BSDS500 dataset we train our methods and state of the art deep networks on the training + validation set (200+100 images augmented with with 8 rotations and 5 scales) and test on the test-set on BSDS dataset (200 images).
Revised: For BSDS500 dataset we train our methods and state of the art deep networks on the training + validation set (200+100 images augmented with with 8 rotations and 5 scales) and test on the test-set on BSDS dataset (200 images).

Original: Results are summarized in Table 3 and a few quantitative samples are show in ﬁgure 9 in Appendix E.
Revised: Results are summarized in Table 3 and a few quantitative samples are show in ﬁgure 9 in Appendix E.

Original: BSDS500 is not the best dataset for training on region segmentation as the regions are not marked based on appearance and instead watershed is used on edge maps to generate segments.
Revised: BSDS500 is not the best dataset for training on region segmentation as the regions are not marked based on appearance and instead watershed is used on edge maps to generate segments.

Original: Hence region with similar looking appearance are often marked as separate regions in images.
Revised: Hence region with similar looking appearance are often marked as separate regions in images.

Original: Despite this drawback ST-DNN performs on par or better than state of the art Deep Neural Networks.
Revised: Despite this drawback ST-DNN performs on par or better than state of the art Deep Neural Networks.

Original: During training all images are resized to 256 × 256 and normalised to have zero mean and unit variance.
Revised: During training all images are resized to 256 × 256 and normalised to have zero mean and unit variance.

Original: In post-processing, the outputs are clustered into 20 regions and then regions with less than 2% pixels
Revised: In post-processing, the outputs are clustered into 20 regions and then regions with less than 2% pixels

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original:  [Figure] [Table]
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: of the entire image are smoothed out using conditional random ﬁelds, we use pydensecrf 1 library.
Revised: of the entire image are smoothed out using conditional random ﬁelds, we use pydensecrf 1 library.

Original: Following which we run 20 iterations of 2 .
Revised: Following which we run 20 iterations of 2 .

Original: For quantitative comparisons we have provided analyses on region metrics on BSDS500 benchmark.
Revised: For quantitative comparisons we have provided analyses on region metrics on BSDS500 benchmark.

Original: We have also tested on a large scale multi-region synthetic texture segmentation dataset 2 .
Revised: We have also tested on a large scale multi-region synthetic texture segmentation dataset 2 .

Original: This dataset is designed as an extension of Khan et al.
Revised: This dataset is designed as an extension of Khan et al.

Original: , we have a maximum of 4 regions per image comprising of different appearance textures.
Revised: , we have a maximum of 4 regions per image comprising of different appearance textures.

Original: The dataset consists of 42000 training images,validation images and 5000 test images.
Revised: The dataset consists of 42000 training images,validation images and 5000 test images.

Original: The images are generated from 1084 textures collected from Brodatz dataset.
Revised: The images are generated from 1084 textures collected from Brodatz dataset.

Original: Our methods is trained on only 200 images from the dataset, other methods are trained on the entire training set.
Revised: Our methods is trained on only 200 images from the dataset, other methods are trained on the entire training set.

Original: Our method outperforms state of the art deep learning based methods by a healthy margin despite being signiﬁcantly smaller in number of parameters and training data required for training.
Revised: Our method outperforms state of the art deep learning based methods by a healthy margin despite being signiﬁcantly smaller in number of parameters and training data required for training.

Original: Similar to BSDS500 dataset we resize images to 256 × 256 and normalise them to have zero mean and unit variance.
Revised: Similar to BSDS500 dataset we resize images to 256 × 256 and normalise them to have zero mean and unit variance.

Original: In post-processing we simply cluster the image intoregions and run 20 iterations of 2 .
Revised: In post-processing we simply cluster the image intoregions and run 20 iterations of 2 .

Original: For quantitative comparison we have provided analyses on region accuracy metric provided with the dataset, motivated from PascalVOC class accuracy metric.
Revised: For quantitative comparison we have provided analyses on region accuracy metric provided with the dataset, motivated from PascalVOC class accuracy metric.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have introduced ST-DNNs, which generalize CNNs to arbitrary shaped regions by stacking layers that solve PDEs.
Revised: We have introduced ST-DNNs, which generalize CNNs to arbitrary shaped regions by stacking layers that solve PDEs.

Original: These are relevant to segmentation since they avoid aggregation of data across segmentation regions.
Revised: These are relevant to segmentation since they avoid aggregation of data across segmentation regions.

Original: They are also covariant to translations and rotations, and robust to deformations.
Revised: They are also covariant to translations and rotations, and robust to deformations.

Original: Experiments showed that ST-DNNs achieve state-of-the-art results on texture segmentation benchmarks: they outperform state-of-the-art deep networks by 20% on edge metrics and 10% on region metrics while being 3 orders of magnitude smaller and using 2 orders of magnitude less training data.
Revised: Experiments showed that ST-DNNs achieve state-of-the-art results on texture segmentation benchmarks: they outperform state-of-the-art deep networks by 20% on edge metrics and 10% on region metrics while being 3 orders of magnitude smaller and using 2 orders of magnitude less training data.

Original: Experiments also validated covariance and robustness of ST-DNNs.
Revised: Experiments also validated covariance and robustness of ST-DNNs.

Original: To show the strength of the formulation we have tested it on four datasets, which include binary and multi-region segmentation datasets.
Revised: To show the strength of the formulation we have tested it on four datasets, which include binary and multi-region segmentation datasets.

Original: A A PPENDIX : O UTLINE
Revised: A A PPENDIX : O UTLINE

Original: The Appendices material is divided into the following sections.
Revised: The Appendices material is divided into the following sections.

Original: • Analytical Proofs of Covariance and Robustness [B] • Alternative Training Method for ST-DNN With Lower Memory Requirements [C] • Implementation Details for Training and Joint Segmentation [D] • Additional Experimental Analysis and Results [E]
Revised: • Analytical Proofs of Covariance and Robustness [B] • Alternative Training Method for ST-DNN With Lower Memory Requirements [C] • Implementation Details for Training and Joint Segmentation [D] • Additional Experimental Analysis and Results [E]

Original: B A NALYTICAL P ROOFS FOR C OVARIANCE AND R OBUSTNESS
Revised: B A NALYTICAL P ROOFS FOR C OVARIANCE AND R OBUSTNESS

Original: The proof of Theorem 1 (covariance of ST-DNN to rotation and translation) follows from basic properties of the Laplace equation Evans (1998); we state these properties in terms of our language of covariant operators, and show the proof for the convenience of the reader.
Revised: The proof of Theorem 1 (covariance of ST-DNN to rotation and translation) follows from basic properties of the Laplace equation Evans (1998); we state these properties in terms of our language of covariant operators, and show the proof for the convenience of the reader.

Original: We will show Theorem(robustness of ST-DNN to domain deformations) as a consequence of properties of linear PDE theory.
Revised: We will show Theorem(robustness of ST-DNN to domain deformations) as a consequence of properties of linear PDE theory.

Original: We repeat the deﬁnition of covariant operator:
Revised: We repeat the deﬁnition of covariant operator:

Original: Deﬁnition 2 An operator T : I → I is covariant to a class W of transformations if
Revised: Deﬁnition 2 An operator T : I → I is covariant to a class W of transformations if

Original: [Equation]
Revised: [Equation]

Original: for every I ∈ I and w ∈ W .
Revised: for every I ∈ I and w ∈ W .

Original: We show covariance of the Laplace operator, and as a consequence, the covariance of the Poisson
Revised: We show covariance of the Laplace operator, and as a consequence, the covariance of the Poisson

Original: PDE and the ST-DNN.
Revised: PDE and the ST-DNN.

Original: Theorem 3 The Laplacian operator ∆ =
Revised: Theorem 3 The Laplacian operator ∆ =

Original: (cid:80) i ∂
Revised: (cid:80) i ∂

Original: 2 ∂x 2 i is covariant to rotations, x → Rx where R is a rotation matrix.
Revised: 2 ∂x 2 i is covariant to rotations, x → Rx where R is a rotation matrix.

Original: Proof 1 Let u ∈
Revised: Proof 1 Let u ∈

Original: I
Revised: I

Original: , and let R be a rotation.
Revised: , and let R be a rotation.

Original: Consider
Revised: Consider

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Original: so,
Revised: so,

Original: [Equation]
Revised: [Equation]

Original: so ,
Revised: so ,

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where the second last equality is due to the invariance of the trace to similarity transformations.
Revised: where the second last equality is due to the invariance of the trace to similarity transformations.

Original: Theorem 4 The Laplacian operator is covariant to translations, x → x + t , where t is a vector.
Revised: Theorem 4 The Laplacian operator is covariant to translations, x → x + t , where t is a vector.

Original: Proof 2 We have,
Revised: Proof 2 We have,

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus, ∆( u ◦ T )( x ) =
Revised: Thus, ∆( u ◦ T )( x ) =

Original: (∆ u ) ◦ T where T ( x ) = x + t .
Revised: (∆ u ) ◦ T where T ( x ) = x + t .

Original: Corollary 1 The solution u of the Poisson equation, i.e.,
Revised: Corollary 1 The solution u of the Poisson equation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: (22) u
Revised: (22) u

Original: = T [ I ] is covariant to translations and rotations.
Revised: = T [ I ] is covariant to translations and rotations.

Original: Proof 

Revised: Proof

Original: This follows from the covariance of the Laplacian, and the identity map.
Revised: This follows from the covariance of the Laplacian, and the identity map.

Original: We can now show covariance of the ST-DNN to translations and rotations.
Revised: We can now show covariance of the ST-DNN to translations and rotations.

Original: Theorem 5 The ST-DNN equation 3 is covariant to translations and rotations.
Revised: Theorem 5 The ST-DNN equation 3 is covariant to translations and rotations.

Original: Proof 4 Since ST-DNNs are composition of solutions of Poisson Equations with fully connected layer across channels and non-linearity in multiple layers.
Revised: Proof 4 Since ST-DNNs are composition of solutions of Poisson Equations with fully connected layer across channels and non-linearity in multiple layers.

Original: A linear combination of channels and point-wise non-linear operation preserves the covariance of rotation and translations, hence the
Revised: A linear combination of channels and point-wise non-linear operation preserves the covariance of rotation and translations, hence the

Original: ST-DNNs are covariant to translations and rotations.
Revised: ST-DNNs are covariant to translations and rotations.

Original: We now show robustness of the ST-DNN to domain deformations with respect to the Sobolev norm; that is, we show that the output of a ST-DNN layer does not change much if deformed by a transformation with small Sobolev norm (a smooth transformation that has small displacement).
Revised: We now show robustness of the ST-DNN to domain deformations with respect to the Sobolev norm; that is, we show that the output of a ST-DNN layer does not change much if deformed by a transformation with small Sobolev norm (a smooth transformation that has small displacement).

Original: The
Revised: The

Original: Sobolev measures the smoothness of the deformation, i.e., the L 2 norm of the deformation and the gradient of the deformation.
Revised: Sobolev measures the smoothness of the deformation, i.e., the L 2 norm of the deformation and the gradient of the deformation.

Original: We have the following theorem:
Revised: We have the following theorem:

Original: Theorem 6 The solution of the ST-DNN is robust to deformations, i.e.,
Revised: Theorem 6 The solution of the ST-DNN is robust to deformations, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where T is the mapping from the image to the solution of the ST-DNN, w : Ω → Ω is a domain deformation, id is the identity map, and H 1 indicates the Sobolev norm.
Revised: where T is the mapping from the image to the solution of the ST-DNN, w : Ω → Ω is a domain deformation, id is the identity map, and H 1 indicates the Sobolev norm.

Original: Proof 

Revised: Proof

Original: This is a consequence of Lemma equation 1 below, which shows that each layer of the
Revised: This is a consequence of Lemma equation 1 below, which shows that each layer of the

Original: ST-DNN is robust.
Revised: ST-DNN is robust.

Original: Stacking such layers preserves the robustness by applying Lemma 1 successively.
Revised: Stacking such layers preserves the robustness by applying Lemma 1 successively.

Original: We now prove robustness of layers of ST-DNN.
Revised: We now prove robustness of layers of ST-DNN.

Original: For convenience in the proof, we assume the input operates on the domain Ω = R 2 , which avoids having to consider the boundary and more complicated formulas that do not affect the essence of the argument.
Revised: For convenience in the proof, we assume the input operates on the domain Ω = R 2 , which avoids having to consider the boundary and more complicated formulas that do not affect the essence of the argument.

Original: Let
Revised: Let

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: be a layer of ST-DNN.
Revised: be a layer of ST-DNN.

Original: Here r is the rectiﬁed linear unit, K is the kernel representing, the Green’s function of the Poisson equation (we assume for this proof that the domain of the image is all of R 2 ),
Revised: Here r is the rectiﬁed linear unit, K is the kernel representing, the Green’s function of the Poisson equation (we assume for this proof that the domain of the image is all of R 2 ),

Original: D is the derivative operator representing oriented gradients (or parial derivatives of an ﬁnite order), and L is a weight matrix of fully connected layer across channels.
Revised: D is the derivative operator representing oriented gradients (or parial derivatives of an ﬁnite order), and L is a weight matrix of fully connected layer across channels.

Original: We state the robustness of a layer of ST-DNN:
Revised: We state the robustness of a layer of ST-DNN:

Original: Lemma 1 A layer, f
Revised: Lemma 1 A layer, f

Original: [ I ] =
Revised: [ I ] =

Original: r [( L ◦ D ◦ K ∗ I ] , of a ST-DNN is Lipschitz continuous with respect to diffeomorphisms in the Sobolev norm, i.e.
Revised: r [( L ◦ D ◦ K ∗ I ] , of a ST-DNN is Lipschitz continuous with respect to diffeomorphisms in the Sobolev norm, i.e.

Original: [Equation]
Revised: [Equation]

Original: where id ( x ) = x is the identity map, and C is a constant (independent of w and only of function of the class of images), and (cid:107) w (cid:107) 2 H 1 = (cid:82) Ω
Revised: where id ( x ) = x is the identity map, and C is a constant (independent of w and only of function of the class of images), and (cid:107) w (cid:107) 2 H 1 = (cid:82) Ω

Original: ( | w ( x ) | + |∇ w ( x ) | 2 ) dx .
Revised: ( | w ( x ) | + |∇ w ( x ) | 2 ) dx .

Original: Note that w − id is the displacement.
Revised: Note that w − id is the displacement.

Original: Proof 6 Let w be a smooth diffeomorphism.
Revised: Proof 6 Let w be a smooth diffeomorphism.

Original: Then by Lipschitz continuity of the ReLu,
Revised: Then by Lipschitz continuity of the ReLu,

Original: [Equation]
Revised: [Equation]

Original: where M = L ◦ D ◦ K .
Revised: where M = L ◦ D ◦ K .

Original: Note that by a change of variables,
Revised: Note that by a change of variables,

Original: [Equation]
Revised: [Equation]

Original: Note that the determinent of the Jacobian appears if we weight the sum by the area measure, which approximates the integral.
Revised: Note that the determinent of the Jacobian appears if we weight the sum by the area measure, which approximates the integral.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: We let w ( y ) =
Revised: We let w ( y ) =

Original: y + v ( y ) .
Revised: y + v ( y ) .

Original: This gives us
Revised: This gives us

Original: [Equation]
Revised: [Equation]

Original: We may bound the second term as
Revised: We may bound the second term as

Original: [Equation]
Revised: [Equation]

Original: by basic inequalities.
Revised: by basic inequalities.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: By Lipschitz continuity of the Poisson kernel and derivatives
Revised: By Lipschitz continuity of the Poisson kernel and derivatives

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the Poisson kernel has a singularity at the origin, so the statement is not precise; however, as common in PDE analysis, as we will below compute integrals of the left hand quantity, we can break the integral into two terms one that integrates the singularity in a small ball (which is ﬁnite) and the other that integrates the right hand side, that we analyze below.
Revised: Note that the Poisson kernel has a singularity at the origin, so the statement is not precise; however, as common in PDE analysis, as we will below compute integrals of the left hand quantity, we can break the integral into two terms one that integrates the singularity in a small ball (which is ﬁnite) and the other that integrates the right hand side, that we analyze below.

Original: The former will disappear to zero as the ball goes to zero.
Revised: The former will disappear to zero as the ball goes to zero.

Original: We omit the details to avoid hiding the main argument.
Revised: We omit the details to avoid hiding the main argument.

Original: We also have that
Revised: We also have that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: For a multi layer multiple layers network with N layers we will have:
Revised: For a multi layer multiple layers network with N layers we will have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: C A LTERNATIVE T RAINING M ETHOD FOR ST-DNN
Revised: C A LTERNATIVE T RAINING M ETHOD FOR ST-DNN

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we provide an alternative algorithm for training ST-DNN that has a smaller memory footprint than the method present in the main paper to avoid having to train on small images.
Revised: we provide an alternative algorithm for training ST-DNN that has a smaller memory footprint than the method present in the main paper to avoid having to train on small images.

Original: Unfortunately, this is not supported in current deep learning packages, so it requires a custom implementation, and it may be more computationally costly depending on the size of the image.
Revised: Unfortunately, this is not supported in current deep learning packages, so it requires a custom implementation, and it may be more computationally costly depending on the size of the image.

Original: The basic idea is instead of back propagating (through large matrices to solve the PDE, and storing matrix multiplies) to determine derivatives of the loss with respect to the weights, we instead forward propagate the derivatives of each layer through variations of the rest of the layers (see Figure 6).
Revised: The basic idea is instead of back propagating (through large matrices to solve the PDE, and storing matrix multiplies) to determine derivatives of the loss with respect to the weights, we instead forward propagate the derivatives of each layer through variations of the rest of the layers (see Figure 6).

Original: The advantage of this method is that the variations of the layers with PDEs can be computed by solving the PDE itself with input being the layer weight derivative.
Revised: The advantage of this method is that the variations of the layers with PDEs can be computed by solving the PDE itself with input being the layer weight derivative.

Original: This can be solved with an iterative technique.
Revised: This can be solved with an iterative technique.

Original: This does not require accumulating matrix multiplies consuming large memory as standard back propagation, as forward propagating through the PDE is analogous to a matrix-vector multiply, which as solved through an iterative technique and does not need to store the matrix explicitly.
Revised: This does not require accumulating matrix multiplies consuming large memory as standard back propagation, as forward propagating through the PDE is analogous to a matrix-vector multiply, which as solved through an iterative technique and does not need to store the matrix explicitly.

Original: Below, we present the mathematics to show that this forward propagation technique is valid for the case of PDEs as layers, and give details of the technique.
Revised: Below, we present the mathematics to show that this forward propagation technique is valid for the case of PDEs as layers, and give details of the technique.

Original: C. V ARIATION OF S HAPE -T AILORED D ESCRIPTORS
Revised: C. V ARIATION OF S HAPE -T AILORED D ESCRIPTORS

Original: The shape-tailored descriptors are given by a Poisson equation with Neumann boundary conditions.
Revised: The shape-tailored descriptors are given by a Poisson equation with Neumann boundary conditions.

Original: [Equation]
Revised: [Equation]

Original: To take the variation of the PDEs we take the variation of each partial differential equation.
Revised: To take the variation of the PDEs we take the variation of each partial differential equation.

Original: To ﬁnd the variation of the ﬁrst equation we ﬁrst calculate the value of u ( x ) when it’s input is perturbed by δI
Revised: To ﬁnd the variation of the ﬁrst equation we ﬁrst calculate the value of u ( x ) when it’s input is perturbed by δI

Original: [Equation]
Revised: [Equation]

Original: The variation in the direction of δI is deﬁned as
Revised: The variation in the direction of δI is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Next, we replace this value in the equation for the variation and get
Revised: Next, we replace this value in the equation for the variation and get

Original: [Equation]
Revised: [Equation]

Original: Similarly for the second term, we have
Revised: Similarly for the second term, we have

Original: [Equation]
Revised: [Equation]

Original: So the variation of the shape-tailored descriptor denoted by u h ( x ) = δu.δI ( x ) in the direction of a perturbation δI ( x ) of the input image
Revised: So the variation of the shape-tailored descriptor denoted by u h ( x ) = δu.δI ( x ) in the direction of a perturbation δI ( x ) of the input image

Original: I ( x ) is given by:
Revised: I ( x ) is given by:

Original: [Equation]
Revised: [Equation]

Original: C. D ERIVATIVE OF E NERGY WITH RESPECT TO W EIGHTS
Revised: C. D ERIVATIVE OF E NERGY WITH RESPECT TO W EIGHTS

Original: First, we ﬁnd the variation of a layer of the shape-tailored network.
Revised: First, we ﬁnd the variation of a layer of the shape-tailored network.

Original: A layer of shape-tailored network in composed of r ◦ L ◦ T .
Revised: A layer of shape-tailored network in composed of r ◦ L ◦ T .

Original: The expression for variation of a layer of the network is given below:
Revised: The expression for variation of a layer of the network is given below:

Original: [Equation]
Revised: [Equation]

Original: where,
Revised: where,

Original: [Equation]
Revised: [Equation]

Original: Next, we calculate the variation through the entire network.
Revised: Next, we calculate the variation through the entire network.

Original: [Equation]
Revised: [Equation]

Original: where, ˜ F i is the output of the i th layer of the network.
Revised: where, ˜ F i is the output of the i th layer of the network.

Original: Now, we derive the expression for derivative of the energy w.r.t to the weights of the network.
Revised: Now, we derive the expression for derivative of the energy w.r.t to the weights of the network.

Original: These derivative would be used to update the weights of the network during the application of the stochastic gradient decent to the energy function.
Revised: These derivative would be used to update the weights of the network during the application of the stochastic gradient decent to the energy function.

Original: To start with
Revised: To start with

Original: [Equation]
Revised: [Equation]

Original: we derive the derivative of the descriptor w.r
Revised: we derive the derivative of the descriptor w.r

Original: [Equation]
Revised: [Equation]

Original: t. the weights of the network ( w i are weights b i are biases of the fully connected layer).
Revised: t. the weights of the network ( w i are weights b i are biases of the fully connected layer).

Original: The derivative w.r.t.
Revised: The derivative w.r.t.

Original: to the weights of the last layer of the network are:
Revised: to the weights of the last layer of the network are:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˜ F i represent the output of the network at the i th layer.
Revised: where ˜ F i represent the output of the network at the i th layer.

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we calculate the derivative of
Revised: we calculate the derivative of

Original: F w.r.t to w i and b i , where w i and b i are the weights and biases of the fully connected layers
Revised: F w.r.t to w i and b i , where w i and b i are the weights and biases of the fully connected layers

Original: [Equation]
Revised: [Equation]

Original: and i represents intermediate layer of the network
Revised: and i represents intermediate layer of the network

Original: [Equation]
Revised: [Equation]

Original: similarly, ∂F
Revised: similarly, ∂F

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: where < ., .
Revised: where < ., .

Original: > L 2 represent the inner produce of two vectors.
Revised: > L 2 represent the inner produce of two vectors.

Original: Now, that we have the derivatives of the energy w.r.t.
Revised: Now, that we have the derivatives of the energy w.r.t.

Original: the weights of the network, we can train the shape-tailored networks for applications like segmentation (see Figure 6 for a visualization of the forward pass for gradients calculation).
Revised: the weights of the network, we can train the shape-tailored networks for applications like segmentation (see Figure 6 for a visualization of the forward pass for gradients calculation).

Original: D I MPLEMENTATION D ETAILS
Revised: D I MPLEMENTATION D ETAILS

Original: In this section we outline the important implementation details for our experiments.
Revised: In this section we outline the important implementation details for our experiments.

Original: Figure 7 shows a schematic of the ST-DNN we use in our experiments.
Revised: Figure 7 shows a schematic of the ST-DNN we use in our experiments.

Original: D. ST-DNN SETUP
Revised: D. ST-DNN SETUP

Original: We setup two versions of ST-DNN, one with MatConvNet where we use a C++ wrapper for the solution of Poisson equation of Eq 35 and a faster version in Pytorch.
Revised: We setup two versions of ST-DNN, one with MatConvNet where we use a C++ wrapper for the solution of Poisson equation of Eq 35 and a faster version in Pytorch.

Original: In the MatConvNet version the
Revised: In the MatConvNet version the

Original: Poisson equation with Neumann boundary condition is solved with conjugate gradient algorithm, and in Pytorch version, we explicitly invert the right hand side of Equation 35 to ﬁnd the solution.
Revised: Poisson equation with Neumann boundary condition is solved with conjugate gradient algorithm, and in Pytorch version, we explicitly invert the right hand side of Equation 35 to ﬁnd the solution.

Original: We noticed that training by solving the PDEs with (conjugate gradient algorithm on full size images)
Revised: We noticed that training by solving the PDEs with (conjugate gradient algorithm on full size images)

Original: MatConvNet was slow compared to the faster PyTorch version, and it didn’t have support available in PyTorch and TensorFlow and the results for the faster Pytorch solution (trained on downsampled images) were almost on par with the MatConvNet version (at least for the datasets we have tested on), however, learning ST-DNNs from full size images might be helpful in other applications and future work.
Revised: MatConvNet was slow compared to the faster PyTorch version, and it didn’t have support available in PyTorch and TensorFlow and the results for the faster Pytorch solution (trained on downsampled images) were almost on par with the MatConvNet version (at least for the datasets we have tested on), however, learning ST-DNNs from full size images might be helpful in other applications and future work.

Original: The two methods are different in how they handle the solution of the PDEs.
Revised: The two methods are different in how they handle the solution of the PDEs.

Original: Below we provide the details of both methods.
Revised: Below we provide the details of both methods.

Original: As a ﬁrst step, we provide the discretisation of the Poisson Equation 35:
Revised: As a ﬁrst step, we provide the discretisation of the Poisson Equation 35:

Original: [Equation]
Revised: [Equation]

Original: where N ( i, j ) represents the 4-pixel neighborhood of pixel ( i, j ) , which represents the i th row and j th column, and intersection means that only neighbors in the region are considered as implied by the
Revised: where N ( i, j ) represents the 4-pixel neighborhood of pixel ( i, j ) , which represents the i th row and j th column, and intersection means that only neighbors in the region are considered as implied by the

Original: Neumann boundary condition in the PDE, avoiding aggregation outside R .
Revised: Neumann boundary condition in the PDE, avoiding aggregation outside R .

Original: For small images, we can linearise the image and write the above equations as:
Revised: For small images, we can linearise the image and write the above equations as:

Original: [Equation]
Revised: [Equation]

Original: where, u and I are linearised image and descriptor respectively.
Revised: where, u and I are linearised image and descriptor respectively.

Original: We can then ﬁnd the solution by simply inverting the system of equations
Revised: We can then ﬁnd the solution by simply inverting the system of equations

Original: [Equation]
Revised: [Equation]

Original: Notice that size of A R is ( m ∗ n, m ∗ n ) , where m and n are number of rows and columns in the image respectively and A R depends of the mask of region R .
Revised: Notice that size of A R is ( m ∗ n, m ∗ n ) , where m and n are number of rows and columns in the image respectively and A R depends of the mask of region R .

Original: For small images (32 × 32) the size of A R is (256 × 256) and we can store and invert it effectively, hence we use small images for training in our Pytorch implementation.
Revised: For small images (32 × 32) the size of A R is (256 × 256) and we can store and invert it effectively, hence we use small images for training in our Pytorch implementation.

Original: Also, notice that if we implement the PDEs solution as in
Revised: Also, notice that if we implement the PDEs solution as in

Original: Equation 51, Pytorch (autograd) takes care of the back-propagation through the PDE layers and we do not have to explicitly implement the back-propagation (see Algorithm 1).
Revised: Equation 51, Pytorch (autograd) takes care of the back-propagation through the PDE layers and we do not have to explicitly implement the back-propagation (see Algorithm 1).

Original: For full size images we had to implement the equations of Section C, where we want to solve the
Revised: For full size images we had to implement the equations of Section C, where we want to solve the

Original: PDEs without explicitly storing the smoothing kernel for each pixel in the image (as it would have huge memory requirement).
Revised: PDEs without explicitly storing the smoothing kernel for each pixel in the image (as it would have huge memory requirement).

Original: We can solve Equation 49 using conjugate gradient algorithm as A in Equation 50 is symmetric positive deﬁnite (more detail in Khan et al.
Revised: We can solve Equation 49 using conjugate gradient algorithm as A in Equation 50 is symmetric positive deﬁnite (more detail in Khan et al.

Original: Notice however that, if we use this approach we can not use autograd algorithms of standard packages like PyTorch and TensorFlow because Equation 50 is the relations from output to input (and not from input to output), so we have to implement the gradient decent equations by calculating equations of Section C explicitly.
Revised: Notice however that, if we use this approach we can not use autograd algorithms of standard packages like PyTorch and TensorFlow because Equation 50 is the relations from output to input (and not from input to output), so we have to implement the gradient decent equations by calculating equations of Section C explicitly.

Original: We have implemented this using MatConvNet, however, we see that the results (at least on the datasets we have tested on) of training on small scale images are on par with results of full size image.
Revised: We have implemented this using MatConvNet, however, we see that the results (at least on the datasets we have tested on) of training on small scale images are on par with results of full size image.

Original: Hence in the paper we have reported results from the Pytorch implementation.
Revised: Hence in the paper we have reported results from the Pytorch implementation.

Original: The full implementation, however, might be useful in other applications and future work.
Revised: The full implementation, however, might be useful in other applications and future work.

Original: D. I NFERENCE (J OINT S EGMENTATION ) A LGORITHM
Revised: D. I NFERENCE (J OINT S EGMENTATION ) A LGORITHM

Original: We present the details of the algorithm for segmentation and joint ST-DNN descriptor updates (Algorithm 2).
Revised: We present the details of the algorithm for segmentation and joint ST-DNN descriptor updates (Algorithm 2).

Original: We use an approach analogous to level set methods, but instead evolve functions φ i : Ω → R that are smooth indicator functions of the regions R i .
Revised: We use an approach analogous to level set methods, but instead evolve functions φ i : Ω → R that are smooth indicator functions of the regions R i .

Original: The gradients of the energy with respect to the regions can be converted into an evolution of the indicator functions as shown in the algorithm.
Revised: The gradients of the energy with respect to the regions can be converted into an evolution of the indicator functions as shown in the algorithm.

Original: Note that κ i is the curvature of the boundary of region R i , and can be written in terms of φ .
Revised: Note that κ i is the curvature of the boundary of region R i , and can be written in terms of φ .

Original: Compute
Revised: Compute

Original: Compute G i evaluated from the neural network.
Revised: Compute G i evaluated from the neural network.

Original: Update pixels x ∈ D
Revised: Update pixels x ∈ D

Original: ( R i )
Revised: ( R i )

Original: ∩ D ( R j ) as follows:
Revised: ∩ D ( R j ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Update all other pixels as
Revised: Update all other pixels as

Original: [Equation]
Revised: [Equation]

Original: Clip between 0 and 1: φ i = max { 0 , min { 1 , φ i }} .
Revised: Clip between 0 and 1: φ i = max { 0 , min { 1 , φ i }} .

Original: 11: until regions have converged
Revised: 11: until regions have converged

Original: E A DDITIONAL E XPERIMENTS AND R ESULTS E. A BLATION S TUDIES
Revised: E A DDITIONAL E XPERIMENTS AND R ESULTS E. A BLATION S TUDIES

Original: In Table 4, we show ablation studies of ST-DNN with varying number of layers and compare it against
Revised: In Table 4, we show ablation studies of ST-DNN with varying number of layers and compare it against

Original: Siamese Khan & Sundaramoorthi (2018)
Revised: Siamese Khan & Sundaramoorthi (2018)

Original: (that takes hand-crafted shape-tailored descriptors as input and then uses a neural network in the channel dimension) on the Real-World Texture Segmentation
Revised: (that takes hand-crafted shape-tailored descriptors as input and then uses a neural network in the channel dimension) on the Real-World Texture Segmentation

Original: Dataset.
Revised: Dataset.

Original: We test the performance against the number of layers in two settings.
Revised: We test the performance against the number of layers in two settings.

Original: First, for both ST-DNN and Siamese, we compute the descriptors, choosing only one region that contains the whole image, i.e., R 0 =
Revised: First, for both ST-DNN and Siamese, we compute the descriptors, choosing only one region that contains the whole image, i.e., R 0 =

Original: Ω to initially compute the descriptors.
Revised: Ω to initially compute the descriptors.

Original: We do not update the descriptors as the regions of segmentation evolve to minimize the energy.
Revised: We do not update the descriptors as the regions of segmentation evolve to minimize the energy.

Original: Secondly, we update the descriptors as the regions evolve.
Revised: Secondly, we update the descriptors as the regions evolve.

Original: This is done to show layering more PDE layers increases performance independent of the joint segmentation/descriptor approach.
Revised: This is done to show layering more PDE layers increases performance independent of the joint segmentation/descriptor approach.

Original: Results under both settings (ﬁrst setting on the left and second setting on the right) are shown in Table 4.
Revised: Results under both settings (ﬁrst setting on the left and second setting on the right) are shown in Table 4.

Original: They show that increasing the number of layers in
Revised: They show that increasing the number of layers in

Original: ST-DNN under both settings increases performance up to 4 layers, and then performance decreases (probably due to overﬁtting with too many parameters after 4 layers).
Revised: ST-DNN under both settings increases performance up to 4 layers, and then performance decreases (probably due to overﬁtting with too many parameters after 4 layers).

Original: Thus, the performance of
Revised: Thus, the performance of

Original: ST-DNN is due to layering multiple (shape-tailored) spatial ﬁltering layers, one of the contributions of this work.
Revised: ST-DNN is due to layering multiple (shape-tailored) spatial ﬁltering layers, one of the contributions of this work.

Original: The fact that performance increases when updating the descriptors as the region evolves, which makes the descriptors shape-tailored to the segmentation, shows that the shape-tailored aspect of the descriptor is essential.
Revised: The fact that performance increases when updating the descriptors as the region evolves, which makes the descriptors shape-tailored to the segmentation, shows that the shape-tailored aspect of the descriptor is essential.

Original: In comparison to Siamese (result reported on 4 layers processing channels of the shape-tailored pre-processing), with four layers ST-DNN out-performs it on all region metrics and performs on par on the contour metric.
Revised: In comparison to Siamese (result reported on 4 layers processing channels of the shape-tailored pre-processing), with four layers ST-DNN out-performs it on all region metrics and performs on par on the contour metric.

Original: Note that this makes perfect sense, since stacking shape-tailored spatial ﬁltering layers smooths the data more than Siamese (which uses only one spatial ﬁltering layer), making it slightly more difﬁcult to localize the boundary, but the large effective spatial receptive size from layering spatial ﬁltering in ST-DNN allows capturing regional properties better, leading to better performance on the region metrics.
Revised: Note that this makes perfect sense, since stacking shape-tailored spatial ﬁltering layers smooths the data more than Siamese (which uses only one spatial ﬁltering layer), making it slightly more difﬁcult to localize the boundary, but the large effective spatial receptive size from layering spatial ﬁltering in ST-DNN allows capturing regional properties better, leading to better performance on the region metrics.

Original: [Table]
Revised: [Table]

Original: E. D EFORMATION R OBUSTNESS
Revised: E. D EFORMATION R OBUSTNESS

Original: We provide more detailed results from the deformation robustness experiments in the main paper.
Revised: We provide more detailed results from the deformation robustness experiments in the main paper.

Original: Figure 8 shows the plots of the performance of ST-DNN against SOTA deep learning methods, showing that ST-DNN outperforms SOTA DL methods in terms of robustness, and the margin of out-performance increases with increasing deformation amount (measured by the Sobolev norm).
Revised: Figure 8 shows the plots of the performance of ST-DNN against SOTA deep learning methods, showing that ST-DNN outperforms SOTA DL methods in terms of robustness, and the margin of out-performance increases with increasing deformation amount (measured by the Sobolev norm).

Original: The robustness is measured through 3 metrics (GT-Cov, Rand Index and Variation of Information).
Revised: The robustness is measured through 3 metrics (GT-Cov, Rand Index and Variation of Information).

Original: The ﬁrst two metrics higher values indicate more robustness, and the last lower values indicate more robustness.
Revised: The ﬁrst two metrics higher values indicate more robustness, and the last lower values indicate more robustness.

Original: Figure 9 shows some visual results of the experiment.
Revised: Figure 9 shows some visual results of the experiment.

Original: E. ST-DNN P ERFORMANCE AGAINST C OMPETING M ETHODS
Revised: E. ST-DNN P ERFORMANCE AGAINST C OMPETING M ETHODS

Original: We show some additional qualitative samples of our results to better motivate our experiments.
Revised: We show some additional qualitative samples of our results to better motivate our experiments.

Original: Figures 10 - 12, shows additional samples of results, where we compare our ST-DNN setup with state-of-the-art deep architectures on the Real-World Texture Segmentation Dataset (RWTSD).
Revised: Figures 10 - 12, shows additional samples of results, where we compare our ST-DNN setup with state-of-the-art deep architectures on the Real-World Texture Segmentation Dataset (RWTSD).

Original: As can be seen from the samples our results are comfortably better than the SOTA deep architectures.
Revised: As can be seen from the samples our results are comfortably better than the SOTA deep architectures.

Original: Tableshows quantitative resuls on RWTSD.
Revised: Tableshows quantitative resuls on RWTSD.

Original: In our notation resnet101/deeplabv3-x-y, resnet101/deeplabv represents the back-bone architecture used, x represents the dataset used in training, ’d’ for DUTS dataset ﬁne-tuned on real-world texture dataset, ’m’ for MSRA dataset ﬁne-tuned on real-world texture dataset and ’all’ for a combination of all datasets in Table 6 and RWTSD without any ﬁnetuning.
Revised: In our notation resnet101/deeplabv3-x-y, resnet101/deeplabv represents the back-bone architecture used, x represents the dataset used in training, ’d’ for DUTS dataset ﬁne-tuned on real-world texture dataset, ’m’ for MSRA dataset ﬁne-tuned on real-world texture dataset and ’all’ for a combination of all datasets in Table 6 and RWTSD without any ﬁnetuning.

Original: With ’y’ we represent the loss function used to train the network, ’ce’ represents cross-entropy loss and ’ours’ represents the loss function we have presented in this paper.
Revised: With ’y’ we represent the loss function used to train the network, ’ce’ represents cross-entropy loss and ’ours’ represents the loss function we have presented in this paper.

Original: We have tested state-ofthe-art deep learning architecture with the loss introduced in this paper to show that although our loss function improves the results compared to cross-entropy, the bulk of improvement of ST-DNN over state-of-the-art networks comes from the construction of shape-tailored covariant descriptors.
Revised: We have tested state-ofthe-art deep learning architecture with the loss introduced in this paper to show that although our loss function improves the results compared to cross-entropy, the bulk of improvement of ST-DNN over state-of-the-art networks comes from the construction of shape-tailored covariant descriptors.

Original: [Figure]
Revised: [Figure]

Original: Figures 13 and 14 represent the visual samples (compared to other texture segmentation methods) on the Real-World Texture Segmentation Dataset and Synthetic Brodatz Texture Segmentation Dataset, respectively.
Revised: Figures 13 and 14 represent the visual samples (compared to other texture segmentation methods) on the Real-World Texture Segmentation Dataset and Synthetic Brodatz Texture Segmentation Dataset, respectively.

Original: Again, our results are better.
Revised: Again, our results are better.

Original: Additional quantitative results are provided in Tables 7 (for
Revised: Additional quantitative results are provided in Tables 7 (for

Original: RWTSD) and 8 (for SBTSD), comparing against more methods than in the main paper
Revised: RWTSD) and 8 (for SBTSD), comparing against more methods than in the main paper

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Table] [Table] F A DDITIONAL C OMMENTS • Choice of Poisson Equation : We have chosen Poisson equation in ST-DNN because it is a linear PDE and can be efﬁciently solved, even for large images since the matrix A in Au

Original: [Table]
Revised: = I is symmetric positive deﬁnite we can use conjugate gradient algorithm to solve for u efﬁciently. However, other PDEs like Heat equation can also be used with our formulation. Pre-processing The zeroth layer (pre-processing) layer of our ST-DNN extracts color, grayscale and oriented gradient channels at multiple scales. We have used this design choice because oriented gradient are shown to be important for textural appearance of segments Khan et al.

Original: [Table]
Revised: ; Sifre & Mallat (2013). • Case for Shape-Tailored descriptors: In Khan et al. the authors show the effect of aggregation of statistics across region boundaries. They show a marked improvement in the performance of the descriptors by simply tailoring it to the region of interest.

Original: [Table]
Revised: [Table]

Revised: caption for details on the measures.

Original: [Table]
Revised: [Table]