Original: Shape-Tailored Deep Neural Networks With PDEs
Revised: Shape-Tailored Deep Neural Networks With PDEs

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We present Shape-Tailored Deep Neural Networks (ST-DNN).
Revised: We present Shape-Tailored Deep Neural Networks (ST-DNN).

Original: ST-DNN are deepnetworks formulated through the use of partial differential equations (PDE) to bedeﬁned on arbitrarily shaped regions.
Revised: ST-DNN are deep networks formulated through the use of partial differential equations (PDE) to be deﬁned on arbitrarily shaped regions.

Original: This is natural for problems in computervision such as segmentation, where descriptors should describe regions (e.g., ofobjects) that have diverse shape.
Revised: This is natural for problems in computer vision such as segmentation, where descriptors should describe regions (e.g., of objects) that have diverse shape.

Original: We formulate ST-DNNs through the Poisson PDE,which can be used to generalize convolution to arbitrary regions.
Revised: We formulate ST-DNNs through the Poisson PDE, which can be used to generalize convolution to arbitrary regions.

Original: We stack multiple
Revised: We stack multiple

Original: PDE layers to generalize a deep CNN to arbitrarily shaped regions.
Revised: PDE layers to generalize a deep CNN to arbitrarily shaped regions.

Original: We show that
Revised: We show that

Original: ST-DNN are provably covariant to translations and rotations and robust to domaindeformations, which are important properties for computer vision tasks.
Revised: ST-DNN are provably covariant to translations and rotations and robust to domain deformations, which are important properties for computer vision tasks.

Original: We showproof-of-concept empirical validation
Revised: We show proof-of-concept empirical validation.

Original:  Introduction
Revised:  Introduction

Original: Poisson Partial differential equations (PDEs) have desirable robustness and covariance properties andcan be tailored to region of interest, which are not present in modern convolutional neural networks(CNNs).
Revised: Poisson Partial differential equations (PDEs) have desirable robustness and covariance properties and can be tailored to region of interest, which are not present in modern convolutional neural networks (CNNs).

Original: In this work we try to bridge this gap by introducing PDEs in CNNs architecture.
Revised: In this work we try to bridge this gap by introducing PDEs in CNNs architecture.

Original: To showthe advantage of these properties, we show their utility in segmentation.
Revised: To show the advantage of these properties, we show their utility in segmentation.

Original: CNNs have been used extensively for segmentation problems in computer vision (10; 11; 6; 23).
Revised: CNNs have been used extensively for segmentation problems in computer vision (10; 11; 6; 23).

Original: CNNs provide a framework for learning descriptors that are able to discriminate different texturedor semantic regions within images.
Revised: CNNs provide a framework for learning descriptors that are able to discriminate different textured or semantic regions within images.

Original: Much progress has been made in segmentation with CNNs butresults are still far from human performance.
Revised: Much progress has been made in segmentation with CNNs but results are still far from human performance.

Original: Also, signiﬁcant engineering must be performed toadapt CNNs to segmentation problems.
Revised: Also, signiﬁcant engineering must be performed to adapt CNNs to segmentation problems.

Original: A basic component in the architecture for segmentationproblems involves labeling or grouping dense descriptors returned by a backbone CNN.
Revised: A basic component in the architecture for segmentation problems involves labeling or grouping dense descriptors returned by a backbone CNN.

Original: A difﬁcultyin grouping these descriptors arises, especially near the boundaries of segmentation regions, as CNNdescriptors aggregate data from ﬁxed shape (square neighborhoods) at each pixel and may thusaggregate data from different segmentation regions.
Revised: A difﬁculty in grouping these descriptors arises, especially near the boundaries of segmentation regions, as CNN descriptors aggregate data from ﬁxed shape (square neighborhoods) at each pixel and may thus aggregate data from different segmentation regions.

Original: This makes grouping these descriptors into aunique region difﬁcult, which often results in errors in the grouping.
Revised: This makes grouping these descriptors into a unique region difﬁcult, which often results in errors in the grouping.

Original: Our contributions are speciﬁcally (see Figure 1): 1) We construct and show how to train ST-DNN,deep networks that perform shape-tailored spatial ﬁltering via the Poisson PDE at each depth soas to generalize a CNN to arbitrarily shaped regions.
Revised: Our contributions are speciﬁcally (see Figure 1): 1) We construct and show how to train ST-DNN, deep networks that perform shape-tailored spatial ﬁltering via the Poisson PDE at each depth so as to generalize a CNN to arbitrarily shaped regions.

Original: We show analytically and empirically that
Revised: We show analytically and empirically that

Original: ST-DNNs are covariant to translations and rotations as they inherit this property from the Poisson
Revised: ST-DNNs are covariant to translations and rotations as they inherit this property from the Poisson

Original: PDE.
Revised: PDE.

Original: In segmentation, covariance (a.k.a., equivariance) to translation and rotation is a desiredproperty: if a segment in an image is found, then the corresponding segment should be found in thetranslated / rotated image (or object).
Revised: In segmentation, covariance (a.k.a., equivariance) to translation and rotation is a desired property: if a segment in an image is found, then the corresponding segment should be found in the translated / rotated image (or object).

Original: This property is not generally present with existing CNN-basedsegmentation methods even when trained with augmented translated and rotated images (3), andrequires special consideration.3)
Revised: This property is not generally present with existing CNN-based segmentation methods even when trained with augmented translated and rotated images (3), and requires special consideration.3)

Original: We show analytically and empirically that ST-DNNs are robust todomain deformations.
Revised: We show analytically and empirically that ST-DNNs are robust to domain deformations.

Original: These result from viewpoint change or object articulation, and so they shouldnot affect the descriptor.
Revised: These result from viewpoint change or object articulation, and so they should not affect the descriptor.

Original: To demonstrate ST-DNN and the properties above, we validate them onthe task of texture segmentation, an important problem in low-level vision (17; 2).
Revised: To demonstrate ST-DNN and the properties above, we validate them on the task of texture segmentation, an important problem in low-level vision (17; 2).

Original: Because of properties of the PDE, ST-DNN also have desirable generalization properties.
Revised: Because of properties of the PDE, ST-DNN also have desirable generalization properties.

Original: This isbecause: a) The robustness and covariance properties are built into our descriptors and do not needto be learned from data, b) The PDE solutions, generalizations of Gabor-like ﬁlters (18; 26), havenatural image structure inherent in their solutions and so this does not need to be learned from data,and c)
Revised: This is because: a) The robustness and covariance properties are built into our descriptors and do not need to be learned from data, b) The PDE solutions, generalizations of Gabor-like ﬁlters (18; 26), have natural image structure inherent in their solutions and so this does not need to be learned from data, and c)

Original: Our networks have fewer parameters compared to existing networks in segmentation.
Revised: Our networks have fewer parameters compared to existing networks in segmentation.

Original: Thisis because the PDE solutions form a basis and only linear combinations of a few basis elementsare needed to learn discriminative descriptors for segmentation.
Revised: This is because the PDE solutions form a basis and only linear combinations of a few basis elements are needed to learn discriminative descriptors for segmentation.

Original: In contrast, CNNs spend a lot ofparameters to learn this structure.
Revised: In contrast, CNNs spend a lot of parameters to learn this structure.

Original: 1.1 Related Work
Revised: 1.1 Related Work

Original: CNNs have been applied to compute descriptors for semantic segmentation, where pixels in an imageare classiﬁed into certain semantic object classes (15; 12; 8; 20; 27; 16).
Revised: CNNs have been applied to compute descriptors for semantic segmentation, where pixels in an image are classiﬁed into certain semantic object classes (15; 12; 8; 20; 27; 16).

Original: Usually these classes arelimited to a few object classes and do not tackle general textures, where the number of classes maybe far greater, and thus such approaches are not directly applicable to texture segmentation.
Revised: Usually these classes are limited to a few object classes and do not tackle general textures, where the number of classes may be far greater, and thus such approaches are not directly applicable to texture segmentation.

Original: Butsemantic segmentation approaches may eventually beneﬁt from our methodology as descriptorsaggregating data only within objects or regions are also relevant to these problems.
Revised: But semantic segmentation approaches may eventually beneﬁt from our methodology as descriptors aggregating data only within objects or regions are also relevant to these problems.

Original: In (14) a learnedshape-tailored descriptor is constructed with a Siamese network on hand-crafted shape-tailoreddescriptors.
Revised: In (14) a learned shape-tailored descriptor is constructed with a Siamese network on hand-crafted shape-tailored descriptors.

Original: However, (14) only does shape-tailored ﬁltering in pre-processing as layering theserequires new methods to train.
Revised: However, (14) only does shape-tailored ﬁltering in pre-processing as layering these requires new methods to train.

Original: We further examine covariance and robustness, not examined in (14).
Revised: We further examine covariance and robustness, not examined in (14).

Original: Deformable convolutions (7) can also learn non rectangular receptive ﬁelds; however, they are notprovably robust and covariant, key motivations of our approach.
Revised: Deformable convolutions (7) can also learn non rectangular receptive ﬁelds; however, they are not provably robust and covariant, key motivations of our approach.

Original: Covariance to rotation in CNNs has been examined in recent works, e.g., (22; 25; 1).
Revised: Covariance to rotation in CNNs has been examined in recent works, e.g., (22; 25; 1).

Original: They, however,are not shape-tailored so do not aggregate data only within shaped regions.
Revised: They, however, are not shape-tailored so do not aggregate data only within shaped regions.

Original: Lack of robustness todeformation (and translation) in CNNs is examined in (3) and theoretically in (4).
Revised: Lack of robustness to deformation (and translation) in CNNs is examined in (3) and theoretically in (4).

Original: (21) constructsdeformation robust descriptors inspired by CNNs, but are hand-crafted.
Revised: (21) constructs deformation robust descriptors inspired by CNNs, but are hand-crafted.

Original:  Shape-tailored DNN through Poisson PDE and it’s Properties
Revised:  Shape-tailored DNN through Poisson PDE and it’s Properties

Original: In this section, we design a deep neural network that outputs descriptors at each pixel within anarbitrary shaped region of interest and aggregates data only from within the region.
Revised: In this section, we design a deep neural network that outputs descriptors at each pixel within an arbitrary shaped region of interest and aggregates data only from within the region.

Original: Shape-tailored Smoothing via Poisson PDE : To construct a shape-tailored deep network, we ﬁrstsmooth the input to a layer using the Poisson PDE so as to aggregate data only within the regionof interest.
Revised: Shape-tailored Smoothing via Poisson PDE : To construct a shape-tailored deep network, we ﬁrst smooth the input to a layer using the Poisson PDE so as to aggregate data only within the region of interest.

Original: Let R ⊂ Ω ⊂ R 2 be the region of interest, where Ω is the domain of the input image
Revised: Let R ⊂ Ω ⊂ R 2 be the region of interest, where Ω is the domain of the input image

Original: I : Ω → R k and k is the number of input channels to the layer.
Revised: I : Ω → R k and k is the number of input channels to the layer.

Original: Let u : R → R M ( M is the numberof output channels) be the result of the smoothing; the components u of u solve the PDE within R :
Revised: Let u : R → R M ( M is the number of output channels) be the result of the smoothing; the components u of u solve the PDE within R :

Original: [Equation]
Revised: [Equation]

Original: where I is a channel of I , ∂R is the boundary of R , N is normal to ∂R , α is the scale of smoothingand ∆ / ∇ are the Laplacian and the gradient respectively.
Revised: where I is a channel of I , ∂R is the boundary of R , N is normal to ∂R , α is the scale of smoothing and ∆ / ∇ are the Laplacian and the gradient respectively.

Original: It can be shown that the smoothing can bewritten in the form u ( x )=
Revised: It can be shown that the smoothing can be written in the form u ( x )

Original:  (cid:82) R K ( x, y ) I ( y ) dy where K ( ., . )
Revised: = (cid:82) R K ( x, y ) I ( y ) dy where K ( ., . )

Original: is the Green’s function of the PDE, asmoothing kernel, which further shows that the PDE aggregates data only within R (see Figure 1).
Revised: is the Green’s function of the PDE, a smoothing kernel, which further shows that the PDE aggregates data only within R (see Figure 1).

Original: Shape-tailored Deep Network : We can now generalize the operation of convolution tailored tothe region of interest by taking linear combinations of partial derivatives of the output of the PDEequation 1.
Revised: Shape-tailored Deep Network : We can now generalize the operation of convolution tailored to the region of interest by taking linear combinations of partial derivatives of the output of the PDE equation 1.

Original: This is motivated by the fact that in R = R 2 , linear combinations of derivatives of
Revised: This is motivated by the fact that in R = R 2 , linear combinations of derivatives of

Original: Gaussians can approximate any kernel arbitrarily well.
Revised: Gaussians can approximate any kernel arbitrarily well.

Original: Gaussian ﬁlters are the solution of theheat equation, and the PDE equation 1 relates to the heat equation, i.e., equation 1 is the steadystate solution of a heat equation.
Revised: Gaussian ﬁlters are the solution of the heat equation, and the PDE equation 1 relates to the heat equation, i.e., equation 1 is the steady state solution of a heat equation.

Original: Thus, linear combinations of derivatives of equation 1 generalizeconvolution to an arbitrary shape R ; in experiments, a few ﬁrst order directional derivatives aresufﬁcient for our segmentation tasks (see Section 4 for details).
Revised: Thus, linear combinations of derivatives of equation 1 generalize convolution to an arbitrary shape R ; in experiments, a few ﬁrst order directional derivatives are sufﬁcient for our segmentation tasks (see Section 4 for details).

Original: A layer of the ST-DNN takes suchlinear combinations and rectiﬁes it as follows:
Revised: A layer of the ST-DNN takes such linear combinations and rectiﬁes it as follows:

Original: [Equation]
Revised: [Equation]

Original: where I : R → R k is the input to the layer, T is an operator that outputs derivatives of the solution ofthe Poisson PDE equation 1, L i ( y ) =
Revised: where I : R → R k is the input to the layer, T is an operator that outputs derivatives of the solution of the Poisson PDE equation 1, L i ( y ) =

Original: w i y
Revised: w i y

Original: + b i is a point-wise linear function (i.e., a 1 × 1 convolution applied to combine different channels), r is the rectiﬁed linear function, and i indexes the layer of thenetwork.
Revised: + b i is a point-wise linear function (i.e., a 1 × 1 convolution applied to combine different channels), r is the rectiﬁed linear function, and i indexes the layer of the network.

Original: Notice that since r and L i are pointwise operations, they preserve the property of T that itaggregates data only within the region R .
Revised: Notice that since r and L i are pointwise operations, they preserve the property of T that it aggregates data only within the region R .

Original: We now compose layers to construct a ST-DNN as follows:
Revised: We now compose layers to construct a ST-DNN as follows:

Original: [Equation]
Revised: [Equation]

Original: where F [ I ]( x ) is the output of the ST-DNN, f 0 , ..., f m are the m + 1 layers of the network, I is theinput image, and s represents the soft-max operation (to bound the output values).
Revised: where F [ I ]( x ) is the output of the ST-DNN, f 0 , ..., f m are the m + 1 layers of the network, I is the input image, and s represents the soft-max operation (to bound the output values).

Original: ST-DNN does not have a pooling layer because the PDE already aggregates data from a neighborhoodby smoothing; further, the lack of reduction in spatial dimension allows for more accurate shapeestimation in our subsequent segmentation, and avoids the need for up-sampling layers.
Revised: ST-DNN does not have a pooling layer because the PDE already aggregates data from a neighborhood by smoothing; further, the lack of reduction in spatial dimension allows for more accurate shape estimation in our subsequent segmentation, and avoids the need for up-sampling layers.

Original: We will showthat we can retain efﬁciency in training and inference.
Revised: We will show that we can retain efﬁciency in training and inference.

Original: 2.1 Covariance and Robustness of ST-DNN
Revised: 2.1 Covariance and Robustness of ST-DNN

Original: In addition to ST-DNN generalizing CNNs to arbitrary shaped regions, the ST-DNN is also covariantto in-plane translation and rotation, and robustness to domain deformations due to properties of the
Revised: In addition to ST-DNN generalizing CNNs to arbitrary shaped regions, the ST-DNN is also covariant to in-plane translation and rotation, and robustness to domain deformations due to properties of the

Original: Poisson PDE.
Revised: Poisson PDE.

Original: This means covariance also extends to our segmentation method, which is importantas any object segmented in an image will also be segmented if the camera undergoes these transformations.
Revised: This means covariance also extends to our segmentation method, which is important as any object segmented in an image will also be segmented if the camera undergoes these transformations.

Original: Robustness to deformations is important as this means that small geometric variability(e.g., shape variations in textons, small viewpoint change, object deformation) will not affect thedescriptors and the segmentation.
Revised: Robustness to deformations is important as this means that small geometric variability (e.g., shape variations in textons, small viewpoint change, object deformation) will not affect the descriptors and the segmentation.

Original: We make these properties more precise, and give intuition forproofs, leaving details to Appendix (see 5.1).
Revised: We make these properties more precise, and give intuition for proofs, leaving details to Appendix (see 5.1).

Original: Deﬁnition 1 An operator S : I → I (from the set of images I to itself) is covariant to a class W oftransformations if S [ I ◦ w ] =
Revised: Deﬁnition 1 An operator S : I → I (from the set of images I to itself) is covariant to a class W of transformations if S [ I ◦ w ] =

Original: [ SI ] ◦ w for every I ∈ I and w ∈ W .[Figure]
Revised: [ SI ] ◦ w for every I ∈ I and w ∈ W .

Original: Theorem 1 The ST-DNN equation 3 is covariant to the set of translations and rotations, i.e., x →
Revised: Theorem 1 The ST-DNN equation 3 is covariant to the set of translations and rotations, i.e., x →

Original: R x + T where R is a 2 × 2 rotation matrix and T ∈ R 2 .
Revised: R x + T where R is a 2 × 2 rotation matrix and T ∈ R 2 .

Original: Theorem 2 The ST-DNN equation 3 is insensitive to deformations, i.e.,
Revised: Theorem 2 The ST-DNN equation 3 is insensitive to deformations, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where w : Ω → Ω is a domain deformation, id is the identity map, H 1 is the Sobolev norm (measuresboth the amount and smoothness of the deformation), and C is a constant independent of w, I .
Revised: where w : Ω → Ω is a domain deformation, id is the identity map, H 1 is the Sobolev norm (measures both the amount and smoothness of the deformation), and C is a constant independent of w, I .

Original:  Training and Inference with ST-DNNs
Revised:  Training and Inference with ST-DNNs

Original: Given the ST-DNN of Section 2, the loss function to train such descriptors from ground truthsegmentation masks (motivated by consistency to the segmentation algorithm based on classicalsegmentation energies (5; 24)) is deﬁned as:
Revised: Given the ST-DNN of Section 2, the loss function to train such descriptors from ground truth segmentation masks (motivated by consistency to the segmentation algorithm based on classical segmentation energies (5; 24)) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where i, j ∈ { 1 , 2 , ..., N } are the indices for the regions in the ground truth segmentation, F W ( x )is the output of the ST-DNN, W are the weights of the network (i.e., weights on derivatives ofthe Poisson PDE solution), | R i | is the area of region i , and a i is the average descriptor within thei th region, i.e., a i = 1 | R i | (cid:82) R i F W ( x ) dx .
Revised: where i, j ∈ { 1 , 2 , ..., N } are the indices for the regions in the ground truth segmentation, F W ( x ) is the output of the ST-DNN, W are the weights of the network (i.e., weights on derivatives of the Poisson PDE solution), | R i | is the area of region i , and a i is the average descriptor within the i th region, i.e., a i = 1 | R i | (cid:82) R i F W ( x ) dx .

Original: The loss function is comprised of two terms.
Revised: The loss function is comprised of two terms.

Original: The ﬁrstcomponent of the loss is minimized when the learned descriptor is constant within regions
Revised: The ﬁrst component of the loss is minimized when the learned descriptor is constant within regions

Original: R i sothat each region consists of parts of the image with uniform descriptor.
Revised: R i so that each region consists of parts of the image with uniform descriptor.

Original: The second term forces thelearned descriptor of different regions to be different to discriminate different textures.
Revised: The second term forces the learned descriptor of different regions to be different to discriminate different textures.

Original: Computing gradients of the loss function for training requires consideration as it involves differentiating through PDEs.
Revised: Computing gradients of the loss function for training requires consideration as it involves differentiating through PDEs.

Original: The most straightforward way to do this involves discretizing the PDE, sothe solution is a linear matrix system as we do in this paper.
Revised: The most straightforward way to do this involves discretizing the PDE, so the solution is a linear matrix system as we do in this paper.

Original: This allows the use of existing deeplearning packages to perform backprop by storing the matrix in memory.
Revised: This allows the use of existing deep learning packages to perform backprop by storing the matrix in memory.

Original: However, this can lead tolarge memory consumption as the matrix can be large and is only feasible for small images.
Revised: However, this can lead to large memory consumption as the matrix can be large and is only feasible for small images.

Original: Themore accurate method, though more difﬁcult to implement, is to avoid storing the matrix and insteadcompute the solution by an iterative numerical PDE method that does not require storage of matrices.
Revised: The more accurate method, though more difﬁcult to implement, is to avoid storing the matrix and instead compute the solution by an iterative numerical PDE method that does not require storage of matrices.

Original: At inference the regions of segmentation are estimated iteratively together with updates of the STDNN for each of the regions as they evolve.
Revised: At inference the regions of segmentation are estimated iteratively together with updates of the STDNN for each of the regions as they evolve.

Original: We minimize the (non-convex) energy i.e. the ﬁrst termof the training loss plus boundary regularisation term, with respect to the region.
Revised: We minimize the (non-convex) energy i.e. the ﬁrst term of the training loss plus boundary regularisation term, with respect to the region.

Original: We use gradientdescent to minimize the energy.
Revised: We use gradient descent to minimize the energy.

Original: The curve (boundary of regions) evolution is implemented with amethod analogous to level set methods (19) by evolving smooth indicator functions of regions forconvenient implementation.
Revised: The curve (boundary of regions) evolution is implemented with a method analogous to level set methods (19) by evolving smooth indicator functions of regions for convenient implementation.

Original: The method involves joint updates of the regions and the shape-tailoreddescriptors within the evolving regions.
Revised: The method involves joint updates of the regions and the shape-tailored descriptors within the evolving regions.

Original: Our method typically takes a few iterations (approx. 20) toconverge in our experiments.
Revised: Our method typically takes a few iterations (approx. 20) to converge in our experiments.

Original: For more details of the inference refer to Appendix Algorithm 1.
Revised: For more details of the inference refer to Appendix Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: For the experiments we have used a four layer ST-DNN with 100, 40, 20, 5 hidden units respectively.
Revised: For the experiments we have used a four layer ST-DNN with 100, 40, 20, 5 hidden units respectively.

Original: The smoothing parameter α is set to 5.
Revised: The smoothing parameter α is set to 5.

Original: Covariance and Robustness of ST-DNN: To demonstrate the covariance of ST-DNN to translationand rotation, each image in the test set was randomly rotated and cropped to a rectangle at randompositions (to simulate translation) in the rotated image.
Revised: Covariance and Robustness of ST-DNN: To demonstrate the covariance of ST-DNN to translation and rotation, each image in the test set was randomly rotated and cropped to a rectangle at random positions (to simulate translation) in the rotated image.

Original: We segment the original and the transformedimage, denoted S
Revised: We segment the original and the transformed image, denoted S

Original: [ I ] and S [ I ◦ w ] , respectively, where w is the transformation used to producethe translated/rotated image.
Revised: [ I ] and S [ I ◦ w ] , respectively, where w is the transformation used to produce the translated/rotated image.

Original: We then measure the difference between S
Revised: We then measure the difference between S

Original: [ I ] ◦ w and S
Revised: [ I ] ◦ w and S

Original: [ I ◦ w ]through GT-covering; both should be equal if the descriptor is covariant.
Revised: [ I ◦ w ] through GT-covering; both should be equal if the descriptor is covariant.

Original: Results are summarizedin Table 1.
Revised: Results are summarized in Table 1.

Original: ST-DNN outperforms resnet101-all-ce by a margin of almost 25%.
Revised: ST-DNN outperforms resnet101-all-ce by a margin of almost 25%.

Original: Note ST-DNN usesno data augmentation, whereas the competing networks are augmented with translated and rotatedimages.
Revised: Note ST-DNN uses no data augmentation, whereas the competing networks are augmented with translated and rotated images.

Original: To demonstrate robustness to deformation, for each image in the dataset we generate randomdeformations of varying deformation norm (cid:107) v (cid:107) 2 between 20 and 80.
Revised: To demonstrate robustness to deformation, for each image in the dataset we generate random deformations of varying deformation norm (cid:107) v (cid:107) 2 between 20 and 80.

Original: We examine the robustnessof descriptors to deformations of increasing norm by comparing the segmentation of the originaland deformed images similar to the covariance experiment.
Revised: We examine the robustness of descriptors to deformations of increasing norm by comparing the segmentation of the original and deformed images similar to the covariance experiment.

Original: Results are in Table 1, which showthat ST-DNN is more robust by large margins than competing descriptors, and the robustness overcompeting methods increases with increasing norm.
Revised: Results are in Table 1, which show that ST-DNN is more robust by large margins than competing descriptors, and the robustness over competing methods increases with increasing norm.

Original: Comparison of ST-DNN to Standard DNNs : We compare our method ST-DNN (ours) on a texturesegmentation dataset (13) against popular deep learning architectures in computer vision - DeepLabv3 (6), and FCN-ResNet101 (11) and with a modiﬁed version of ST-DNN where the descriptor iscalculated on the entire image (ST-DNN (RL)).
Revised: Comparison of ST-DNN to Standard DNNs : We compare our method ST-DNN (ours) on a texture segmentation dataset (13) against popular deep learning architectures in computer vision - DeepLabv3 (6), and FCN-ResNet101 (11) and with a modiﬁed version of ST-DNN where the descriptor is calculated on the entire image (ST-DNN (RL)).

Original: ST-DNN networks have 8900 parameters and istrained with only 128 images.
Revised: ST-DNN networks have 8900 parameters and is trained with only 128 images.

Original: ST-DNN is around 3 orders of magnitude smaller than standard deepnetworks and takes around 2 orders of magnitude less training data (e.g., FCN-ResNet101 uses 50,000images plus augmented data and has 45 million parameters), but still outperforms these networks.
Revised: ST-DNN is around 3 orders of magnitude smaller than standard deep networks and takes around 2 orders of magnitude less training data (e.g., FCN-ResNet101 uses 50, images plus augmented data and has 45 million parameters), but still outperforms these networks.

Original: We have trained FCN-ResNet101 and DeepLab-v3 on saliency detection datasets (large datasets forbinary segmentation) and ﬁne tuned on the base dataset augmented with 8 rotations and 5 scales.
Revised: We have trained FCN-ResNet101 and DeepLab-v3 on saliency detection datasets (large datasets for binary segmentation) and ﬁne tuned on the base dataset augmented with 8 rotations and 5 scales.

Original: Wehave used both cross entropy loss (denoted with ce) and our loss (denoted with ’ours’) for standarddeep learning networks in our experiments.
Revised: We have used both cross entropy loss (denoted with ce) and our loss (denoted with ’ours’) for standard deep learning networks in our experiments.

Original: Quantitative results are shown in Table 2.
Revised: Quantitative results are shown in Table 2.

Original: ST-DNNoutperforms all other deep networks by good margins.
Revised: ST-DNN outperforms all other deep networks by good margins.

Original: Notice that because of the desirable propertiesof PDEs STDNN (RL) perform better than DNN based architecture ST-DNN (ours) performs betterthan ST-DNN (RL) because of the iterative update of the region.
Revised: Notice that because of the desirable properties of PDEs STDNN (RL) perform better than DNN based architecture ST-DNN (ours) performs better than ST-DNN (RL) because of the iterative update of the region.

Original: [Figure]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: 5.1 Analytical Proofs for Covariance and Robustness
Revised: 5.1 Analytical Proofs for Covariance and Robustness

Original: The proof of Theorem 1 (covariance of ST-DNN to rotation and translation) follows from basicproperties of the Laplace equation (9); we state these properties in terms of our language of covariantoperators, and show the proof for the convenience of the reader.
Revised: The proof of Theorem 1 (covariance of ST-DNN to rotation and translation) follows from basic properties of the Laplace equation (9); we state these properties in terms of our language of covariant operators, and show the proof for the convenience of the reader.

Original: We will show Theorem 2 (robustnessof ST-DNN to domain deformations) as a consequence of properties of linear PDE theory.
Revised: We will show Theorem 2 (robustness of ST-DNN to domain deformations) as a consequence of properties of linear PDE theory.

Original: We repeat the deﬁnition of covariant operator:
Revised: We repeat the deﬁnition of covariant operator:

Original: Deﬁnition 2 An operator T : I → I is covariant to a class W of transformations if
Revised: Deﬁnition 2 An operator T : I → I is covariant to a class W of transformations if

Original: [Equation]
Revised: [Equation]

Original: , for every I ∈ I and w ∈ W .
Revised: for every I ∈ I and w ∈ W .

Original: We show covariance of the Laplace operator, and as a consequence, the covariance of the Poisson
Revised: We show covariance of the Laplace operator, and as a consequence, the covariance of the Poisson

Original: PDE and the ST-DNN.
Revised: PDE and the ST-DNN.

Original: Theorem 3 The Laplacian operator ∆ =
Revised: Theorem 3 The Laplacian operator ∆ =

Original: (cid:80) i ∂
Revised: (cid:80) i ∂

Original: 2 ∂x 2 i is covariant to rotations, x → Rx where R is arotation matrix.
Revised: 2 ∂x 2 i is covariant to rotations, x → Rx where R is a rotation matrix.

Original: Proof 1 Let u ∈
Revised: Proof 1 Let u ∈

Original: I
Revised: I

Original: , and let R be a rotation.
Revised: , and let R be a rotation.

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Original: so,
Revised: so,

Original: [Equation]
Revised: [Equation]

Original: so ,
Revised: so ,

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second last equality is due to the invariance of the trace to similarity transformations.
Revised: where the second last equality is due to the invariance of the trace to similarity transformations.

Original: Theorem 4 The Laplacian operator is covariant to translations, x → x + t , where t is a vector.
Revised: Theorem 4 The Laplacian operator is covariant to translations, x → x + t , where t is a vector.

Original: Proof 2 We have,
Revised: Proof 2 We have,

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Thus, ∆( u ◦ T )( x ) =
Revised: Thus, ∆( u ◦ T )( x ) =

Original: (∆ u ) ◦ T where T ( x ) = x + t .
Revised: (∆ u ) ◦ T where T ( x ) = x + t .

Original: Corollary 1 The solution u of the Poisson equation, i.e.,
Revised: Corollary 1 The solution u of the Poisson equation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: 15) u
Revised: 15) u

Original: = T [ I ] is covariant to translations and rotations.
Revised: = T [ I ] is covariant to translations and rotations.

Original: Proof 

Revised: Proof

Original: This follows from the covariance of the Laplacian, and the identity map.
Revised: This follows from the covariance of the Laplacian, and the identity map.

Original: We can now show covariance of the ST-DNN to translations and rotations.
Revised: We can now show covariance of the ST-DNN to translations and rotations.

Original: Theorem 5 The ST-DNN equation (3 main paper) is covariant to translations and rotations.
Revised: Theorem 5 The ST-DNN equation (3 main paper) is covariant to translations and rotations.

Original: Proof 4 Since ST-DNNs are composition of solutions of Poisson Equations with fully connectedlayer across channels and non-linearity in multiple layers.
Revised: Proof 4 Since ST-DNNs are composition of solutions of Poisson Equations with fully connected layer across channels and non-linearity in multiple layers.

Original: A linear combination of channels andpoint-wise non-linear operation preserves the covariance of rotation and translations, hence the
Revised: A linear combination of channels and point-wise non-linear operation preserves the covariance of rotation and translations, hence the

Original: ST-DNNs are covariant to translations and rotations.
Revised: ST-DNNs are covariant to translations and rotations.

Original: We now show robustness of the ST-DNN to domain deformations with respect to the Sobolevnorm; that is, we show that the output of a ST-DNN layer does not change much if deformed by atransformation with small Sobolev norm (a smooth transformation that has small displacement).
Revised: We now show robustness of the ST-DNN to domain deformations with respect to the Sobolev norm; that is, we show that the output of a ST-DNN layer does not change much if deformed by a transformation with small Sobolev norm (a smooth transformation that has small displacement).

Original: The
Revised: The

Original: Sobolev measures the smoothness of the deformation, i.e., the L 2 norm of the deformation and thegradient of the deformation.
Revised: Sobolev measures the smoothness of the deformation, i.e., the L 2 norm of the deformation and the gradient of the deformation.

Original: We have the following theorem:
Revised: We have the following theorem:

Original: Theorem 6 The solution of the ST-DNN is robust to deformations, i.e.,
Revised: Theorem 6 The solution of the ST-DNN is robust to deformations, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where T is the mapping from the image to the solution of the ST-DNN, w : Ω → Ω is a domaindeformation, id is the identity map, and H 1 indicates the Sobolev norm.
Revised: where T is the mapping from the image to the solution of the ST-DNN, w : Ω → Ω is a domain deformation, id is the identity map, and H 1 indicates the Sobolev norm.

Original: Proof 

Revised: Proof

Original: This is a consequence of Lemma equation 1 below, which shows that each layer of the
Revised: This is a consequence of Lemma equation 1 below, which shows that each layer of the

Original: ST-DNN is robust.
Revised: ST-DNN is robust.

Original: Stacking such layers preserves the robustness by applying Lemma 1 successively.
Revised: Stacking such layers preserves the robustness by applying Lemma 1 successively.

Original: We now prove robustness of layers of ST-DNN.
Revised: We now prove robustness of layers of ST-DNN.

Original: For convenience in the proof, we assume the inputoperates on the domain Ω = R 2 , which avoids having to consider the boundary and more complicatedformulas that do not affect the essence of the argument.
Revised: For convenience in the proof, we assume the input operates on the domain Ω = R 2 , which avoids having to consider the boundary and more complicated formulas that do not affect the essence of the argument.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: ) be a layer of ST-DNN.
Revised: be a layer of ST-DNN.

Original: Here r is the rectiﬁed linear unit, K is the kernel representing, the Green’sfunction of the Poisson equation (we assume for this proof that the domain of the image is all of R 2 ),
Revised: Here r is the rectiﬁed linear unit, K is the kernel representing, the Green’s function of the Poisson equation (we assume for this proof that the domain of the image is all of R 2 ),

Original: D is the derivative operator representing oriented gradients (or parial derivatives of an ﬁnite order),and L is a weight matrix of fully connected layer across channels.
Revised: D is the derivative operator representing oriented gradients (or parial derivatives of an ﬁnite order), and L is a weight matrix of fully connected layer across channels.

Original: We state the robustness of a layer of ST-DNN:
Revised: We state the robustness of a layer of ST-DNN:

Original: Lemma 1 A layer, f
Revised: Lemma 1 A layer, f

Original: [ I ] =
Revised: [ I ] =

Original: r [( L ◦ D ◦ K ∗ I ] , of a ST-DNN is Lipschitz continuous with respect todiffeomorphisms in the Sobolev norm, i.e., 
Revised: r [( L ◦ D ◦ K ∗ I ] , of a ST-DNN is Lipschitz continuous with respect to diffeomorphisms in the Sobolev norm, i.e., [Equation]

Original: [Equation]
Revised: [Equation]

Original: (18) where id ( x ) = x is the identity map, and C is a constant (independent of w and only of function ofthe class of images), and (cid:107) w (cid:107) 2 H 1 = (cid:82) Ω
Revised: where id ( x ) = x is the identity map, and C is a constant (independent of w and only of function of the class of images), and (cid:107) w (cid:107) 2 H 1 = (cid:82) Ω

Original: ( | w ( x ) | + |∇ w ( x ) | 2 ) dx .
Revised: ( | w ( x ) | + |∇ w ( x ) | 2 ) dx .

Original: Note that w − id is the displacement.
Revised: Note that w − id is the displacement.

Original: Proof 6 Let w be a smooth diffeomorphism.
Revised: Proof 6 Let w be a smooth diffeomorphism.

Original: Then by Lipschitz continuity of the ReLu,
Revised: Then by Lipschitz continuity of the ReLu,

Original: [Equation]
Revised: [Equation]

Original: where M = L ◦ D ◦ K .
Revised: where M = L ◦ D ◦ K .

Original: Note that by a change of variables,
Revised: Note that by a change of variables,

Original: [Equation]
Revised: [Equation]

Original: Note that the determinent of the Jacobian appears if we weight the sum by the area measure, whichapproximates the integral.
Revised: Note that the determinent of the Jacobian appears if we weight the sum by the area measure, which approximates the integral.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: We let w ( y ) =
Revised: We let w ( y ) =

Original: y + v ( y ) .
Revised: y + v ( y ) .

Original: This gives us
Revised: This gives us

Original: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: We may bound the second term as
Revised: We may bound the second term as

Original: [Equation]
Revised: [Equation]

Original: by basic inequalities.
Revised: by basic inequalities.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: By Lipschitz continuity of the Poisson kernel and derivatives, we have
Revised: By Lipschitz continuity of the Poisson kernel and derivatives

Original: [Equation]
Revised: [Equation]

Original:  Note that the Poisson kernel has a singularity at the origin, so the statement is not precise; however,as common in PDE analysis, as we will below compute integrals of the left hand quantity, we canbreak the integral into two terms one that integrates the singularity in a small ball (which is ﬁnite)and the other that integrates the right hand side, that we analyze below.
Revised: we have [Equation] Note that the Poisson kernel has a singularity at the origin, so the statement is not precise; however, as common in PDE analysis, as we will below compute integrals of the left hand quantity, we can break the integral into two terms one that integrates the singularity in a small ball (which is ﬁnite) and the other that integrates the right hand side, that we analyze below.

Original: The former will disappear tozero as the ball goes to zero.
Revised: The former will disappear to zero as the ball goes to zero.

Original: We omit the details to avoid hiding the main argument.
Revised: We omit the details to avoid hiding the main argument.

Original: We also have that
Revised: We also have that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: For a multi layer multiple layers network with N layers we will have:
Revised: For a multi layer multiple layers network with N layers we will have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 1: Input: An initialization of φ i 2: repeat 3:
Revised: 1: Input: An initialization of φ i 2: repeat 3:

Original: Set regions: R i = { x ∈ Ω : i = argmax j φ j ( x ) } 4:
Revised: Set regions: R i = { x ∈ Ω : i = argmax j φ j ( x ) } 4:

Original: Compute dilations, D ( R i ) , of R i 5:
Revised: Compute dilations, D ( R i ) , of R i 5:

Original: Compute F R i in D ( R i ) , compute a
Revised: Compute F R i in D ( R i ) , compute a

Original: i
Revised: i

Original: = 1 / | R i | · (cid:82) R
Revised: = 1 / | R i | · (cid:82) R

Original: i F R i ( x ) dx .
Revised: i F R i ( x ) dx .

Original: Compute band pixels B i
Revised: Compute band pixels B i

Original: = D ( R i )
Revised: = D ( R i )

Original: ∩ D (Ω \ R i ) 7:
Revised: ∩ D (Ω \ R i ) 7:

Original: Compute G i =
Revised: Compute G i =

Original: (cid:107) F R i ( x ))
Revised: (cid:107) F R i ( x ))

Original: − a i (cid:107) 22 for x ∈ B i .
Revised: − a i (cid:107) 22 for x ∈ B i .

Original: F is evaluated from the neural network.
Revised: F is evaluated from the neural network.

Original: Update pixels x ∈ D
Revised: Update pixels x ∈ D

Original: ( R i )
Revised: ( R i )

Original: ∩ D ( R j ) as follows:
Revised: ∩ D ( R j ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Update all other pixels as
Revised: Update all other pixels as

Original: [Equation]
Revised: [Equation]

Original: Clip between 0 and 1: φ i = max { 0 , min { 1 , φ i }} .
Revised: Clip between 0 and 1: φ i = max { 0 , min { 1 , φ i }} .

Original: 11: until regions have converged
Revised: 11: until regions have converged