Original: C ONTINUOUS C ONVOLUTIONAL N EURAL N ETWORKS FOR I MAGE C LASSIFICATION
Revised: C ONTINUOUS C ONVOLUTIONAL N EURAL N ETWORKS FOR I MAGE C LASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces the concept of continuous convolution to neural networks and deep learning applications in general.
Revised: This paper introduces the concept of continuous convolution to neural networks and deep learning applications in general.

Original: Rather than directly using discretized information, input data is ﬁrst projected into a high-dimensional Reproducing Kernel Hilbert Space (RKHS), where it can be modeled as a continuous function using a series of kernel bases.
Revised: Rather than directly using discretized information, input data is ﬁrst projected into a high-dimensional Reproducing Kernel Hilbert Space (RKHS), where it can be modeled as a continuous function using a series of kernel bases.

Original: We then proceed to derive a closed-form solution to the continuous convolution operation between two arbitrary functions operating in different RKHS.
Revised: We then proceed to derive a closed-form solution to the continuous convolution operation between two arbitrary functions operating in different RKHS.

Original: Within this framework, convolutional ﬁlters also take the form of continuous functions, and the training procedure involves learning the RKHS to which each of these ﬁlters is projected, alongside their weight parameters.
Revised: Within this framework, convolutional ﬁlters also take the form of continuous functions, and the training procedure involves learning the RKHS to which each of these ﬁlters is projected, alongside their weight parameters.

Original: This results in much more expressive ﬁlters, that do not require spatial discretization and beneﬁt from properties such as adaptive support and non-stationarity.
Revised: This results in much more expressive ﬁlters, that do not require spatial discretization and beneﬁt from properties such as adaptive support and non-stationarity.

Original: Experiments on image classiﬁcation are performed, using classical datasets, with results indicating that the proposed continuous convolutional neural network is able to achieve competitive accuracy rates with far fewer parameters and a faster convergence rate.
Revised: Experiments on image classiﬁcation are performed, using classical datasets, with results indicating that the proposed continuous convolutional neural network is able to achieve competitive accuracy rates with far fewer parameters and a faster convergence rate.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, convolutional neural networks (CNNs) have become widely popular as a deep learning tool for addressing various sorts of problems, most predominantly in computer vision, such as image classiﬁcation (Krizhevsky et al., 2012;
Revised: In recent years, convolutional neural networks (CNNs) have become widely popular as a deep learning tool for addressing various sorts of problems, most predominantly in computer vision, such as image classiﬁcation (Krizhevsky et al., 2012;

Original: He et al., 2016), object detection (Ren et al., 2015) and semantic segmentation (Ghiasi & Fowlkes, 2016).
Revised: He et al., 2016), object detection (Ren et al., 2015) and semantic segmentation (Ghiasi & Fowlkes, 2016).

Original: The introduction of convolutional ﬁlters produces many desirable effects, including: translational invariance, since the same patterns are detected in the entire image; spatial connectivity, as neighboring information is taken into consideration during the convolution process; and shared weights, which results in signiﬁcantly fewer training parameters and smaller memory footprint.
Revised: The introduction of convolutional ﬁlters produces many desirable effects, including: translational invariance, since the same patterns are detected in the entire image; spatial connectivity, as neighboring information is taken into consideration during the convolution process; and shared weights, which results in signiﬁcantly fewer training parameters and smaller memory footprint.

Original: Even though the convolution operation is continuous in nature (Hirschman & Widder, 1955), a common assumption in most computational tasks is data discretization, since that is usually how information is obtained and stored: 2D images are divided into pixels, 3D point clouds are divided into voxels, and so forth.
Revised: Even though the convolution operation is continuous in nature (Hirschman & Widder, 1955), a common assumption in most computational tasks is data discretization, since that is usually how information is obtained and stored: 2D images are divided into pixels, 3D point clouds are divided into voxels, and so forth.

Original: Because of that, the exact convolution formulation is often substituted by a discrete approximation (Damelin & Miller, 2011), calculated by sliding the ﬁlter over the input data and calculating the dot product of overlapping areas.
Revised: Because of that, the exact convolution formulation is often substituted by a discrete approximation (Damelin & Miller, 2011), calculated by sliding the ﬁlter over the input data and calculating the dot product of overlapping areas.

Original: While much simpler to compute, it requires substantially more computational power, especially for larger datasets and ﬁlter sizes (Pavel & David, 2013).
Revised: While much simpler to compute, it requires substantially more computational power, especially for larger datasets and ﬁlter sizes (Pavel & David, 2013).

Original: The fast Fourier transform has been shown to signiﬁcantly increase performance in convolutional neural network calculations (Highlander & Rodriguez, 2015; Rippel et al., 2015), however these improvements are mostly circumstantial, with the added cost of performing such transforms, and do not address memory requirements.
Revised: The fast Fourier transform has been shown to signiﬁcantly increase performance in convolutional neural network calculations (Highlander & Rodriguez, 2015; Rippel et al., 2015), however these improvements are mostly circumstantial, with the added cost of performing such transforms, and do not address memory requirements.

Original: To the best of our knowledge, all versions of CNNs currently available in the literature use this discrete approximation to convolution, as a way to simplify calculations at the expense of a potentially more descriptive model.
Revised: To the best of our knowledge, all versions of CNNs currently available in the literature use this discrete approximation to convolution, as a way to simplify calculations at the expense of a potentially more descriptive model.

Original: In Liu et al.
Revised: In Liu et al.

Original: (2015) a sparse network was used to dramatically decrease computational times by exploiting redundancies, and in Graham (2014) spatial sparseness was exploited to achieve state-of-the-art results in various image classiﬁcation datasets.
Revised: (2015) a sparse network was used to dramatically decrease computational times by exploiting redundancies, and in Graham (2014) spatial sparseness was exploited to achieve state-of-the-art results in various image classiﬁcation datasets.

Original: Similarly, Riegler et al.
Revised: Similarly, Riegler et al.

Original: (2016) used octrees to efﬁciently partition the space during convolution, thus focusing memory allocation and computation to denser regions.
Revised: (2016) used octrees to efﬁciently partition the space during convolution, thus focusing memory allocation and computation to denser regions.

Original: A quantized version was proposed in Wu et al.
Revised: A quantized version was proposed in Wu et al.

Original: (2016) to improve performance on mobile devices, with simultaneous computational acceleration and model compression.
Revised: (2016) to improve performance on mobile devices, with simultaneous computational acceleration and model compression.

Original: A lookup-based network is described in Bagherinezhad et al.
Revised: A lookup-based network is described in Bagherinezhad et al.

Original: (2016), that encodes convolution as a series of lookups to a dictionary that is trained to cover the observed weight space.
Revised: (2016), that encodes convolution as a series of lookups to a dictionary that is trained to cover the observed weight space.

Original: This paper takes a different approach and introduces the concept of continuous convolution to neural networks and deep learning applications in general.
Revised: This paper takes a different approach and introduces the concept of continuous convolution to neural networks and deep learning applications in general.

Original: This is achieved by projecting information into a Reproducing Kernel Hilbert Space (RKHS) (Scholkopf & Smola, 2001), in which point evaluation takes the form of a continuous linear functional.
Revised: This is achieved by projecting information into a Reproducing Kernel Hilbert Space (RKHS) (Scholkopf & Smola, 2001), in which point evaluation takes the form of a continuous linear functional.

Original: We employ the Hilbert Maps framework, initially described in Ramos & Ott (2015), to reconstruct discrete input data as a continuous function, based on the methodology proposed in Guizilini & Ramos (2016).
Revised: We employ the Hilbert Maps framework, initially described in Ramos & Ott (2015), to reconstruct discrete input data as a continuous function, based on the methodology proposed in Guizilini & Ramos (2016).

Original: Within this framework, we derive a closed-form solution to the continuous convolution between two functions that takes place directly in this high-dimensional space, where arbitrarily complex patterns are represented using a series of simple kernels, that can be efﬁciently convolved to produce a third RKHS modeling activation values.
Revised: Within this framework, we derive a closed-form solution to the continuous convolution between two functions that takes place directly in this high-dimensional space, where arbitrarily complex patterns are represented using a series of simple kernels, that can be efﬁciently convolved to produce a third RKHS modeling activation values.

Original: Optimizing this neural network involves learning not only weight parameters, but also
Revised: Optimizing this neural network involves learning not only weight parameters, but also

Original: the
Revised: the

Original: RKHS that deﬁnes each convolutional ﬁlter, which results is much more descriptive feature maps that can be used for both discriminative and generative tasks.
Revised: RKHS that deﬁnes each convolutional ﬁlter, which results is much more descriptive feature maps that can be used for both discriminative and generative tasks.

Revised: The use of high-dimensional projection, including inﬁnite-layer neural networks Hazan & Jaakkola (2015); Globerson & Livni (2016), has been extensively studied in recent times, as a way to combine kernel-based learning with deep learning applications.

Original: Note that, while Mairal (2016) takes a similar approach of projecting input data into a RKHS, using the kernel trick, it still relies on discretized image patches, whereas ours operates solely on data already projected to these highdimensional spaces.
Revised: Note that, while Mairal (2016) takes a similar approach of projecting input data into a RKHS, using the kernel trick, it still relies on discretized image patches, whereas ours operates solely on data already projected to these high-dimensional spaces.

Original: The proposed technique, entitled Continuous Convolutional Neural Networks (CCNNs), was evaluated in an image classiﬁcation context, using standard computer vision benchmarks, and achieved competitive accuracy results with substantially smaller network sizes.
Revised: The proposed technique, entitled Continuous Convolutional Neural Networks (CCNNs), was evaluated in an image classiﬁcation context, using standard computer vision benchmarks, and achieved competitive accuracy results with substantially smaller network sizes.

Original: We also demonstrate its applicability to unsupervised learning, by describing a convolutional auto-encoder that is able to produce latent feature representations in the form of continuous functions, which are then used as initial ﬁlters for classiﬁcation using labeled data.
Revised: We also demonstrate its applicability to unsupervised learning, by describing a convolutional auto-encoder that is able to produce latent feature representations in the form of continuous functions, which are then used as initial ﬁlters for classiﬁcation using labeled data.

Original:  H ILBERT M APS
Revised:  H ILBERT M APS

Original: The Hilbert Maps (HM) framework, initially proposed in Ramos & Ott (2015), approximates realworld complexity by projecting discrete input data into a continuous Reproducing Kernel Hilbert
Revised: The Hilbert Maps (HM) framework, initially proposed in Ramos & Ott (2015), approximates realworld complexity by projecting discrete input data into a continuous Reproducing Kernel Hilbert

Original: Space (RKHS), where calculations take place.
Revised: Space (RKHS), where calculations take place.

Original: Since its introduction, it has been primarily used as a classiﬁcation tool for occupancy mapping (Guizilini & Ramos, 2016;
Revised: Since its introduction, it has been primarily used as a classiﬁcation tool for occupancy mapping (Guizilini & Ramos, 2016;

Original: Doherty et al., 2016;
Revised: Doherty et al., 2016;

Original: Senanayake et al., 2016) and more recently terrain modeling (Guizilini & Ramos, 2017).
Revised: Senanayake et al., 2016) and more recently terrain modeling (Guizilini & Ramos, 2017).

Original: This section provides an overview of its fundamental concepts, before moving on to a description of the feature vector used in this work, and ﬁnally we show how model weights and kernel parameters can be jointly learned to produce more ﬂexible representations.
Revised: This section provides an overview of its fundamental concepts, before moving on to a description of the feature vector used in this work, and ﬁnally we show how model weights and kernel parameters can be jointly learned to produce more ﬂexible representations.

Original: 2. O VERVIEW
Revised: 2. O VERVIEW

Original: We start by assuming a dataset D =
Revised: We start by assuming a dataset D =

Original: ( x , y ) Ni =1 , in which x i ∈ R D are observed points and y i
Revised: ( x , y ) Ni =1 , in which x i ∈ R D are observed points and y i

Original: = {− 1 , +1 } are their corresponding occupancy values.
Revised: = {− 1 , +1 } are their corresponding occupancy values.

Original: This dataset is used to learn a discriminative model p ( y | x , w ) , parametrized by a weight vector w .
Revised: This dataset is used to learn a discriminative model p ( y | x , w ) , parametrized by a weight vector w .

Original: Since calculations will be performed in a high-dimensional space, a simple linear classiﬁer is almost always adequate to model even highly complex functions (Komarek, 2004).
Revised: Since calculations will be performed in a high-dimensional space, a simple linear classiﬁer is almost always adequate to model even highly complex functions (Komarek, 2004).

Original: Here we use a Logistic Regression (LR) classiﬁer (Freedman, 2005), due to its computational speed and direct extension to online learning.
Revised: Here we use a Logistic Regression (LR) classiﬁer (Freedman, 2005), due to its computational speed and direct extension to online learning.

Original: The probability of occupancy for a query point x ∗ is given by:
Revised: The probability of occupancy for a query point x ∗ is given by:

Original: [Equation]
Revised: [Equation]

Original: (1) where Φ( . ) is the feature vector, that projects input data into the RKHS.
Revised: where Φ( . ) is the feature vector, that projects input data into the RKHS.

Original: To optimize the weight parameters w based on the information contained in D , we minimize the following negative loglikelihood cost function:
Revised: To optimize the weight parameters w based on the information contained in D , we minimize the following negative loglikelihood cost function:

Original: [Equation]
Revised: [Equation]

Original: (2) with R ( w ) serving as a regularization term, used to avoid over-ﬁtting.
Revised: with R ( w ) serving as a regularization term, used to avoid over-ﬁtting.

Original: Once training is complete, the resulting model can be used to query the occupancy state of any input point x ∗ using Equation 1, at arbitrary resolutions and without the need of space discretization.
Revised: Once training is complete, the resulting model can be used to query the occupancy state of any input point x ∗ using Equation 1, at arbitrary resolutions and without the need of space discretization.

Original: 2. L OCALIZED L ENGTH -S CALES
Revised: 2. L OCALIZED L ENGTH -S CALES

Original: The choice of feature vector is very important, since it deﬁnes how input points will be represented when projected to the RKHS.
Revised: The choice of feature vector is very important, since it deﬁnes how input points will be represented when projected to the RKHS.

Original: In Guizilini & Ramos (2016), the authors propose a feature vector Φ( . ) that places a set M of inducing points throughout the input space, either as a grid-like structure or by clustering D .
Revised: In Guizilini & Ramos (2016), the authors propose a feature vector Φ( . ) that places a set M of inducing points throughout the input space, either as a grid-like structure or by clustering D .

Original: These inducing points are used to correlate input data based on a kernel function, here chosen to be a Gaussian distribution with automatic relevance determination:
Revised: These inducing points are used to correlate input data based on a kernel function, here chosen to be a Gaussian distribution with automatic relevance determination:

Original: [Equation]
Revised: [Equation]

Original: where d i = x − µ i .
Revised: where d i = x − µ i .

Original: Each inducing point maintains its own mean and covariance parameters, so that
Revised: Each inducing point maintains its own mean and covariance parameters, so that

Original: M = { µ , Σ } Mi =1 .
Revised: M = { µ , Σ } Mi =1 .

Original: The resulting feature vector Φ( x , M ) is given by the concatenation of all kernel values calculated for x in relation to M :
Revised: The resulting feature vector Φ( x , M ) is given by the concatenation of all kernel values calculated for x in relation to M :

Original: [Equation]
Revised: [Equation]

Original: This feature vector lends itself naturally to non-stationarity, since different areas of the input space are governed by their own subset of inducing points, with varying properties.
Revised: This feature vector lends itself naturally to non-stationarity, since different areas of the input space are governed by their own subset of inducing points, with varying properties.

Original: To increase efﬁciency, only a subset of nearest neighbors can be used for feature vector calculation, while all others are set to zero.
Revised: To increase efﬁciency, only a subset of nearest neighbors can be used for feature vector calculation, while all others are set to zero.

Original: Indeed, this feature vector has been successfully applied to accurately reconstruct large-scale 3D datasets at a fraction of the computational cost required by other similar kernel-based techniques (Callaghan & Ramos, 2012), which makes it attractive for big data processing.
Revised: Indeed, this feature vector has been successfully applied to accurately reconstruct large-scale 3D datasets at a fraction of the computational cost required by other similar kernel-based techniques (Callaghan & Ramos, 2012), which makes it attractive for big data processing.

Original: 2. J OINT K ERNEL L EARNING
Revised: J OINT K ERNEL L EARNING

Original: In the original implementation, the parameters { µ , Σ } of each kernel in M are ﬁxed and calculated based on statistical information obtained directly from D .
Revised: In the original implementation, the parameters { µ , Σ } of each kernel in M are ﬁxed and calculated based on statistical information obtained directly from D .

Original: Only the classiﬁer weights w are optimized during the training process, according to Equation 2.
Revised: Only the classiﬁer weights w are optimized during the training process, according to Equation 2.

Original: However, this approach severely limits the descriptiveness of each kernel, especially if training data is not readily available for preprocessing.
Revised: However, this approach severely limits the descriptiveness of each kernel, especially if training data is not readily available for preprocessing.

Original: Here we show how the HM training methodology can be reformulated to include the optimization of all its parameters P = { µ , Σ , w } Mi =1 .
Revised: Here we show how the HM training methodology can be reformulated to include the optimization of all its parameters P = { µ , Σ , w } Mi =1 .

Original: The key insight is realizing that the HM framework is analogous to a neural network layer (Haykin, 1998), in which input data is described as a Gram Matrix (Hazewinkel, 2001) in relation to the inducing set M , such that:
Revised: The key insight is realizing that the HM framework is analogous to a neural network layer (Haykin, 1998), in which input data is described as a Gram Matrix (Hazewinkel, 2001) in relation to the inducing set M , such that:

Original: [Equation]
Revised: [Equation]

Original: where X = { x } Ni =1 are input points, K X M is a N × M matrix with coefﬁcients K ij = k
Revised: where X = { x } N i =1 are input points, K X M is a N × M matrix with coefﬁcients K ij = k

Original: ( x i , M j ) as deﬁned by Equation 3, σ is the sigmoid activation function and b h is an optional bias term.
Revised: ( x i , M j ) as deﬁned by Equation 3, σ is the sigmoid activation function and b h is an optional bias term.

Original: Within this alternative framework, standard back-propagation (LeCun et al., 1998) can be used to jointly optimize kernel parameters, using the corresponding partial derivatives:
Revised: Within this alternative framework, standard back-propagation (LeCun et al., 1998) can be used to jointly optimize kernel parameters, using the corresponding partial derivatives:

Original: [Equation]
Revised: [Equation]

Original: which can be efﬁciently calculated during feature vector generation.
Revised: which can be efﬁciently calculated during feature vector generation.

Original: An example of this joint learning process can be found in Figure 1, for a simple 1D classiﬁcation problem.
Revised: An example of this joint learning process can be found in Figure 1, for a simple 1D classiﬁcation problem.

Original: In the left column, the standard HM framework was used, with only weight learning, whereas in the right column kernel parameters were also learned, using the proposed HL framework (Hilbert Layer).
Revised: In the left column, the standard HM framework was used, with only weight learning, whereas in the right column kernel parameters were also learned, using the proposed HL framework (Hilbert Layer).

Original: In the top rowinducing points were used, initially equally spaced, while in the bottom row only 6 inducing points were used.
Revised: In the top rowinducing points were used, initially equally spaced, while in the bottom row only 6 inducing points were used.

Original: Note that, for higher densities, HM converges to good results, however it fails to capture some occupancy behaviors in lower densities, due to its smaller descriptive power. 
Revised: Note that, for higher densities, HM converges to good results, however it fails to capture some occupancy behaviors in lower densities, due to its smaller descriptive power. On the other

Original: On the other hand, HL is able to achieve considerably better results in both cases, with reasonable convergence even in lower densities.
Revised: [Figure] hand, HL is able to achieve considerably better results in both cases, with reasonable convergence even in lower densities.

Original: Particularly, in Figure 1b we can see how inducing points migrate to discontinuous areas, thus ensuring sharper transitions, and in Figure 1d one inducing points assumed a larger length-scale to reach a particular occupied area that was under-represented.[Figure]
Revised: Particularly, in Figure 1b we can see how inducing points migrate to discontinuous areas, thus ensuring sharper transitions, and in Figure 1d one inducing points assumed a larger length-scale to reach a particular occupied area that was under-represented.

Original: Lastly, note that, while the standard HM framework as described in Section 2.1 only addresses classiﬁcation tasks, the proposed joint learning methodology can be easily modiﬁed to address general regression tasks, simply by removing the activation function σ and optimizing a different loss function (i.e. mean squared error instead of cross-entropy).
Revised: Lastly, note that, while the standard HM framework as described in Section 2.1 only addresses classiﬁcation tasks, the proposed joint learning methodology can be easily modiﬁed to address general regression tasks, simply by removing the activation function σ and optimizing a different loss function (i.e. mean squared error instead of cross-entropy).

Original:  C ONTINUOUS C ONVOLUTIONAL L AYER
Revised:  C ONTINUOUS C ONVOLUTIONAL L AYER

Original: In this section we show how the Hilbert layer, as deﬁned by Equation 5, can be extended to a convolutional scenario, in which two functions in different RKHS are convolved to produce a third
Revised: In this section we show how the Hilbert layer, as deﬁned by Equation 5, can be extended to a convolutional scenario, in which two functions in different RKHS are convolved to produce a third

Original: RKHS deﬁning functions that approximate activation values.
Revised: RKHS deﬁning functions that approximate activation values.

Original: We start by formulating a closed-form solution to the continuous convolution between kernels that describe a feature vector for projection in the HL framework, move on to a formal deﬁnition of the Convolutional Hilbert Layer (CHL), and lastly describe how this novel framework can be used for image classiﬁcation tasks.
Revised: We start by formulating a closed-form solution to the continuous convolution between kernels that describe a feature vector for projection in the HL framework, move on to a formal deﬁnition of the Convolutional Hilbert Layer (CHL), and lastly describe how this novel framework can be used for image classiﬁcation tasks.

Original: 3. C LOSED -F ORM K ERNEL C ONVOLUTION
Revised: 3. C LOSED -F ORM K ERNEL C ONVOLUTION

Original: Convolution is a mathematical operation that takes two functions f and g and produces a third function h
Revised: Convolution is a mathematical operation that takes two functions f and g and produces a third function h

Original: = f ∗ g , that can be viewed as the amount of overlapping between both functions as one is reversed and shifted over the other.
Revised: = f ∗ g , that can be viewed as the amount of overlapping between both functions as one is reversed and shifted over the other.

Original: Formally, it is deﬁned as:
Revised: Formally, it is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Solving Equation 8 analytically can be a difﬁcult (or even impossible) task for most functions that deﬁne real-world phenomena, due to their complexity.
Revised: Solving Equation 8 analytically can be a difﬁcult (or even impossible) task for most functions that deﬁne real-world phenomena, due to their complexity.

Original: However, as shown in Sec. 2.2, the Hilbert
Revised: However, as shown in Sec. 2.2, the Hilbert

Original: Maps framework is able to approximate arbitrarily complex functions using simple kernels, by projecting input data into a high-dimensional RKHS.
Revised: Maps framework is able to approximate arbitrarily complex functions using simple kernels, by projecting input data into a high-dimensional RKHS.

Original: Although the proposed methodology can be applied to any two kernel functions with a closed-form convolution formula, for notation simplicity we assume, without loss of generality, that the kernels describing both functions are given by Equation 3.
Revised: Although the proposed methodology can be applied to any two kernel functions with a closed-form convolution formula, for notation simplicity we assume, without loss of generality, that the kernels describing both functions are given by Equation 3.

Original: This choice greatly simpliﬁes the problem because the convolution of two Gaussian distributions is also a Gaussian distribution with automatic relevance determination (Bromiley, 2003):
Revised: This choice greatly simpliﬁes the problem because the convolution of two Gaussian distributions is also a Gaussian distribution with automatic relevance determination (Bromiley, 2003):

Original: [Equation]
Revised: [Equation]

Original: where M i + M j = { µ i + µ j , Σ i + Σ j }
Revised: where M i + M j = { µ i + µ j , Σ i + Σ j }

Original: Note that this new kernel does not model function states in the RKHS, but rather activation values, representing convolution results between M i and M j , and can be queried at arbitrary resolutions using the HL framework described in Section 2.3.
Revised: Note that this new kernel does not model function states in the RKHS, but rather activation values, representing convolution results between M i and M j , and can be queried at arbitrary resolutions using the HL framework described in Section 2.3.

Original: More importantly, it can be optimized using the same training methodology, to produce better generative or discriminative models.
Revised: More importantly, it can be optimized using the same training methodology, to produce better generative or discriminative models.

Original: 3. C ONVOLUTIONAL H ILBERT L AYER
Revised: 3. C ONVOLUTIONAL H ILBERT L AYER

Original: Now that convolution between kernels that deﬁne functions in the RKHS has been established, here we show how convolution between two parameter sets P f and P g , each representing a different function in its respective RKHS, is performed.
Revised: Now that convolution between kernels that deﬁne functions in the RKHS has been established, here we show how convolution between two parameter sets P f and P g , each representing a different function in its respective RKHS, is performed.

Original: Comparing to the Hilbert layer deﬁned in Equation 5, the convolutional Hilbert Layer takes the form:
Revised: Comparing to the Hilbert layer deﬁned in Equation 5, the convolutional Hilbert Layer takes the form:

Original: [Equation]
Revised: [Equation]

Original: where K kfg is a M f × M g matrix, with coefﬁcients K fg =
Revised: where K kfg is a M f × M g matrix, with coefﬁcients K fg =

Original: k ( µ kh , M if + M jg ) .
Revised: k ( µ kh , M if + M jg ) .

Original: Note that K fg is a block-matrix, so weight multiplications are performed independently for each entry before summation, which beneﬁts the use of parallel computing for faster calculations.
Revised: Note that K fg is a block-matrix, so weight multiplications are performed independently for each entry before summation, which beneﬁts the use of parallel computing for faster calculations.

Original: The output h are the weights that approximate f ∗ g in the RKHS deﬁned by the cluster set M h , and together these compose the parameter set P h
Revised: The output h are the weights that approximate f ∗ g in the RKHS deﬁned by the cluster set M h , and together these compose the parameter set P h

Original: = {M h , h } = { µ h , Σ h , w h } .
Revised: = {M h , h } = { µ h , Σ h , w h } .

Original: Pseudo-code for this convolution process is given in Algorithm 1, where we can see how it operates: for each inducing point in P f and P g , their kernel convolution is calculated (Equation 9) and used to evaluate all points in P h
Revised: Pseudo-code for this convolution process is given in Algorithm 1, where we can see how it operates: for each inducing point in P f and P g , their kernel convolution is calculated (Equation 9) and used to evaluate all points in P h

Original: (Eq. 3), each one contributing to its corresponding weight parameter w kh .
Revised: (Eq. 3), each one contributing to its corresponding weight parameter w kh .

Original: Multiple P input channels and Q ﬁlters can be incorporated by concatenating the various K pqfg into a single block-matrix, while augmenting w f and w g accordingly, such that:
Revised: Multiple P input channels and Q ﬁlters can be incorporated by concatenating the various K pqfg into a single block-matrix, while augmenting w f and w g accordingly, such that:

Original: [Equation]
Revised: [Equation]

Original: where K fg is a P × Q block-matrix with entries K ij =
Revised: where K fg is a P × Q block-matrix with entries K ij =

Original: K pqfg , W [Equation] (cid:104)(cid:2) w 1 g (cid:3) , ..., (cid:2) w Qg (cid:3)(cid:105) , b h is now a Q × 1 bias vector and P h
Revised: K pqfg , W f = (cid:104)(cid:2) w 1 f (cid:3) , ..., (cid:2) w Pf (cid:3)(cid:105) , W g = (cid:104)(cid:2) w 1 g (cid:3) , ..., (cid:2) w Qg (cid:3)(cid:105) , b h is now a Q × 1 bias vector and P h

Original: = {M h , H } = { µ h , Σ h , { w } Qq =1 } contains multiple weight channels deﬁned in the same RKHS.
Revised: = {M h , H } = { µ h , Σ h , { w } Qq =1 } contains multiple weight channels deﬁned in the same RKHS.

Original: An example of the output produced by a convolutional Hilbert Layer, for a simple 1D classiﬁcation problem, is depicted in Figure 2, alongside the corresponding discrete convolution results.
Revised: An example of the output produced by a convolutional Hilbert Layer, for a simple 1D classiﬁcation problem, is depicted in Figure 2, alongside the corresponding discrete convolution results.

Original: As expected, areas in the input that are similar to the ﬁlter have higher activation values, and these vary continuously throughout the input space, being able to capture small variations and partial matches to a higher detail than discrete convolution.
Revised: As expected, areas in the input that are similar to the ﬁlter have higher activation values, and these vary continuously throughout the input space, being able to capture small variations and partial matches to a higher detail than discrete convolution.

Original: Lastly, note that pooling (Krizhevsky et al., 2012) can be naturally incorporated into the convolutional Hilbert layer simply by decreasing the density of clusters in the output RKHS.
Revised: Lastly, note that pooling (Krizhevsky et al., 2012) can be naturally incorporated into the convolutional Hilbert layer simply by decreasing the density of clusters in the output RKHS.

Original: This processhas two effects: 1) decrease computational cost, which allows for larger ﬁlter sizes and number of channels while combating over-ﬁtting; and 2) decrease the resolution of the approximated continuous function, thus aggregating statistics of spatially close regions to capture patterns in a larger scale. An un-pooling effect is similarly straightforward, generated by increasing the density of clusters in the output RKHS, thus increasing the resolution of the approximated continuous function at the expense of a larger number of parameters.
Revised: This process

Original: Algorithm 1 Convolutional Hilbert Layer
Revised: Algorithm 1 Convolutional Hilbert Layer

Original: Require: Input P f and ﬁlter P g parameter sets, convolved
Revised: Require: Input P f and ﬁlter [Table] [Table] [Table]

Original: Convolved P
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: has two effects: 1) decrease computational cost, which allows for larger ﬁlter sizes and number of channels while combating over-ﬁtting; and 2) decrease the resolution of the approximated continuous function, thus aggregating statistics of spatially close regions to capture patterns in a larger scale.

Revised: An un-pooling effect is similarly straightforward, generated by increasing the density of clusters in the output RKHS, thus increasing the resolution of the approximated continuous function at the expense of a larger number of parameters.

Original: 3. C LASSIFICATION T OPOLOGY
Revised: 3. C LASSIFICATION T OPOLOGY

Original: A diagram depicting how the proposed convolutional Hilbert layer can be applied to an image classiﬁcation task, to create a Continuous Convolutional Neural Network (CCNN), is shown in Figure 3.
Revised: A diagram depicting how the proposed convolutional Hilbert layer can be applied to an image classiﬁcation task, to create a Continuous Convolutional Neural Network (CCNN), is shown in Figure 3.

Original: The original image, composed of discrete data D n = { x , y n } , with x ∈ R 2 being the pixel coordinates and y n = [0 , 1] C 0 their corresponding intensity values ( C 0 is the number of input channels, i.e. 1 for grayscale and 3 for color images), is ﬁrst modeled as a continuous function via projection to a RKHS.
Revised: The original image, composed of discrete data D n = { x , y n } , with x ∈ R 2 being the pixel coordinates and y n = [0 , 1] C 0 their corresponding intensity values ( C 0 is the number of input channels, i.e. 1 for grayscale and 3 for color images), is ﬁrst modeled as a continuous function via projection to a RKHS.

Original: Note that the same RKHS is used to model all input images, deﬁned by the cluster set
Revised: Note that the same RKHS is used to model all input images, deﬁned by the cluster set

Original: M f 0 , and within this projection each image is represented with a different set of model weights
Revised: M f 0 , and within this projection each image is represented with a different set of model weights

Original: W nf 0 .
Revised: W nf 0 .

Original: The resulting parameter set P nf is convolved with the ﬁlters contained in P g 1 to produce the hidden feature maps P f 1 , that also share the same RKHS for all input images, deﬁned by M f 1 , but with individual model weights W nf 1 .
Revised: The resulting parameter set P nf is convolved with the ﬁlters contained in P g 1 to produce the hidden feature maps P f 1 , that also share the same RKHS for all input images, deﬁned by M f 1 , but with individual model weights W nf 1 .

Original: This process is repeated for each convolutional Hilbert layer, with the ﬁnal model weights being ﬂattened to serve as input for a standard fully connected neural network, that performs the classiﬁcation between different categories.
Revised: This process is repeated for each convolutional Hilbert layer, with the ﬁnal model weights being ﬂattened to serve as input for a standard fully connected neural network, that performs the classiﬁcation between different categories.

Original: The convolutional trainable parameters to be optimized in this topology are the various cluster sets M f = { µ , Σ } , that deﬁne the RKHS for each feature map, and the various ﬁlter sets
Revised: The convolutional trainable parameters to be optimized in this topology are the various cluster sets M f = { µ , Σ } , that deﬁne the RKHS for each feature map, and the various ﬁlter sets

Original: P g = { µ , Σ , W } , that perform the transformation between feature maps in each layer.
Revised: P g = { µ , Σ , W } , that perform the transformation between feature maps in each layer.

Original: Note that each of these parameters represents a different property of the convolution process: µ deﬁnes location, Σ deﬁnes length-scale and W deﬁnes weight, and therefore should be treated accordingly during optimization.
Revised: Note that each of these parameters represents a different property of the convolution process: µ deﬁnes location, Σ deﬁnes length-scale and W deﬁnes weight, and therefore should be treated accordingly during optimization.

Original: To improve parameter initialization, we employ a continuous fully-convolutional auto-encoder (CCAE), that ﬁrst encodes data into a lower-dimensional latent feature vector representation and then decodes it back to produce a reconstruction of the original input.
Revised: To improve parameter initialization, we employ a continuous fully-convolutional auto-encoder (CCAE), that ﬁrst encodes data into a lower-dimensional latent feature vector representation and then decodes it back to produce a reconstruction of the original input.

Original: Particularly, the encoding pipeline is composed by the convolutional Hilbert layers from the classiﬁcation topology, and the decoding pipeline has these same layers in reverse order (without parameter sharing), as depicted in Figure 4.
Revised: Particularly, the encoding pipeline is composed by the convolutional Hilbert layers from the classiﬁcation topology, and the decoding pipeline has these same layers in reverse order (without parameter sharing), as depicted in Figure 4.

Original: A lower-dimensional representation is achieved by decreasing the number of clusters used for feature map projection in deeper layers, thus simulating a pooling effect.
Revised: A lower-dimensional representation is achieved by decreasing the number of clusters used for feature map projection in deeper layers, thus simulating a pooling effect.

Original: Similarly, the number of clusters used for ﬁlter projection can also be modiﬁed, emulating different kernel sizes in standard discrete convolution, however since the location of these clusters is a trainable parameter their support is inherently adaptive, only changing in complexity as more clusters are added.
Revised: Similarly, the number of clusters used for ﬁlter projection can also be modiﬁed, emulating different kernel sizes in standard discrete convolution, however since the location of these clusters is a trainable parameter their support is inherently adaptive, only changing in complexity as more clusters are added.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Here we present and discuss experimental results performed to validate the proposed convolutional
Revised: Here we present and discuss experimental results performed to validate the proposed convolutional

Original: Hilbert layer in an image classiﬁcation scenario 1 .
Revised: Hilbert layer in an image classiﬁcation scenario 1 .

Original: Four different standard benchmarks were considered: the MNIST dataset, composed of 60000 training and 10000 test images with dimensionality  28 × 28 × 1 ; the CIFAR-10 dataset, composed of 50000 training and 10000 test images with dimensionality 32 × 32 × 3 ; the STL-10 dataset, composed of 5000 training and 8000 tests images with dimensionality 96 × 96 × 3 plus 100000 unlabeled images; and the SVHN dataset, composed of 604388 training and 26032 test images with dimensionality 32 × 32 × 3 .
Revised: Four different standard benchmarks were considered: the MNIST dataset, composed of 60000 training and 10000 test images with dimensionality anonymity [Figure] [Figure] 28 × 28 × 1 ; the CIFAR-10 dataset, composed of 50000 training and 10000 test images with dimensionality 32 × 32 × 3 ; the STL-10 dataset, composed of 5000 training and 8000 tests images with dimensionality 96 × 96 × 3 plus 100000 unlabeled images; and the SVHN dataset, composed of 604388 training and 26032 test images with dimensionality 32 × 32 × 3 .

Original: No preprocessing or data augmentation of any kind was performed in these datasets during training or test phases.
Revised: No preprocessing or data augmentation of any kind was performed in these datasets during training or test phases.

Original: Examples of reconstruction results for the MNIST and CIFAR-10 datasets, using the proposed HL framework, are depicted in Figure 5.
Revised: Examples of reconstruction results for the MNIST and CIFAR-10 datasets, using the proposed HL framework, are depicted in Figure 5.

Original: These results were obtained by projecting all images from each dataset into the same input RKHS, deﬁned by the cluster set M f 0 , and producing the weight parameters W nf 0 individual to each image (for RGB images, each channel was treated independently).Both [Figure] [Figure]
Revised: These results were obtained by projecting all images from each dataset into the same input RKHS, deﬁned by the cluster set M f 0 , and producing the weight parameters W nf 0 individual to each image (for RGB images, each channel was treated independently).

Original:  the cluster set parameters and individual model weights were optimized using the proposed joint learning methodology from Section 2.3, to minimize the squared reconstruction error.
Revised: Both the cluster set parameters and individual model weights were optimized using the proposed joint learning methodology from Section 2.3, to minimize the squared reconstruction error.

Original: Once training was complete, these parameters served as input for a continuous convolutional neural network (CCNN) for image classiﬁcation, in which each projected image is mapped to its corresponding label via cross-entropy minimization and a softmax activation function in the output layer (see Figure 3).
Revised: Once training was complete, these parameters served as input for a continuous convolutional neural network (CCNN) for image classiﬁcation, in which each projected image is mapped to its corresponding label via cross-entropy minimization and a softmax activation function in the output layer (see Figure 3).

Original: To initialize the convolutional parameters of this network, a continuous convolutional auto-encoder (CCAE) was used, mirroring convolutional layers to produce a ﬁnal reconstruction of the original input, via direct squared error minimization over the discretized output (see Figure 4).
Revised: To initialize the convolutional parameters of this network, a continuous convolutional auto-encoder (CCAE) was used, mirroring convolutional layers to produce a ﬁnal reconstruction of the original input, via direct squared error minimization over the discretized output (see Figure 4).

Original: To test the expressiveness of the proposed continuous feature maps, we compared CCNN image classiﬁcation results against the standard DCNN (Discrete Convolutional Neural Network) architecture, in the special case when very few ﬁlters are used (here, ranging from 1 to 20 , with size 3 ×for discrete and 9 clusters for continuous).
Revised: To test the expressiveness of the proposed continuous feature maps, we compared CCNN image classiﬁcation results against the standard DCNN (Discrete Convolutional Neural Network) architecture, in the special case when very few ﬁlters are used (here, ranging from 1 to 20 , with size 3 ×for discrete and 9 clusters for continuous).

Original: A single convolutional layer was used, followed by a fully-connected layer with 200 nodes and the output layer (no dropout or regularization of any sort was used).
Revised: A single convolutional layer was used, followed by a fully-connected layer with 200 nodes and the output layer (no dropout or regularization of any sort was used).

Original: Classiﬁcation results for the MNIST dataset are depicted in Figure 6, where we can see that a single continuous ﬁlter is able to achieve better overall results than all twenty discrete ﬁlters, both in training and test accuracy.
Revised: Classiﬁcation results for the MNIST dataset are depicted in Figure 6, where we can see that a single continuous ﬁlter is able to achieve better overall results than all twenty discrete ﬁlters, both in training and test accuracy.

Original: Interestingly, while a single discrete ﬁlter actually achieves worst loss function training values than a straightforward fully connected neural network (FCNN) without convolutional layers, a single continuous ﬁlter continues to improve the loss function over time, which was still decreasing after the alloted number of iterations.
Revised: Interestingly, while a single discrete ﬁlter actually achieves worst loss function training values than a straightforward fully connected neural network (FCNN) without convolutional layers, a single continuous ﬁlter continues to improve the loss function over time, which was still decreasing after the alloted number of iterations.

Original: We also noticed less over-ﬁtting, as it can be shown by loss function values for testing data, that started to consistently increase for
Revised: We also noticed less over-ﬁtting, as it can be shown by loss function values for testing data, that started to consistently increase for

Original: DCNN after a certain number of training iterations, while CCNN was able to maintain lower values throughout the entire training process.
Revised: DCNN after a certain number of training iterations, while CCNN was able to maintain lower values throughout the entire training process.

Original: Furthermore, we can see that CCNN produces much larger ranges of loss function values both for training and testing data, indicating that the choice of initial parameter values play a more signiﬁcant role during the optimization process, especially when fewer ﬁlters are considered (which is to be expected, since they are able to capture a larger range of patterns to use during the convolution process).
Revised: Furthermore, we can see that CCNN produces much larger ranges of loss function values both for training and testing data, indicating that the choice of initial parameter values play a more signiﬁcant role during the optimization process, especially when fewer ﬁlters are considered (which is to be expected, since they are able to capture a larger range of patterns to use during the convolution process).

Original: Examples of convolutional ﬁlters obtained in these experiments are depicted in Figure 7, where we can see their variability and ability to model different patterns that will be useful during the classiﬁcation process.
Revised: Examples of convolutional ﬁlters obtained in these experiments are depicted in Figure 7, where we can see their variability and ability to model different patterns that will be useful during the classiﬁcation process.

Original: Classiﬁcation results, in terms of percentual test accuracy error, for the three datasets considered here are presented in Tables 1-4, in relation to other image classiﬁcation techniques found in the literature.
Revised: Classiﬁcation results, in terms of percentual test accuracy error, for the three datasets considered here are presented in Tables 1-4, in relation to other image classiﬁcation techniques found in the literature.

Original: The same CCNN architecture was used in all cases, composed of three convolutional layers with 20-40-60 ﬁlters of sizes 25-16-9 and pooling ratios of 2-3-4 in relation to input data dimensionality, followed by two fully-connected layers with 512-1024 nodes and 0.5 dropout (Srivastava et al., 2014).
Revised: The same CCNN architecture was used in all cases, composed of three convolutional layers with 20-40-60 ﬁlters of sizes 25-16-9 and pooling ratios of 2-3-4 in relation to input data dimensionality, followed by two fully-connected layers with 512-1024 nodes and 0.5 dropout (Srivastava et al., 2014).

Original: Note that this architecture is much simpler than the ones presented by networks capable of achieving state-of-the-art classiﬁcation results in these datasets, possessing a total ofconvolutional parameters, 588 + 261 + 147 = 996 of which deﬁne the intermediary RKHS for feature maps representation and the remaining 6000 + 51200 + 86400 = 143600 representing the ﬁlters within these RKHS. 
Revised: Note that this architecture is much simpler than the ones presented by networks capable of achieving state-of-the-art classiﬁcation results in these datasets, possessing a total ofconvolutional parameters, 588 + 261 + 147 = 996 of which deﬁne the intermediary RKHS for feature maps representation and the remaining 6000 + 51200 + 86400 = 143600 representing the ﬁlters within these RKHS. Nevertheless, we can see that the proposed convolutional Hilbert layer is [Table]

Original: Nevertheless, we can see that the proposed convolutional Hilbert layer is able to achieve competitive results in all three datasets, even with such shallow and narrow architecture, which further exempliﬁes the descriptive power of a continuous representation when applied in conjunction with the convolution operation.
Revised: [Table] [Table] [Table] able to achieve competitive results in all three datasets, even with such shallow and narrow architecture, which further exempliﬁes the descriptive power of a continuous representation when applied in conjunction with the convolution operation.

Original: Particularly, the introduction of unsupervised pretraining, using the proposed CCAE architecture to generate initial parameter estimates, signiﬁcantly improves accuracy results.
Revised: Particularly, the introduction of unsupervised pretraining, using the proposed CCAE architecture to generate initial parameter estimates, signiﬁcantly improves accuracy results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper introduced a novel technique for data representation that takes place in a highdimensional Reproducing Kernel Hilbert Space (RKHS), where arbitrarily complex functions can be approximated in a continuous fashion using a series of simple kernels.
Revised: This paper introduced a novel technique for data representation that takes place in a highdimensional Reproducing Kernel Hilbert Space (RKHS), where arbitrarily complex functions can be approximated in a continuous fashion using a series of simple kernels.

Original: We show how these kernels can be efﬁciently convolved to produce approximations of convolution results between two functions in different RKHS, and how this can be applied in an image classiﬁcation scenario, via the introMethod [Table] Frac. Max-Pooling (Graham,
Revised: We show how these kernels can be efﬁciently convolved to produce approximations of convolution results between two functions in different RKHS, and how this can be applied in an image classiﬁcation scenario, via the introduction of a novel deep learning architecture entitled Continuous Convolutional Neural Networks (CCNN).

Original: [Table]

Original: [Table]

Original: CCNN (CCAE init.)

Original: [Table]

Original: Conv.

Original: Kernel Net.

Original: (Mairal et al., 2014)

Original: [Table]

Original: Maxout Net.

Original: (Goodfellow et al., 2013)

Original: [Table]

Original: CCNN

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: duction of a novel deep learning architecture entitled Continuous Convolutional Neural Networks (CCNN).

Original: Experimental tests using standard benchmark datasets show that this proposed architecture is able to achieve competitive results with much smaller network sizes, by focusing instead on more descriptive individual ﬁlters that are used to extract more complex patterns.
Revised: Experimental tests using standard benchmark datasets show that this proposed architecture is able to achieve competitive results with much smaller network sizes, by focusing instead on more descriptive individual ﬁlters that are used to extract more complex patterns.

Original: Although promising, there are still several potential improvements that are left for future work, such as: RKHS sparsiﬁcation, so only a subset of clusters are used for feature vector calculation, which would greatly improve computational speed and memory requirements; different learning rates and optimization strategies for each class of parameter (cluster location, length-scale and weight), to improve convergence rates; and the use of different kernels for feature vector representation, as a way to encode different properties in the resulting feature maps.
Revised: Although promising, there are still several potential improvements that are left for future work, such as: RKHS sparsiﬁcation, so only a subset of clusters are used for feature vector calculation, which would greatly improve computational speed and memory requirements; different learning rates and optimization strategies for each class of parameter (cluster location, length-scale and weight), to improve convergence rates; and the use of different kernels for feature vector representation, as a way to encode different properties in the resulting feature maps.