Original: W HY C ONVOLUTIONAL N ETWORKS L EARN O RIENTED B ANDPASS F ILTERS : A H YPOTHESIS
Revised: W HY C ONVOLUTIONAL N ETWORKS L EARN O RIENTED B ANDPASS F ILTERS : A H YPOTHESIS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass ﬁlters.
Revised: It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass ﬁlters.

Original: A standard explanation of this result is that these ﬁlters reﬂect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass ﬁlters are matched to such structure.
Revised: A standard explanation of this result is that these ﬁlters reﬂect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass ﬁlters are matched to such structure.

Original: The present paper offers an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures.
Revised: The present paper offers an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures.

Original: In particular, complex exponentials are the eigenfunctions of convolution.
Revised: In particular, complex exponentials are the eigenfunctions of convolution.

Original: These eigenfunctions are deﬁned globally; however, convolutional architectures operate locally.
Revised: These eigenfunctions are deﬁned globally; however, convolutional architectures operate locally.

Original: To enforce locality, one can apply a windowing function to the eigenfunctions, which leads to oriented bandpass ﬁlters as the natural operators to be learned with convolutional architectures.
Revised: To enforce locality, one can apply a windowing function to the eigenfunctions, which leads to oriented bandpass ﬁlters as the natural operators to be learned with convolutional architectures.

Original: From a representational point of view, these ﬁlters allow for a local systematic way to characterize and operate on an image or other signal.
Revised: From a representational point of view, these ﬁlters allow for a local systematic way to characterize and operate on an image or other signal.

Original:  I NTRODUCTION 1. M OTIVATION
Revised:  I NTRODUCTION 1. M OTIVATION

Original: Convolutional networks (ConvNets) in conjunction with deep learning have shown state-of-the-art performance in application to computer vision, ranging across both classiﬁcation, e.g., (Krizhevsky et al., 2012; Tran et al., 2015;
Revised: Convolutional networks (ConvNets) in conjunction with deep learning have shown state-of-the-art performance in application to computer vision, ranging across both classiﬁcation, e.g., (Krizhevsky et al., 2012; Tran et al., 2015;

Original: Ge et al., 2019) and regression, e.g., (Szegedy et al., 2013; Eigen &
Revised: Ge et al., 2019) and regression, e.g., (Szegedy et al., 2013; Eigen &

Original: Fergus, 2015; Zhou et al., 2017) tasks.
Revised: Fergus, 2015; Zhou et al., 2017) tasks.

Original: However, understanding of how these systems achieve their remarkable results lags behind their performance.
Revised: However, understanding of how these systems achieve their remarkable results lags behind their performance.

Original: This state of affairs is unsatisfying not only from a scientiﬁc point of view, but also from an applications point of view.
Revised: This state of affairs is unsatisfying not only from a scientiﬁc point of view, but also from an applications point of view.

Original: As these systems move beyond the lab into real-world applications better theoretical understanding can help establish performance bounds and increase conﬁdence in deployment.
Revised: As these systems move beyond the lab into real-world applications better theoretical understanding can help establish performance bounds and increase conﬁdence in deployment.

Original: Visualization studies of ﬁlters that have been learned during training have been one of the key tools marshaled to lend insight into the internal representations maintained by ConvNets in application to computer vision, e.g., (Zeiler & Fergus, 2014; Yosinski et al., 2015; Mahendran & Vedaldi, 2015;
Revised: Visualization studies of ﬁlters that have been learned during training have been one of the key tools marshaled to lend insight into the internal representations maintained by ConvNets in application to computer vision, e.g., (Zeiler & Fergus, 2014; Yosinski et al., 2015; Mahendran & Vedaldi, 2015;

Original: Shang et al., 2016; Feichtenhofer et al., 2018).
Revised: Shang et al., 2016; Feichtenhofer et al., 2018).

Original: Here, an interesting repeated observation is that early layers in the studied networks tend to learn oriented bandpass ﬁlters, both in two image spatial dimenstions, ( x, y ) (cid:62) , in application to single image analysis as well as in three spatiotemporal dimensions, ( x, y, t ) (cid:62) , in application to video.
Revised: Here, an interesting repeated observation is that early layers in the studied networks tend to learn oriented bandpass ﬁlters, both in two image spatial dimenstions, ( x, y ) (cid:62) , in application to single image analysis as well as in three spatiotemporal dimensions, ( x, y, t ) (cid:62) , in application to video.

Original: An example is shown in Figure 1.
Revised: An example is shown in Figure 1.

Original: Emergence of such ﬁlters seems reasonable, because local orientation captures the ﬁrst-order correlation structure of the data, which provides a reasonable building block for inferring more complex structure (e.g., local measurements of oriented structure can be assembled into intersections to capture corner structure, etc.).
Revised: Emergence of such ﬁlters seems reasonable, because local orientation captures the ﬁrst-order correlation structure of the data, which provides a reasonable building block for inferring more complex structure (e.g., local measurements of oriented structure can be assembled into intersections to capture corner structure, etc.).

Original: Notably, however, more rigorous analyses of exactly why oriented bandpass ﬁlters might be learned has been limited.
Revised: Notably, however, more rigorous analyses of exactly why oriented bandpass ﬁlters might be learned has been limited.

Original: This state of affairs motivates the current paper in its argument that the analytic structure of ConvNets constrains them to learn oriented bandpass ﬁlters.
Revised: This state of affairs motivates the current paper in its argument that the analytic structure of ConvNets constrains them to learn oriented bandpass ﬁlters.

Original: 1. R ELATED RESEARCH
Revised: 1. R ELATED RESEARCH

Original: Visualization of receptive ﬁeld proﬁles (i.e., pointspread functions (Lim, 1990)) of the convolutional ﬁlters learned by contemporary ConvNets is a popular tool for providing insight into the image properties that are being represented by a network.
Revised: Visualization of receptive ﬁeld proﬁles (i.e., pointspread functions (Lim, 1990)) of the convolutional ﬁlters learned by contemporary ConvNets is a popular tool for providing insight into the image properties that are being represented by a network.

Original: A notable trend across these studies is that early layers appear to learn oriented bandpass ﬁlters in both two spatial dimensions, e.g., (Zeiler & Fergus, 2014;
Revised: A notable trend across these studies is that early layers appear to learn oriented bandpass ﬁlters in both two spatial dimensions, e.g., (Zeiler & Fergus, 2014;

Original: Springenberg et al., 2015; Yosinski et al., 2015; Shang et al., 2016) as well as three spatiotemporal dimensions, e.g., (Feichtenhofer et al., 2018).
Revised: Springenberg et al., 2015; Yosinski et al., 2015; Shang et al., 2016) as well as three spatiotemporal dimensions, e.g., (Feichtenhofer et al., 2018).

Original: Indeed, earlier studies with architectures that also constrained their ﬁlters to be convolutional in nature, albeit using a Hebbian learning strategy (MacKay, 2003) rather than the currently dominant back-propagation approach (Rumelhart et al., 1986), also yielded ﬁlters that visualized as having oriented bandpass ﬁlter characteristics (Linsker, 1986).
Revised: Indeed, earlier studies with architectures that also constrained their ﬁlters to be convolutional in nature, albeit using a Hebbian learning strategy (MacKay, 2003) rather than the currently dominant back-propagation approach (Rumelhart et al., 1986), also yielded ﬁlters that visualized as having oriented bandpass ﬁlter characteristics (Linsker, 1986).

Original: Interestingly, biological vision systems also are known to show the presence of oriented bandpass ﬁlters at their earlier layers of processing in visual cortex;  for  reviewsee, e.g., DeValois & DeValois (1988).
Revised: Interestingly, biological vision systems also are known to show the presence of oriented bandpass ﬁlters at their earlier layers of processing in visual cortex; see Hubel & Wiesel (1962) for pioneering work along these lines and for more general review DeValois & DeValois (1988).

Original: The presence of oriented bandpass ﬁlters in biological systems often has been attributed to their being well matched to the statistics of natural images (Field, 1987; Olshausen & Field, 1996;  Simoncelli & Olshausen, 2001), e.g., the dominance of oriented contours at multiple scales.
Revised: The presence of oriented bandpass ﬁlters in biological systems often has been attributed to their being well matched to the statistics of natural images (Field, 1987; Olshausen & Field, 1996; Karklin & Lewicki, 2009; Simoncelli & Olshausen, 2001), e.g., the dominance of oriented contours at multiple scales.

Original: Similar arguments have been made regarding why such ﬁlters are learned by ConvNets.
Revised: Similar arguments have been made regarding why such ﬁlters are learned by ConvNets.

Original: Signiﬁcantly, however, studies have shown that even when trained with images comprised of random noise patterns, convolutional architectures still learn oriented bandpass ﬁlters (Linsker, 1986).
Revised: Signiﬁcantly, however, studies have shown that even when trained with images comprised of random noise patterns, convolutional architectures still learn oriented bandpass ﬁlters (Linsker, 1986).

Original: These later results suggest that the emergence of such ﬁlter tunings cannot be solely attributed to systems being driven to learn ﬁlters that were matched to their training data.
Revised: These later results suggest that the emergence of such ﬁlter tunings cannot be solely attributed to systems being driven to learn ﬁlters that were matched to their training data.

Original: Interestingly, some recent multilayer convolutional architectures have speciﬁed their earliest layers to have oriented bandpass characteristics (Bruna & Mallat, 2013; Jacobsen et al., 2016; Hadji &
Revised: Interestingly, some recent multilayer convolutional architectures have speciﬁed their earliest layers to have oriented bandpass characteristics (Bruna & Mallat, 2013; Jacobsen et al., 2016; Hadji &

Original: Wildes, 2017); indeed, some have speciﬁed such ﬁlters across all layers (Bruna & Mallat, 2013;
Revised: Wildes, 2017); indeed, some have speciﬁed such ﬁlters across all layers (Bruna & Mallat, 2013;

Original: Hadji & Wildes, 2017).
Revised: Hadji & Wildes, 2017).

Original: These design decisions have been variously motivated in terms of being well matched to primitive image structure (Hadji & Wildes, 2017) or providing useful building blocks for learning higher-order structures (Jacobsen et al., 2016) and capturing invariances (Bruna & Mallat, 2013); however, none have motivated their choices in terms of inherent properties of convolutional architectures.
Revised: These design decisions have been variously motivated in terms of being well matched to primitive image structure (Hadji & Wildes, 2017) or providing useful building blocks for learning higher-order structures (Jacobsen et al., 2016) and capturing invariances (Bruna & Mallat, 2013). Other work has noted that purely mathematical considerations show that ConNets are well suited to designs that capture multiscale, windowed spectra Bruna et al.

Revised: (2016); however,it did not explictly established the relationship to eigenfunctions of convolution nor offered an explanation for why deep-learning yields oriented bandpass ﬁlters when applied to ConvNets.

Original: In the light of previous research, the present work appears to be the ﬁrst to offer an explanation of why ConvNets learn oriented bandpass ﬁlters by appeal to the inherent properties of their architectures.
Revised: In the light of previous research, the present work appears to be the ﬁrst to offer an explanation of why ConvNets learn oriented bandpass ﬁlters by appeal to the inherent properties of their architectures.

Original: By deﬁnition, the convolutional layers of a ConvNet are governed by the properties of convolution.
Revised: By deﬁnition, the convolutional layers of a ConvNet are governed by the properties of convolution.

Original: For present purposes, a key property is that the eigenfunctions of convolution are com- plex exponentials.
Revised: For present purposes, a key property is that the eigenfunctions of convolution are complex exponentials.

Original: Imposing locality on the eigenfunctions leads to oriented bandpass ﬁlters, which therefore are the appropriate ﬁlters to be learned by a ConvNet.
Revised: Imposing locality on the eigenfunctions leads to oriented bandpass ﬁlters, which therefore are the appropriate ﬁlters to be learned by a ConvNet.

Original:  A NALYTIC APPROACH
Revised:  A NALYTIC APPROACH

Original: This section details a novel explanation for why ConvNets learn oriented bandpass ﬁlters.
Revised: This section details a novel explanation for why ConvNets learn oriented bandpass ﬁlters.

Original: The ﬁrst two subsections largely review standard material regarding linear systems theory (Oppenheim et al., 1983) and related topics (Kaiser, 2011; Kusse & Westwig, 2006), but are necessary to motivate properly our explanation.
Revised: The ﬁrst two subsections largely review standard material regarding linear systems theory (Oppenheim et al., 1983) and related topics (Kaiser, 2011; Kusse & Westwig, 2006), but are necessary to motivate properly our explanation.

Original: The ﬁnal subsection places the material in the speciﬁc context of ConvNets.
Revised: The ﬁnal subsection places the material in the speciﬁc context of ConvNets.

Original: 2. E IGENFUNCTIONS OF CONVOLUTION
Revised: 2. E IGENFUNCTIONS OF CONVOLUTION

Original: Let L be a linear operator on a function space.
Revised: Let L be a linear operator on a function space.

Original: The set of eigenfunctions φ n associated with this operator satisfy the condition (Kusse & Westwig, 2006)
Revised: The set of eigenfunctions φ n associated with this operator satisfy the condition (Kusse & Westwig, 2006)

Original: [Equation]
Revised: [Equation]

Original: That is, the operator acts on the eigenfunctions simply via multiplication with a constant, λ n , referred to as the eigenvector.
Revised: That is, the operator acts on the eigenfunctions simply via multiplication with a constant, λ n , referred to as the eigenvalue.

Original: It sometimes also is useful to introduce a (positive deﬁnite) weighting function, w , which leads to the corresponding constraint
Revised: It sometimes also is useful to introduce a (positive deﬁnite) weighting function, w , which leads to the corresponding constraint

Original: [Equation]
Revised: [Equation]

Original: For cases where any function in the space can be expanded as a linear sum of the eigenfunctions, it is said that the collection of eignefunctions form a complete set.
Revised: For cases where any function in the space can be expanded as a linear sum of the eigenfunctions, it is said that the collection of eigenfunctions form a complete set.

Original: Such a set provides a convenient and canonical spanning representation.
Revised: Such a set provides a convenient and canonical spanning representation.

Original: Let x = ( x 1 , x 2 , . . .
Revised: Let x = ( x 1 , x 2 , . . .

Original: , x n ) (cid:62) , a = ( a 1 , a 2 , . . .
Revised: , x n ) (cid:62) , a = ( a 1 , a 2 , . . .

Original: , a n ) (cid:62) and u = ( u 1 , u 2 , . . .
Revised: , a n ) (cid:62) and u = ( u 1 , u 2 , . . .

Original: , u n ) (cid:62) .
Revised: , u n ) (cid:62) .

Original: For the space of convolutions, with the convolution of two functions, f ( x ) and h ( x ) deﬁned as
Revised: For the space of convolutions, with the convolution of two functions, f ( x ) and h ( x ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: it is well known that functions of the form f ( x ) =
Revised: it is well known that functions of the form f ( x ) =

Original: e i u (cid:62) x are eigenfunctions of convolution (Oppenheim et al., 1983), i.e.,
Revised: e i u (cid:62) x are eigenfunctions of convolution (Oppenheim et al., 1983), i.e.,

Original: [Equation]
Revised: [Equation]

Original: with the equality achieved via appealing to e i u (cid:62) ( x 
Revised: with the equality achieved via appealing to e i u (cid:62) ( x − a )

Original: [Equation] a ) =
Revised: =

Original: e i u (cid:62) x e − i u (cid:62) a and subsequently factoring e i u (cid:62) x outside the integral as it is independent of a .
Revised: e i u (cid:62) x e − i u (cid:62) a and subsequently factoring e i u (cid:62) x outside the integral as it is independent of a .

Original: The integral on the right hand side of (4), 
Revised: The integral on the right hand side of (4), (cid:90)

Original: [Equation]
Revised: [Equation]

Original: is the eigenvector, referred to as the modulation transfer function (MTF) in signal processing (Oppenheim et al., 1983).
Revised: is the eigenvalue, referred to as the modulation transfer function (MTF) in signal processing (Oppenheim et al., 1983).

Original: Noting that e i u (cid:62) x
Revised: Noting that e i u (cid:62) x

Original: = cos( u (cid:62) x ) + i sin( u (cid:62) x ) leads to the standard interpretation of u in terms of frequency of the function (e.g., input signal).
Revised: = cos( u (cid:62) x ) + i sin( u (cid:62) x ) leads to the standard interpretation of u in terms of frequency of the function (e.g., input signal).

Original: Given the eigenfunctions of convolution are parameterized in terms of their frequencies, it is useful to appeal to the Fourier transform of function f ( x ) , where we use the form (Horn, 1986)
Revised: Given the eigenfunctions of convolution are parameterized in terms of their frequencies, it is useful to appeal to the Fourier transform of function f ( x ) , where we use the form (Horn, 1986)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: because any convolution can be represented in terms of how it operates via simple multiplication of the eigenvectors, (5), with the eigenfunctions, e i u (cid:62) x , with u given by (6).
Revised: because any convolution can be represented in terms of how it operates via simple multiplication of the eigenvectors, (5), with the eigenfunctions, e i u (cid:62) x , with u given by (6).

Original: Thus, this decomposition provides a canonical way to decompose f ( x ) and explicate how a convolution operates on it.
Revised: Thus, this decomposition provides a canonical way to decompose f ( x ) and explicate how a convolution operates on it.

Original: 2. I MPOSING LOCALITY
Revised: 2. I MPOSING LOCALITY

Original: Understanding convolution purely in terms of its eigenvectors and eigenvalues provides only a global representation of operations, as notions of signal locality, x , are lost in the global transformation to the frequency domain, u .
Revised: Understanding convolution purely in terms of its eigenvectors and eigenvalues provides only a global representation of operations, as notions of signal locality, x , are lost in the global transformation to the frequency domain, u .

Original: This state of affairs often is unsatisfactory from a representational point of view because one wants to understand the structure of the signal (e.g., an image) on a more local basis (e.g., one wants to detect objects as well as their image coordinates).
Revised: This state of affairs often is unsatisfactory from a representational point of view because one wants to understand the structure of the signal (e.g., an image) on a more local basis (e.g., one wants to detect objects as well as their image coordinates).

Original: This limitation can be ameliorated by deﬁning a windowed Fourier transform (Kaiser, 2011), as follows (Jahne &
Revised: This limitation can be ameliorated by deﬁning a windowed Fourier transform (Kaiser, 2011), as follows (Jahne &

Original: Hausbecker, 2000).
Revised: Hausbecker, 2000).

Original: Let w ( x ) be a windowing function that is positive valued, symmetric and monotonically decreasing from its center so as to provide greatest emphasis at its center.
Revised: Let w ( x ) be a windowing function that is positive valued, symmetric and monotonically decreasing from its center so as to provide greatest emphasis at its center.

Original: A Windowed Fourier Transform (WFT) of f ( x ) can then be deﬁned as
Revised: A Windowed Fourier Transform (WFT) of f ( x ) can then be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Making use of the symmetry constraint that we have enforced on the windowing function allows for w ( x ) = w ( − x ) so that the WFT, (7), can be rewritten as
Revised: Making use of the symmetry constraint that we have enforced on the windowing function allows for w ( x ) = w ( − x ) so that the WFT, (7), can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: which has the form of a convolution
Revised: which has the form of a convolution

Original: [Equation]
Revised: [Equation]

Original: with the inclusion of an additional phase component, e − i u
Revised: with the inclusion of an additional phase component, e − i u

Original: [Equation]
Revised: [Equation]

Original: c x .
Revised: c x .

Original: To provide additional insight into the impact the WFT convolution, (9), has on the function, f ( x ) , it is useful to examine the pointspread function, w ( x ) e i u (cid:62) c
Revised: To provide additional insight into the impact the WFT convolution, (9), has on the function, f ( x ) , it is useful to examine the pointspread function, w ( x ) e i u (cid:62) c

Original: x , in the frequency domain by taking its
Revised: x , in the frequency domain by taking its

Original: Fourier transform (6), i.e., calculate its MTF.
Revised: Fourier transform (6), i.e., calculate its MTF.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: which via grouping by coefﬁcients of x becomes
Revised: which via grouping by coefﬁcients of x becomes

Original: [Equation]
Revised: [Equation]

Original: Examination of (11) reveals that it is exactly the Fourier transform of the window function, cf.
Revised: Examination of (11) reveals that it is exactly the Fourier transform of the window function, cf.

Original: (6), as shifted to the center frequencies, u c .
Revised: (6), as shifted to the center frequencies, u c .

Original: Thus, operation of the WFT convolution, (9), on a function, f ( x ) , passes central frequency, u c , relatively unattenuated, while it suppresses those that are further away from the central frequency according to the shape of the window function, w ( x ) , i.e., it operates as a bandpass ﬁlter.
Revised: Thus, operation of the WFT convolution, (9), on a function, f ( x ) , passes central frequency, u c , relatively unattenuated, while it suppresses those that are further away from the central frequency according to the shape of the window function, w ( x ) , i.e., it operates as a bandpass ﬁlter.

Original: Thus, convolution with a bank of such ﬁlters with varying central frequencies, u c , has exactly the desired result of providing localized measures of the frequency content of the function f ( x ) .
Revised: Thus, convolution with a bank of such ﬁlters with varying central frequencies, u c , has exactly the desired result of providing localized measures of the frequency content of the function f ( x ) .

Original: Returning to the pointspread function itself, w ( x ) e i u (cid:62) c x , and recalling that e i u (cid:62) x
Revised: Returning to the pointspread function itself, w ( x ) e i u (cid:62) c x , and recalling that e i u (cid:62) x

Original: = cos( u (cid:62) x ) + i sin( u (cid:62) c x ) , it is seen that in the signal domain, x , the ﬁlter will oscillate along the direction of u c while remaining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center according to the shape of w ( x ) , i.e., we have an oriented bandpass ﬁlter.
Revised: = cos( u (cid:62) x ) + i sin( u (cid:62) c x ) , it is seen that in the signal domain, x , the ﬁlter will oscillate along the direction of u c while remaining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center according to the shape of w ( x ) , i.e., we have an oriented bandpass ﬁlter.

Original: As a speciﬁc example (Jahne & Hausbecker, 2000), taking w ( x ) to be an n-dimensional Gaussianlike function, g ( x ; σ ) = κ e −(cid:107) x (cid:107) 2 /σ 2 , with σ the standard deviation and κ a scaling factor, yields an n-dimensional Gabor-like ﬁlter,
Revised: As a speciﬁc example (Jahne & Hausbecker, 2000), taking w ( x ) to be an n-dimensional Gaussianlike function, g ( x ; σ ) = κ e −(cid:107) x (cid:107) 2 /σ 2 , with σ the standard deviation and κ a scaling factor, yields an n-dimensional Gabor-like ﬁlter,

Original: [Equation]
Revised: [Equation]

Original: which provides good joint localization of signal content in the signal and frequency domains Gabor (1946)
Revised: which provides good joint localization of signal content in the signal and frequency domains Gabor (1946)

Original: Indeed, visualization of these ﬁlters in two spatial dimensions (Figure 2) provides strikingly similar appearance to those presented in Figure 1, if in an idealized form.
Revised: Indeed, visualization of these ﬁlters in two spatial dimensions (Figure 2) provides strikingly similar appearance to those presented in Figure 1, if in an idealized form.

Original: In particular, their pointspread functions oscillate according to a frequency (cid:107)
Revised: In particular, their pointspread functions oscillate according to a frequency (cid:107)

Original: u c (cid:107) along the direction, u c (cid:107) u c (cid:107) , while remaining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center.
Revised: u c (cid:107) along the direction, u c (cid:107) u c (cid:107) , while remaining relatively constant in the orthogonal direction, even as there is an overall amplitude fall-off with distance from the center.

Original: In the frequency domain, they have peak power at u c with a fall-off following a Gaussian-like shape with standard deviation, 1 /σ , that is the inverse of that used in specifying the window, w ( x ) .
Revised: In the frequency domain, they have peak power at u c with a fall-off following a Gaussian-like shape with standard deviation, 1 /σ , that is the inverse of that used in specifying the window, w ( x ) .

Original: These observations hold because we already have seen, (11), that the frequency domain representation of such a function is the Fourier transform of the window function, w ( x ) , shifted to the center frequencies, u c ; furthermore, the Fourier tranform of a function of the form g ( x ; σ ) has a similar form, albeit with an inverse standard deviation Bracewell (1986).
Revised: These observations hold because we already have seen, (11), that the frequency domain representation of such a function is the Fourier transform of the window function, w ( x ) , shifted to the center frequencies, u c ; furthermore, the Fourier tranform of a function of the form g ( x ; σ ) has a similar form, albeit with an inverse standard deviation Bracewell (1986).

Original: 2. I MPLICATIONS FOR C ONV N ETS
Revised: 2. I MPLICATIONS FOR C ONV N ETS

Original: Convolutions in ConvNets serve to ﬁlter the input signal to highlight its features according to the learned pointspread functions (convolutional kernels).
Revised: Convolutions in ConvNets serve to ﬁlter the input signal to highlight its features according to the learned pointspread functions (convolutional kernels).

Original: Thus, convolution with the oriented ﬁlters shown in Figure 1 will serve to highlight aspects of an image that are correspondingly oriented and at corresponding scales.
Revised: Thus, convolution with the oriented ﬁlters shown in Figure 1 will serve to highlight aspects of an image that are correspondingly oriented and at corresponding scales.

Original: The question at hand is, “Why did the ConvNet learn such ﬁlters?”
Revised: The question at hand is, “Why did the ConvNet learn such ﬁlters?”

Original: The previous parts of this section have reviewed the fact that complex exponentials of the form e i u (cid:62) x
Revised: The previous parts of this section have reviewed the fact that complex exponentials of the form e i u (cid:62) x

Original: = cos( u (cid:62) x ) + i sin( u (cid:62) x ) are the eigenfunctions of convolution.
Revised: = cos( u (cid:62) x ) + i sin( u (cid:62) x ) are the eigenfunctions of convolution.

Original: Thus, such frequency dependent functions similarly serve as the eigenfunctions of the convolutional operations in ConvNets.
Revised: Thus, such frequency dependent functions similarly serve as the eigenfunctions of the convolutional operations in ConvNets.

Original: In particular, this result is a basic property of the convolutional nature of the architecture, independent of the input to the system.
Revised: In particular, this result is a basic property of the convolutional nature of the architecture, independent of the input to the system.

Original: Thus, for any convolution in a ConvNet the frequency dependent eigenfunctions, e i u (cid:62) x , provide a systematic way to represent their input.
Revised: Thus, for any convolution in a ConvNet the frequency dependent eigenfunctions, e i u (cid:62) x , provide a systematic way to represent their input.

Original: As with the general discussion of locality presented in Subsection 2.2, for the speciﬁcs of ConvNets it also is of interest to be able to characterize and operate locally on a signal.
Revised: As with the general discussion of locality presented in Subsection 2.2, for the speciﬁcs of ConvNets it also is of interest to be able to characterize and operate locally on a signal.

Original: At the level of convolution, such processing is realized via pointspread functions that operate as bandpass ﬁlters, (9).
Revised: At the level of convolution, such processing is realized via pointspread functions that operate as bandpass ﬁlters, (9).

Original: Like any practical system, ConvNets will not capture a continuous range of bandpass characteristics, as given by u c and the sampling will be limited by the number of ﬁlters the designer allows at each layer, i.e., as a metaparameter of the system.
Revised: Like any practical system, ConvNets will not capture a continuous range of bandpass characteristics, as given by u c and the sampling will be limited by the number of ﬁlters the designer allows at each layer, i.e., as a metaparameter of the system.

Original: Nevertheless, making use of these ﬁlters provides a systematic approach to representing the input signal.
Revised: Nevertheless, making use of these ﬁlters provides a systematic approach to representing the input signal.

Original: Overall, the very convolutional nature of ConvNets inherently constrain and even deﬁne the ﬁlters that they learn, independent of their input or training.
Revised: Overall, the very convolutional nature of ConvNets inherently constrain and even deﬁne the ﬁlters that they learn, independent of their input or training.

Original: In particular, learning bandpass ﬁlters provides a canonical way to represent and operate on their input, as these serve as the localized eigenfunctions of convolution.
Revised: In particular, learning bandpass ﬁlters provides a canonical way to represent and operate on their input, as these serve as the localized eigenfunctions of convolution.

Original: As a ConvNet is exposed to more and more training data, its representation is optimized by spanning as much of the data as it can.
Revised: As a ConvNet is exposed to more and more training data, its representation is optimized by spanning as much of the data as it can.

Original: Within the realm of convolution, in which
Revised: Within the realm of convolution, in which

Original: ConvNet conv layers are deﬁned, oriented bandpass ﬁlters provide the solution.
Revised: ConvNet conv layers are deﬁned, oriented bandpass ﬁlters provide the solution.

Original: They arise as the locality constrained eigenfunctions of convolution and thereby have potential to provide a span of any input signal in a localized manner.
Revised: They arise as the locality constrained eigenfunctions of convolution and thereby have potential to provide a span of any input signal in a localized manner.

Original: Thus, ConvNets are optimized by learning exactly such ﬁlters.
Revised: Thus, ConvNets are optimized by learning exactly such ﬁlters.

Original: Notably, since this explanation for why ConvNets learn oriented bandpass ﬁlters is independent of training data, it can explain why such ﬁlters emerge even when the training data lacks such pattern structure, including training on random input signals, e.g., (Linsker, 1986). 
Revised: Notably, since this explanation for why ConvNets learn oriented bandpass ﬁlters is independent of training data, it can explain why such ﬁlters emerge even when the training data lacks such pattern structure, including training on random input signals, e.g., (Linsker, 1986). Two ﬁnal implications suggest directions for future research.

Revised: First, the analysis presented in this paper has been targeted toward an explanation for why ConvNets learn oriented bandpass ﬁlters in their early layers, as suggested in visualization studies discussed in Sec. 1.2.

Revised: However, our analysis of oriented bandpass ﬁltes as the localized eigenfunctions of convolution is not speciﬁc to early

Revised: ConvNet layers, but rather applies to any convolutional layer.

Revised: The result thereby raises the question of whether deeper ConvNet layers also learn oriented bandpass ﬁlters.

Original: S UMMARY
Revised: Here, empirical data is lacking as visualization studies to date for any given layer concentrate on the combined result across all previous layers, rather than the ﬁltering characteristics at just that layer. Thus, interesting future research in examining the ﬁlters at each layer in isolation from those at other layers is motivated by the analysis presented in the current paper. Along these lines, it is interesting to note that certain hand-crafted ConvNets already make use of the same set of oriented bandpass ﬁlters at all layers of the architecture and do so to good advantage in their targeted tasks of single image and video texture analysis (Bruna & Mallat, 2013; Hadji & Wildes, 2017). A second direction for future work involves use of parameterized ﬁlters during learning.

Revised: In particular, the presented theoretical motivation for learning of oriented bandpass operators as the convolutional ﬁlters in ConvNets suggests the possibility of constraining the ﬁlters to be of that form in a learning-based framework.

Revised: In such a framework, it would not be necessary for the training process to learn the numerical values for each and every individual ﬁlter value (i.e., each ﬁlter tap), but rather would merely need to learn a much smaller number of parameters, e.g., the values of the center frequency, u c and the standard deviation σ associated with the Gabor-like ﬁlter derived above (12).

Revised: Such a constrained learning approach would require a much less intensive training procedure (e.g. involving far less data) compared to learnig values fo all individual ﬁlter taps, owing to the drastically reduced number of parameters that need to be estimated, even while being able to tune to the speciﬁcs of the task that is being optimized.

Revised:  S UMMARY

Original: Previous studies have demonstated that learned ﬁlters at the early layers of convolutional networks visualize as oriented bandpass ﬁlters.
Revised: Previous studies have demonstated that learned ﬁlters at the early layers of convolutional networks visualize as oriented bandpass ﬁlters.

Original: This phenomenon typically is explained via appeal to natural image statistics, i e., natural images are dominated by oriented contours manifest across a variety of scales and oriented bandpass ﬁlters are well matched to such structure.
Revised: This phenomenon typically is explained via appeal to natural image statistics, i.e., natural images are dominated by oriented contours manifest across a variety of scales and oriented bandpass ﬁlters are well matched to such structure.

Original: We have offered an alternative explanation in terms of the structure of convolutional networks themselves.
Revised: We have offered an alternative explanation in terms of the structure of convolutional networks themselves.

Original: Given that their convolutional layers necessarily operate within the space of convolutions, learning oriented bandpass ﬁlters provides the system with the potential to span possible input even while preserving a notion of locality in the signal domain
Revised: Given that their convolutional layers necessarily operate within the space of convolutions, learning oriented bandpass ﬁlters provides the system with the potential to span possible input, even while preserving a notion of locality in the signal domain.