Original: M
Revised: M

Original: ULTIPLICATIVE F ILTER N ETWORKS
Revised: ULTIPLICATIVE F ILTER N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although deep networks are typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as function approximators for low-dimensional-but-complex functions, such as representing images as a function of pixel coordinates, solving differential equations, or representing signed distance functions or neural radiance ﬁelds.
Revised: Although deep networks are typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as function approximators for low-dimensional-but-complex functions, such as representing images as a function of pixel coordinates, solving differential equations, or representing signed distance functions or neural radiance ﬁelds.

Original: Key to these recent successes has been the use of new elements such as sinusoidal nonlinearities or Fourier features in positional encodings, which vastly outperform simple
Revised: Key to these recent successes has been the use of new elements such as sinusoidal nonlinearities or Fourier features in positional encodings, which vastly outperform simple

Original: ReLU networks.
Revised: ReLU networks.

Original: In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative ﬁlter networks.
Revised: In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative ﬁlter networks.

Original: In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal or Gabor wavelet functions applied to the input.
Revised: In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal or Gabor wavelet functions applied to the input.

Original: This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over an exponential number of Fourier or Gabor basis functions, respectively.
Revised: This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over an exponential number of Fourier or Gabor basis functions, respectively.

Original: Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks or sinusoidal activation networks, we show that these multiplicative ﬁlter networks largely outperform or match the performance of these approaches on the domains highlighted in these past works.
Revised: Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks or sinusoidal activation networks, we show that these multiplicative ﬁlter networks largely outperform or match the performance of these approaches on the domains highlighted in these past works.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks are most commonly used to approximate functions over high-dimensional input spaces, such as functions that operate on images or long text sequences.
Revised: Neural networks are most commonly used to approximate functions over high-dimensional input spaces, such as functions that operate on images or long text sequences.

Original: However, there has been a recent growing interest in neural networks used to approximate low-dimensional-but-complex functions: for example, one could represent a continuous image as a function f : R 2 → R 3 where the input to this function speciﬁes ( x, y ) coordinates of a location in the image, and the output speciﬁes the RGB value of the pixel at that location.
Revised: However, there has been a recent growing interest in neural networks used to approximate low-dimensional-but-complex functions: for example, one could represent a continuous image as a function f : R 2 → R 3 where the input to this function speciﬁes ( x, y ) coordinates of a location in the image, and the output speciﬁes the RGB value of the pixel at that location.

Original: However, two recent papers in particular have argued that speciﬁc architectural changes are required to make (fully-connected) deep networks suitable to this task: Sitzmann et al.
Revised: However, two recent papers in particular have argued that speciﬁc architectural changes are required to make (fully-connected) deep networks suitable to this task: Sitzmann et al.

Original: (2020) employ sinusoidal activation functions within a multi-layer networks (called the SIREN architecture); and Tancik et al.
Revised: (2020) employ sinusoidal activation functions within a multi-layer networks (called the SIREN architecture); and Tancik et al.

Original: (2020) propose random Fourier features input to a traditional ReLU-based network.
Revised: (2020) propose random Fourier features input to a traditional ReLU-based network.

Original: Both papers show that the resulting networks can approximate these low-dimensional functions much better than simple feedforward ReLU networks, and achieve striking results in representing fairly complex functions (e.g. 3D signed distance ﬁelds or neural radiance ﬁelds) with a high degree of ﬁdelity.
Revised: Both papers show that the resulting networks can approximate these low-dimensional functions much better than simple feedforward ReLU networks, and achieve striking results in representing fairly complex functions (e.g. 3D signed distance ﬁelds or neural radiance ﬁelds) with a high degree of ﬁdelity.

Original: However, the precise beneﬁt of sinusoidal bases or a ﬁrst layer of Fourier features seems difﬁcult to characterize, and it remains unclear why such representations work well for these tasks.
Revised: However, the precise beneﬁt of sinusoidal bases or a ﬁrst layer of Fourier features seems difﬁcult to characterize, and it remains unclear why such representations work well for these tasks.

Original: In this paper, however, we argue and empirically demonstrate that an arguably simpler class of functions can work as well or better than these previously-proposed networks on this task.
Revised: In this paper, however, we argue and empirically demonstrate that an arguably simpler class of functions can work as well or better than these previously-proposed networks on this task.

Original: Speciﬁcally, we propose an architecture we call the multiplicative ﬁlter network (MFN).
Revised: Speciﬁcally, we propose an architecture we call the multiplicative ﬁlter network (MFN).

Original: Unlike a traditional multi-layer network that achieves representation power through compositional depth, the MFN instead simply repeatedly applies nonlinear ﬁlters (such as a sinusoid or a Gabor wavelet function) to the network’s input, then multiplies together linear functions of these features.
Revised: Unlike a traditional multi-layer network that achieves representation power through compositional depth, the MFN instead simply repeatedly applies nonlinear ﬁlters (such as a sinusoid or a Gabor wavelet function) to the network’s input, then multiplies together linear functions of these features.

Original: The notable advantage of this representation that, owing to the multiplicative properties of Fourier and Gabor ﬁlters, the entire function is ultimately just a linear function of (an exponential number of) these Fourier or Gabor features of the input.
Revised: The notable advantage of this representation that, owing to the multiplicative properties of Fourier and Gabor ﬁlters, the entire function is ultimately just a linear function of (an exponential number of) these Fourier or Gabor features of the input.

Original: Indeed, we can express the exact linear form of these MFNs, which can make their analysis considerably simpler than that for deep networks, where compositions of nonlinear activation’s make the entire function difﬁcult to characterize.
Revised: Indeed, we can express the exact linear form of these MFNs, which can make their analysis considerably simpler than that for deep networks, where compositions of nonlinear activation’s make the entire function difﬁcult to characterize.

Original: In this work, we show that despite this simplicity, the proposed networks often perform as well or better than the previously proposed SIREN or Fourier feature networks.
Revised: In this work, we show that despite this simplicity, the proposed networks often perform as well or better than the previously proposed SIREN or Fourier feature networks.

Original: Speciﬁcally, we compare our approach on networks with comparable numbers of parameters to the exact benchmarks proposed in the SIREN and Fourier features papers.
Revised: Speciﬁcally, we compare our approach on networks with comparable numbers of parameters to the exact benchmarks proposed in the SIREN and Fourier features papers.

Original: We show that MFNs achieve better performance deltas when increasing the depth or width of the networks.
Revised: We show that MFNs achieve better performance deltas when increasing the depth or width of the networks.

Original: Despite this, we do emphasize that
Revised: Despite this, we do emphasize that

Original: SIREN networks, in particular, appear to retain some notable advantages over MFNs, such as a bias towards smoother regions in the represented function and its gradients.
Revised: SIREN networks, in particular, appear to retain some notable advantages over MFNs, such as a bias towards smoother regions in the represented function and its gradients.

Original: However, especially given the fact that MFNs ultimately just correspond to a linear Fourier or Wavelet representation of a lowdimensional function, we believe they should be considered a standard benchmark for future work on such problems, to indicate where the compositional depth of typical deep networks can propose a substantial beneﬁt.
Revised: However, especially given the fact that MFNs ultimately just correspond to a linear Fourier or Wavelet representation of a lowdimensional function, we believe they should be considered a standard benchmark for future work on such problems, to indicate where the compositional depth of typical deep networks can propose a substantial beneﬁt.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Our approach is related to many previous works in Fourier and Wavelet transforms, random Fourier features, and implicit neural representations.
Revised: Our approach is related to many previous works in Fourier and Wavelet transforms, random Fourier features, and implicit neural representations.

Original: We explore the connection among the areas below.
Revised: We explore the connection among the areas below.

Original: Fourier and Wavelet transforms.
Revised: Fourier and Wavelet transforms.

Original: Transforming time or space domain signals to frequency domain using transforms such as Fourier and Wavelet transforms have been at the heart of many developments in image processing, signal processing, and computer vision.
Revised: Transforming time or space domain signals to frequency domain using transforms such as Fourier and Wavelet transforms have been at the heart of many developments in image processing, signal processing, and computer vision.

Original: In particular, the Fourier transform (Bracewell & Bracewell, 1986; Vetterli et al., 2014) and its various forms have found usage in myriad applications, such as spectroscopy, quantum mechanics, signal processing.
Revised: In particular, the Fourier transform (Bracewell & Bracewell, 1986; Vetterli et al., 2014) and its various forms have found usage in myriad applications, such as spectroscopy, quantum mechanics, signal processing.

Original: Wavelet transforms, which in particular aid in multi-scale analysis, have been found to be particularly useful in data compression, JPEG2000 (Rabbani, 2002) being one example.
Revised: Wavelet transforms, which in particular aid in multi-scale analysis, have been found to be particularly useful in data compression, JPEG2000 (Rabbani, 2002) being one example.

Original: Random Fourier features.
Revised: Random Fourier features.

Original: A seminal work by Rahimi & Recht (2008) demonstrates the power of
Revised: A seminal work by Rahimi & Recht (2008) demonstrates the power of

Original: Fourier transform in machine learning applications.
Revised: Fourier transform in machine learning applications.

Original: They show that simply projecting the original dataset into random Fourier bases vastly improves the expressiveness of models as it approximates kernel computations.
Revised: They show that simply projecting the original dataset into random Fourier bases vastly improves the expressiveness of models as it approximates kernel computations.

Original: Many subsequent works apply the Fourier features and variations (Rahimi &
Revised: Many subsequent works apply the Fourier features and variations (Rahimi &

Original: Recht, 2009; Le et al., 2013; Yu et al., 2016) to improve machine learning algorithm performance in many domain areas, including classiﬁcation (Sun et al., 2018; Rawat et al., 2019), regression (Avron et al., 2017; Brault et al., 2016), clustering (Chitta et al., 2012; Liu et al., 2019), online learning (Lin et al., 2014; Hu et al., 2015), and deep learning (Xue et al., 2019; Mehrkanoon & Suykens, 2018;
Revised: Recht, 2009; Le et al., 2013; Yu et al., 2016) to improve machine learning algorithm performance in many domain areas, including classiﬁcation (Sun et al., 2018; Rawat et al., 2019), regression (Avron et al., 2017; Brault et al., 2016), clustering (Chitta et al., 2012; Liu et al., 2019), online learning (Lin et al., 2014; Hu et al., 2015), and deep learning (Xue et al., 2019; Mehrkanoon & Suykens, 2018;

Original: Rick Chang et al., 2016; Mairal et al., 2014;
Revised: Rick Chang et al., 2016; Mairal et al., 2014;

Original: Jacot et al., 2018; Tancik et al., 2020).
Revised: Jacot et al., 2018; Tancik et al., 2020).

Original: Implicit neural representations.
Revised: Implicit neural representations.

Original: A recent line of work in representing signals as a continuous function parameterized by neural network (instead of using the traditional discrete representation) is gaining popularity.
Revised: A recent line of work in representing signals as a continuous function parameterized by neural network (instead of using the traditional discrete representation) is gaining popularity.

Original: This strategy has been used to represent different objects such as images (Nguyen et al., 2015; Stanley, 2007), shapes (Park et al., 2019; Genova et al., 2019; Chen & Zhang, 2019; Chabra et al., 2020), scenes (Mildenhall et al., 2020; Sitzmann et al., 2019; Jiang et al., 2020;
Revised: This strategy has been used to represent different objects such as images (Nguyen et al., 2015; Stanley, 2007), shapes (Park et al., 2019; Genova et al., 2019; Chen & Zhang, 2019; Chabra et al., 2020), scenes (Mildenhall et al., 2020; Sitzmann et al., 2019; Jiang et al., 2020;

Original: Niemeyer et al., 2020), and textures (Oechsle et al., 2019; Henzler et al., 2020).
Revised: Niemeyer et al., 2020), and textures (Oechsle et al., 2019; Henzler et al., 2020).

Original: In most of these applications, the standard neural networks architecture with multi-layer perceptrons and ReLU activation function is often used.
Revised: In most of these applications, the standard neural networks architecture with multi-layer perceptrons and ReLU activation function is often used.

Original: Recently, motivated by the success of Fourier transform in machine learning, a few papers have suggested architectural changes that integrate periodic nonlinearities into the network.
Revised: Recently, motivated by the success of Fourier transform in machine learning, a few papers have suggested architectural changes that integrate periodic nonlinearities into the network.

Original: Mildenhall et al.
Revised: Mildenhall et al.

Original: ; Zhong et al.
Revised: ; Zhong et al.

Original: ; Tancik et al.
Revised: ; Tancik et al.

Original: (2020) proposed the use of sinusoidal mapping of the input features (Rahimi & Recht, 2008) that uses positional encoding and
Revised: (2020) proposed the use of sinusoidal mapping of the input features (Rahimi & Recht, 2008) that uses positional encoding and

Original: Gaussian random distribution in the mapping.
Revised: Gaussian random distribution in the mapping.

Original: Others (Klocek et al., 2019; Sitzmann et al., 2020) have proposed the use of sinusoidal activation function within a multi-layer perceptron architecture.
Revised: Others (Klocek et al., 2019; Sitzmann et al., 2020) have proposed the use of sinusoidal activation function within a multi-layer perceptron architecture.

Original: Both of these strategies are demonstrated to vastly improve the results on many object representation tasks.
Revised: Both of these strategies are demonstrated to vastly improve the results on many object representation tasks.

Original:  M ULTIPLICATIVE FILTER NETWORKS
Revised:  M ULTIPLICATIVE FILTER NETWORKS

Original: A traditional k -layer deep network f : R n → R m is typically deﬁned by a recurrence such as:
Revised: A traditional k -layer deep network f : R n → R m is typically deﬁned by a recurrence such as:

Original: [Equation]
Revised: [Equation]

Original: where σ denotes a nonlinearity applied elementwise, W ( i ) ∈
Revised: where σ denotes a nonlinearity applied elementwise, W ( i ) ∈

Original: R d i  ×
Revised: R d i +1 ×

Original: d i+1 and b ( i ) ∈
Revised: d i and b ( i ) ∈

Original: R d i +1 denote the weight and bias of the i th layer, and z ( i
Revised: R d i +1 denote the weight and bias of the i th layer, and z ( i

Original: ) ∈
Revised: ) ∈

Original: R d i denotes the hidden unit at layer i .
Revised: R d i denotes the hidden unit at layer i .

Original: We refer to these networks as compositional depth networks, because each nonlinearity is applied compositionally to outputs of the previous nonlinearity in order to achieve its representational complexity.
Revised: We refer to these networks as compositional depth networks, because each nonlinearity is applied compositionally to outputs of the previous nonlinearity in order to achieve its representational complexity.

Original: The SIREN or Fourier feature networks of (Sitzmann et al., 2020) and Tancik et al.
Revised: The SIREN or Fourier feature networks of (Sitzmann et al., 2020) and Tancik et al.

Original: (2020) respectively can be viewed as simple specializations of this structure.
Revised: (2020) respectively can be viewed as simple specializations of this structure.

Original: In a SIREN network, one uses the sinusoid σ ( x ) = sin( x ) as the nonlinearity, plus proper initialization of the weights and scaling of the input.
Revised: In a SIREN network, one uses the sinusoid σ ( x ) = sin( x ) as the nonlinearity, plus proper initialization of the weights and scaling of the input.

Original: In a Fourier features network, one replaces the input layer with
Revised: In a Fourier features network, one replaces the input layer with

Original: [Equation]
Revised: [Equation]

Original: where Ω ∈ R d 12 × n is matrix of random N (0 , τ 2 ) variables ( τ being a hyperparameter of the method), but with the typical ReLU nonlinearities σ ( x )
Revised: where Ω ∈ R d 12 × n is matrix of random N (0 , τ 2 ) variables ( τ being a hyperparameter of the method), but with the typical ReLU nonlinearities σ ( x )

Original: = ReLU( x ) .
Revised: = ReLU( x ) .

Original: Our proposed multiplicative ﬁlter network , in contrast, uses a different recursion that never results in composition of nonlinear functions.
Revised: Our proposed multiplicative ﬁlter network , in contrast, uses a different recursion that never results in composition of nonlinear functions.

Original: Speciﬁcally, an MFN is deﬁned via the following recursion
Revised: Speciﬁcally, an MFN is deﬁned via the following recursion

Original: [Equation]
Revised: [Equation]

Original: where ◦ denotes elementwise multiplication, W ( i ) , b ( i ) ,
Revised: where ◦ denotes elementwise multiplication, W ( i ) , b ( i ) ,

Original: z ( i ) are all deﬁned as above, but where g : R n → R d i is parameterized by parameters θ ( i ) (the size of θ ( i ) can vary to implicitly deﬁne the output dimensions
Revised: z ( i ) are all deﬁned as above, but where g : R n → R d i is parameterized by parameters θ ( i ) (the size of θ ( i ) can vary to implicitly deﬁne the output dimensions

Original: d i ) and denotes a nonlinear ﬁlter applied to the input directly.
Revised: d i ) and denotes a nonlinear ﬁlter applied to the input directly.

Original: Of immediate importance here is that in such a network, we never apply a nonlinearity to the output of a previous nonlinearity.
Revised: Of immediate importance here is that in such a network, we never apply a nonlinearity to the output of a previous nonlinearity.

Original: All the nonlinearity of the network occurs within the g functions; layers z ( i ) , after passing through a linear function, are simply multiplied by new ﬁlters of the input.
Revised: All the nonlinearity of the network occurs within the g functions; layers z ( i ) , after passing through a linear function, are simply multiplied by new ﬁlters of the input.

Original: This results in a considerably different type of function that is currently employed by most multi-layer networks, and indeed it is largely only by convention that we refer to such a function as a “network” at all.
Revised: This results in a considerably different type of function that is currently employed by most multi-layer networks, and indeed it is largely only by convention that we refer to such a function as a “network” at all.

Original: We now present two instantiations of the MFN, using sinusoids or a Gabor wavelet as the ﬁlter g ; we call these two networks the F OURIER N ET and G ABOR N ET respectively.
Revised: We now present two instantiations of the MFN, using sinusoids or a Gabor wavelet as the ﬁlter g ; we call these two networks the F OURIER N ET and G ABOR N ET respectively.

Original: As we show, the crucial property of a function f represented by a F OURIER N ET or G ABOR N ET is that the entire function f can also be written as a linear combination of sinusoids and Gabor wavelets of the input respectively (albeit an exponentially large number of such features, but of course also with a highly reduced space of allowable coefﬁcients on this exponential number of terms, since there are only a polynomial number of parameters that deﬁne the MFN).
Revised: As we show, the crucial property of a function f represented by a F OURIER N ET or G ABOR N ET is that the entire function f can also be written as a linear combination of sinusoids and Gabor wavelets of the input respectively (albeit an exponentially large number of such features, but of course also with a highly reduced space of allowable coefﬁcients on this exponential number of terms, since there are only a polynomial number of parameters that deﬁne the MFN).

Original: Thus, we would claim that the MFN really looks more like a (rich) Fourier or Wavelet representation of the underlying signal, just one that happens to have a similar parameterization as deep networks (and which can be tuned by typical gradient descent methods).
Revised: Thus, we would claim that the MFN really looks more like a (rich) Fourier or Wavelet representation of the underlying signal, just one that happens to have a similar parameterization as deep networks (and which can be tuned by typical gradient descent methods).

Original: 3. M ULTIPLICATIVE F OURIER N ETWORKS
Revised: 3. M ULTIPLICATIVE F OURIER N ETWORKS

Original: As our ﬁrst instantiation of the MFN, we consider using a simple sinusoidal ﬁlter
Revised: As our ﬁrst instantiation of the MFN, we consider using a simple sinusoidal ﬁlter

Original: [Equation]
Revised: [Equation]

Original: (4) with parameters θ ( i ) = { ω ( i ) ∈
Revised: (4) with parameters θ ( i ) = { ω ( i ) ∈

Original: R d i × n , φ ( i )
Revised: R d i × n , φ ( i )

Original: ∈ R d i } .
Revised: ∈ R d i } .

Original: We term such a network the F OURIER N ET , as the sinusoidal activation (with arbitrary phase shifts, to represent sine or cosine functions equally) corresponds naturally to a Fourier random feature representation of the entire function.
Revised: We term such a network the F OURIER N ET , as the sinusoidal activation (with arbitrary phase shifts, to represent sine or cosine functions equally) corresponds naturally to a Fourier random feature representation of the entire function.

Original: An immediate and compelling feature of the F OURIER N ET , compared to networks based upon composition, is that its output can be directly viewed as a linear function of (an exponential number of)
Revised: An immediate and compelling feature of the F OURIER N ET , compared to networks based upon composition, is that its output can be directly viewed as a linear function of (an exponential number of)

Original: Fourier bases, with a low-rank set of coefﬁcients determined by the parameters of the network.
Revised: Fourier bases, with a low-rank set of coefﬁcients determined by the parameters of the network.

Original: This is conveyed by the following theorem.
Revised: This is conveyed by the following theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: The output of a Fourier Network is given by a linear combination of sinusoidal bases,
Revised: The output of a Fourier Network is given by a linear combination of sinusoidal bases,

Original: [Equation]
Revised: [Equation]

Original: for some coefﬁcients ¯ α 1: T , frequencies ¯ ω 1: T , phase offsets ¯ φ 1: T , and bias term ¯ b .
Revised: for some coefﬁcients ¯ α 1: T , frequencies ¯ ω 1: T , phase offsets ¯ φ 1: T , and bias term ¯ b .

Original: In other words, the F OURIER N ET represents its ﬁnal function as a linear combination of traditional
Revised: In other words, the F OURIER N ET represents its ﬁnal function as a linear combination of traditional

Original: Fourier bases, just as do “classical” random Fourier features, for instance.
Revised: Fourier bases, just as do “classical” random Fourier features, for instance.

Original: The key element of the proof (given in the appendix) is the fact that for two Fourier ﬁlters, with parameters ω, φ and τ, ψ respectively, their elementwise product can be transformed to a sum of the same type of ﬁlters
Revised: The key element of the proof (given in the appendix) is the fact that for two Fourier ﬁlters, with parameters ω, φ and τ, ψ respectively, their elementwise product can be transformed to a sum of the same type of ﬁlters

Original: [Equation]
Revised: [Equation]

Original: (note that the cosine can be expressed as a sine with a separate phase offset).
Revised: (note that the cosine can be expressed as a sine with a separate phase offset).

Original: Moreover, an inspection of the proof also lets us compute the exact coefﬁcients of the linear expansion, as a function of the network parameters.
Revised: Moreover, an inspection of the proof also lets us compute the exact coefﬁcients of the linear expansion, as a function of the network parameters.

Original: This is shown in the following corollary.
Revised: This is shown in the following corollary.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let i 1 , i 2 , . .
Revised: Let i 1 , i 2 , . .

Original: i k − 1 range over all (cid:81) k − 1 j =1 d
Revised: i k − 1 range over all (cid:81) k − 1 j =1 d

Original: j possible indices of each hidden unit of each layer of an MFN, and let s 2 , . .
Revised: j possible indices of each hidden unit of each layer of an MFN, and let s 2 , . .

Original: . , s k ∈ {− 1 , +1 } range over all 2 k − 1 possible binary signs; then the expansion of z ( k ) i k from (2) is given by all the terms
Revised: . , s k ∈ {− 1 , +1 } range over all 2 k − 1 possible binary signs; then the expansion of z ( k ) i k from (2) is given by all the terms

Original: [Equation]
Revised: [Equation]

Original: with a similar form for terms that begin at the i > 1 layer, multiplied by the corresponding b ( j ) i j term.
Revised: with a similar form for terms that begin at the i > 1 layer, multiplied by the corresponding b ( j ) i j term.

Original: This corollary follows simply by inspection of the proof in the appendix, noting that each additional multiplicative layer creates both a positive and negative combination of frequencies in the sinusoid terms, and a multiplication of the corresponding entries of W .
Revised: This corollary follows simply by inspection of the proof in the appendix, noting that each additional multiplicative layer creates both a positive and negative combination of frequencies in the sinusoid terms, and a multiplication of the corresponding entries of W .

Original: In other words, the multiplicative “depth” of the F OURIER N ET allows it to represent an exponential number of sinusoidal functions, but with the constraint that the actual number of coefﬁcients on these features is given by a “lowrank” tensor consisting mainly of the coefﬁcients in the W matrices.
Revised: In other words, the multiplicative “depth” of the F OURIER N ET allows it to represent an exponential number of sinusoidal functions, but with the constraint that the actual number of coefﬁcients on these features is given by a “lowrank” tensor consisting mainly of the coefﬁcients in the W matrices.

Original: This expansion also suggests a method for initializing parameters speciﬁc to this network in a manner that scales appropriately with the network size.
Revised: This expansion also suggests a method for initializing parameters speciﬁc to this network in a manner that scales appropriately with the network size.

Original: Speciﬁcally, however Ω ( i ) s are initialized (typically random uniform or Gaussian, though here with an additional scaling factor that depends on the relative scale of the input), one should divide these terms by √ k , to ensure that the variance of the ﬁnal frequency ω t is independent of the number of layers.
Revised: Speciﬁcally, however W ( i ) s are initialized (typically random uniform or Gaussian, though here with an additional scaling factor that depends on the relative scale of the input), one should divide these terms by √ k , to ensure that the variance of the ﬁnal frequency ω t is independent of the number of layers.

Original: 3. M ULTIPLICATIVE G ABOR N ETWORKS
Revised: 3. M ULTIPLICATIVE G ABOR N ETWORKS

Original: A well-known deﬁciency of the pure Fourier bases is that they have global support, and thus may have difﬁculty representing more local features.
Revised: A well-known deﬁciency of the pure Fourier bases is that they have global support, and thus may have difﬁculty representing more local features.

Original: A common alternative to these bases is the use of
Revised: A common alternative to these bases is the use of

Original: Gabor ﬁlter to capture both a frequency and spatial locality component.
Revised: Gabor ﬁlter to capture both a frequency and spatial locality component.

Original: Speciﬁcally, we consider a
Revised: Speciﬁcally, we consider a

Original: Gabor ﬁlter of the form
Revised: Gabor ﬁlter of the form

Original: [Equation]
Revised: [Equation]

Original: with parameters θ
Revised: with parameters θ

Original: [Equation]
Revised: [Equation]

Revised: i

Original:  i ) 1: d i ∈ R n , φ ( i ) 1: d i ∈ R (cid:111) (here µ ( i ) j denotes the mean of the j th Gabor ﬁlter and γ ( i )
Revised: [Equation] = [Equation] i ) 1: d i ∈ R n , φ ( i ) 1: d i ∈ R (cid:111) (here µ ( i ) j denotes the mean of the j th Gabor ﬁlter and γ ( i )

Original: j denotes the scale term), and where for simplicity we specify the functional form of each j = 1 . . .
Revised: j denotes the scale term), and where for simplicity we specify the functional form of each j = 1 . . .

Original: , d i coordinates of the function g : R n → R d i .
Revised: , d i coordinates of the function g : R n → R d i .

Original: We call the
Revised: We call the

Original: MFN using this ﬁlter the GaborNet .
Revised: MFN using this ﬁlter the GaborNet .

Original: As with the FourierNet, a compelling feature of the Gabor network is that the ﬁnal function f can be represented as a linear combination of Gabor ﬁlters.
Revised: As with the FourierNet, a compelling feature of the Gabor network is that the ﬁnal function f can be represented as a linear combination of Gabor ﬁlters.

Original: This is captured by the following theorem:
Revised: This is captured by the following theorem:

Original: Theorem 2.
Revised: Theorem 2.

Original: The output of a Gabor Network is given by a linear combination of sinusoidal bases,
Revised: The output of a Gabor Network is given by a linear combination of Gabor bases,

Original: [Equation]
Revised: [Equation]

Original: The proof is given in the appendix, but the basic procedure is the same as the above: using the fact that just like Fourier ﬁlters, the product of Gabor ﬁlters is also a linear combination of (a different set of) Gabor ﬁlters.
Revised: The proof is given in the appendix, but the basic procedure is the same as the above: using the fact that just like Fourier ﬁlters, the product of Gabor ﬁlters is also a linear combination of (a different set of) Gabor ﬁlters.

Original: Likewise, we can also compute the explicit form of the coefﬁcients and for this linear basis expansion, with the explicit form again given in the appendix.
Revised: Likewise, we can also compute the explicit form of the coefﬁcients and for this linear basis expansion, with the explicit form again given in the appendix.

Original: One relevant point, though, is how we choose initializations for the γ and µ parameters.
Revised: One relevant point, though, is how we choose initializations for the γ and µ parameters.

Original: Since γ effectively acts as an inverse covariance term of a Gaussian, a Gamma( α, β ) random variable (the conjugate prior of the
Revised: Since γ effectively acts as an inverse covariance term of a Gaussian, a Gamma( α, β ) random variable (the conjugate prior of the

Original: Gaussian inverse covariance), is a reasonable choice for this parameter.
Revised: Gaussian inverse covariance), is a reasonable choice for this parameter.

Original: And since the ¯ γ functions in the ﬁnal linear expansion end up being a sum of the individual random γ ( i ) terms at each layer, we scale each layer’s α term by 1 /k to effectively control this parameter at the ﬁnal layer.
Revised: And since the ¯ γ functions in the ﬁnal linear expansion end up being a sum of the individual random γ ( i ) terms at each layer, we scale each layer’s α term by 1 /k to effectively control this parameter at the ﬁnal layer.

Original: We also simply choose each µ ( i ) to be uniformly distributed over the range of the allowable input space x .
Revised: We also simply choose each µ ( i ) to be uniformly distributed over the range of the allowable input space x .

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We test MFNs on a broad range of representation tasks, showing that the relative simplicity of MFNs improves upon the performance of existing neural representation methods.
Revised: We test MFNs on a broad range of representation tasks, showing that the relative simplicity of MFNs improves upon the performance of existing neural representation methods.

Original: Our set of experiments draws from those presented in Sitzmann et al.
Revised: Our set of experiments draws from those presented in Sitzmann et al.

Original: (2020) alongside SIREN (image representation, shape representation, and differential equation experiments) and in Tancik et al.
Revised: (2020) alongside SIREN (image representation, shape representation, and differential equation experiments) and in Tancik et al.

Original: (2020) alongside Fourier feature networks with Gaussian random features, which we call FF Gaussian (image generalization and 3D inverse rendering experiments).
Revised: (2020) alongside Fourier feature networks with Gaussian random features, which we call FF Gaussian (image generalization and 3D inverse rendering experiments).

Original: In each case, we compare against the set of models tested in the original experiment (generally, either SIREN or FF Gaussian, along with a basic ReLU MLP).
Revised: In each case, we compare against the set of models tested in the original experiment (generally, either SIREN or FF Gaussian, along with a basic ReLU MLP).

Original: Full details on hyperparameters and training speciﬁcations are available in the appendix.
Revised: Full details on hyperparameters and training speciﬁcations are available in the appendix.

Original: 4. I MAGE R EPRESENTATION & G ENERALIZATION
Revised: 4. I MAGE R EPRESENTATION & G ENERALIZATION

Original: We ﬁrst examine the ability of several networks architectures in the task of image representation as described in Section 1, where we ﬁt the network to a function f : R 2 → R c using a dataset where input coordinates ( x, y ) corresponding to the output pixel value at those coordinates (with c = 1 or 3 for grayscale or RGB images, respectively).
Revised: We ﬁrst examine the ability of several networks architectures in the task of image representation as described in Section 1, where we ﬁt the network to a function f : R 2 → R c using a dataset where input coordinates ( x, y ) corresponding to the output pixel value at those coordinates (with c = 1 or 3 for grayscale or RGB images, respectively).

Original: To demonstrate this, we construct such a dataset from a 256 × 256 pixel grayscale image and ﬁt various models using a simple mean squared error (MSE) loss, including SIREN, G ABOR N ET , F OURIER N ET , and ReLU MLPs with and without positional encoding (PE).
Revised: To demonstrate this, we construct such a dataset from a 256 × 256 pixel grayscale image and ﬁt various models using a simple mean squared error (MSE) loss, including SIREN, G ABOR N ET , F OURIER N ET , and ReLU MLPs with and without positional encoding (PE).

Original: Visual results and a plot of PSNR early in training are shown in (Figure 1).
Revised: Visual results and a plot of PSNR early in training are shown in (Figure 1).

Original: In particular, both F OURIER N ET and G ABOR N ET show quicker initial convergence than other architectures.
Revised: In particular, both F OURIER N ET and G ABOR N ET show quicker initial convergence than other architectures.

Original: PSNRs after training (Table 1) show that SIREN eventually outperforms
Revised: PSNRs after training (Table 1) show that SIREN eventually outperforms

Original: F OURIER N ET , while G ABOR N ET remains the best model throughout the training.
Revised: F OURIER N ET , while G ABOR N ET remains the best model throughout the training.

Original: Indeed, after only 1000 training iterations, G ABOR N ET performs reconstruction better than all other models trained for 10 times longer.
Revised: Indeed, after only 1000 training iterations, G ABOR N ET performs reconstruction better than all other models trained for 10 times longer.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We can broaden the task above to represent video by appending a third dimension to the input: the output corresponding to the input ( x, y, t ) is the pixel value at ( x, y ) at frame t .
Revised: We can broaden the task above to represent video by appending a third dimension to the input: the output corresponding to the input ( x, y, t ) is the pixel value at ( x, y ) at frame t .

Original: We aim to represent a 300 frame color video with 512 × 512 resolution in this manner, testing all of the architectures used in the previous experiment.
Revised: We aim to represent a 300 frame color video with 512 × 512 resolution in this manner, testing all of the architectures used in the previous experiment.

Original: As shown in Figure 1, SIREN and G ABOR N ET are most capable of reproducing ﬁne details of the original video, such as whiskers and and eye color.
Revised: As shown in Figure 1, SIREN and G ABOR N ET are most capable of reproducing ﬁne details of the original video, such as whiskers and and eye color.

Original: This is reﬂected in the PSNR of these reconstructions (Table 1); SIREN performs best with a PSNR over 30 dB, while the G ABOR N ET reconstruction comes withindB of SIREN and exhibits much lower variation across frames than any other model.
Revised: This is reﬂected in the PSNR of these reconstructions (Table 1); SIREN performs best with a PSNR over 30 dB, while the G ABOR N ET reconstruction comes withindB of SIREN and exhibits much lower variation across frames than any other model.

Original: [Table]
Revised: [Table]

Original: In addition to representing images, we demonstrate that the MFNs are able to generalize the representation to unseen pixels.
Revised: In addition to representing images, we demonstrate that the MFNs are able to generalize the representation to unseen pixels.

Original: We train the networks using only 25% of the image pixels (every other pixel in the width and height dimensions) and evaluate using the complete images.
Revised: We train the networks using only 25% of the image pixels (every other pixel in the width and height dimensions) and evaluate using the complete images.

Original: We compare the results of our methods with the
Revised: We compare the results of our methods with the

Original: Fourier feature networks on two datasets (natural and text images) presented in Tancik et al.
Revised: Fourier feature networks on two datasets (natural and text images) presented in Tancik et al.

Original: The peak signal-to-noise ratio (PSNR) metric is used to evaluate the performance.
Revised: The peak signal-to-noise ratio (PSNR) metric is used to evaluate the performance.

Original: As we can see from Table 2, both F OURIER N ET and G ABORN ET outperform all versions of the Fourier feature networks that uses basic, positional encoding, and random Gaussian features.
Revised: As we can see from Table 2, both F OURIER N ET and G ABORN ET outperform all versions of the Fourier feature networks that uses basic, positional encoding, and random Gaussian features.

Original: Some examples of the generated images are presented in Figure 2, with additional images in Appendix B.2.
Revised: Some examples of the generated images are presented in Figure 2, with additional images in Appendix B.2.

Original: Visually, the MFNs’ generated images also have better quality over the baselines, particularly in Text datasets.
Revised: Visually, the MFNs’ generated images also have better quality over the baselines, particularly in Text datasets.

Original: Some parts of the text are missing in the baselines images (highlighted with red rectangles in Figure 2), whereas the MFNs completely generate all parts of the text in the images.
Revised: Some parts of the text are missing in the baselines images (highlighted with red rectangles in Figure 2), whereas the MFNs completely generate all parts of the text in the images.

Original: [Figure]
Revised: [Figure]

Original: 4. D IFFERENTIAL EQUATIONS
Revised: 4. D IFFERENTIAL EQUATIONS

Original: In this section, we aim to solve boundary value problems which are supervised by different forms of gradient information from the functional at hand.
Revised: In this section, we aim to solve boundary value problems which are supervised by different forms of gradient information from the functional at hand.

Original: We ﬁrst focus on the Poisson equation, where we demonstrate image reconstruction in two settings where the supervision for the model is brought about by gradients and Laplacians respectively.
Revised: We ﬁrst focus on the Poisson equation, where we demonstrate image reconstruction in two settings where the supervision for the model is brought about by gradients and Laplacians respectively.

Original: It is worth noting that the model is never presented with real function values.
Revised: It is worth noting that the model is never presented with real function values.

Original: We then focus on two 2 nd order differential equations, namely, the
Revised: We then focus on two 2 nd order differential equations, namely, the

Original: Helmholtz equation and the wave equation, where we solve for the wave ﬁeld, where the network is supervised by a known source function.
Revised: Helmholtz equation and the wave equation, where we solve for the wave ﬁeld, where the network is supervised by a known source function.

Original: We demonstrate image reconstruction using gradients and compare the performance of F OURIERN ET and G ABOR N ET with SIREN and ReLU MLP for the Poisson equation.
Revised: We demonstrate image reconstruction using gradients and compare the performance of F OURIERN ET and G ABOR N ET with SIREN and ReLU MLP for the Poisson equation.

Original: We use the same loss function as in (Sitzmann et al., 2020) in (5).
Revised: We use the same loss function as in (Sitzmann et al., 2020) in (5).

Original: Figures 3a and 3b show that the image is reconstructed successfully when the networks are trained while being supervised by gradients and Laplacians respectively, while ReLU fails spectacularly.
Revised: Figures 3a and 3b show that the image is reconstructed successfully when the networks are trained while being supervised by gradients and Laplacians respectively, while ReLU fails spectacularly.

Original: Table 3 depicts the losses of each method afteriterations, where it can be seen that G ABOR N ET beats other baselines in terms of performance.
Revised: Table 3 depicts the losses of each method afteriterations, where it can be seen that G ABOR N ET beats other baselines in terms of performance.

Original: [Figure]
Revised: [Figure]

Original: The Helmholtz and wave equations are related to the physical modeling of diffusion and waves and are closely related to a Fourier transform.
Revised: The Helmholtz and wave equations are related to the physical modeling of diffusion and waves and are closely related to a Fourier transform.

Original: Hence, we focus our attention on describing the Helmholtz equation.
Revised: Hence, we focus our attention on describing the Helmholtz equation.

Original: We aim to solve for the wave ﬁeld and compare the performance of F OURIER N ET and G ABOR N ET with SIREN and ReLU MLP.
Revised: We aim to solve for the wave ﬁeld and compare the performance of F OURIER N ET and G ABOR N ET with SIREN and ReLU MLP.

Original: To accommodate for complex-valued solutions, the network is conﬁgured to output two values which can be interpreted as the real and imaginary parts.
Revised: To accommodate for complex-valued solutions, the network is conﬁgured to output two values which can be interpreted as the real and imaginary parts.

Original: We use the same loss function as used in (Sitzmann et al., 2020)
Revised: We use the same loss function as used in (Sitzmann et al., 2020)

Original: (see Section 4.
Revised: (see Section 4.

Original: of (Sitzmann et al., 2020) for details).
Revised: of (Sitzmann et al., 2020) for details).

Original: Figure 4 shows the magnitude and the phase of the reconstructed wave front for a single Gaussian source placed at the center of a medium with uniform wave propagation velocity.
Revised: Figure 4 shows the magnitude and the phase of the reconstructed wave front for a single Gaussian source placed at the center of a medium with uniform wave propagation velocity.

Original: Table 3 depicts the losses of each method after 50000 iterations, where it can be seen that G ABOR N ET beats F OURIER N ET and SIREN in terms of performance, while ReLU MLP fails miserably.
Revised: Table 3 depicts the losses of each method after 50000 iterations, and shows that G ABOR N ET beats F OURIER N ET and SIREN in terms of performance, while, as previously shown in (Sitzmann et al., 2020), ReLU MLP fails miserably.

Original: Details pertaining to the network and training are relegated to Appendix B.3.
Revised: Details pertaining to the network and training are relegated to Appendix B.3.

Original: 4. S HAPE R EPRESENTATION VIA SIGNED DISTANCE FIELDS
Revised: 4. S HAPE R EPRESENTATION VIA SIGNED DISTANCE FIELDS

Original: Recent work (Park et al., 2019) has explored the problem of 3D shape representation with neural architectures, often with surprisingly effective results.
Revised: Recent work (Park et al., 2019) has explored the problem of 3D shape representation with neural architectures, often with surprisingly effective results.

Original: This is done by training on raw geometric data: given an oriented point cloud, we seek to learn a function f : R 3 → R that takes points as input such that the zero level set { x | f ( x ) = 0 } of the network accurately represents the surfaces of the shape.
Revised: This is done by training on raw geometric data: given an oriented point cloud, we seek to learn a function f : R 3 → R that takes points as input such that the zero level set { x | f ( x ) = 0 } of the network accurately represents the surfaces of the shape.

Original: Effective training objectives for this task has been explored in considerable depth; we use the training loss presented by Park et al.
Revised: Effective training objectives for this task has been explored in considerable depth; we use the training loss presented by Park et al.

Original: and used in Sitzmann et al.
Revised: and used in Sitzmann et al.

Original: (2020), which includes terms involving the network’s output (penalties to encourage SDF values near and away from 0 for surface and off-surface points, respectively) as well as its gradients (a term encouraging the gradient to match the surface points’ normals, and a gradient norm penalty throughout the entire 3D space).
Revised: (2020), which includes terms involving the network’s output (penalties to encourage SDF values near and away from 0 for surface and off-surface points, respectively) as well as its gradients (a term encouraging the gradient to match the surface points’ normals, and a gradient norm penalty throughout the entire 3D space).

Original: A complete description of the loss function is left to the appendix.
Revised: A complete description of the loss function is left to the appendix.

Original: Figure 5a shows the results of our shape representation task on SIREN, a standard ReLU MLP, and both variants of MFN.
Revised: Figure 5a shows the results of our shape representation task on SIREN, a standard ReLU MLP, and both variants of MFN.

Original: As can be seen, the ReLU network’s fails to represent some features of the scene entirely, such as doorways, picture frames, and pillows.
Revised: As can be seen, the ReLU network’s fails to represent some features of the scene entirely, such as doorways, picture frames, and pillows.

Original: Both MFN architectures far outperform this baseline, and are able to reconstruct the room and objects within it to a recognizable degree.
Revised: Both MFN architectures far outperform this baseline, and are able to reconstruct the room and objects within it to a recognizable degree.

Original: However, likely owing to its strong ability to produce smooth outputs and gradients, SIREN is largely able to avoid the visual artifacts on ﬂat surfaces like walls that remain in reconstructions by F OURIER N ET and, to a lesser extent, G ABOR N ET .
Revised: However, likely owing to its strong ability to produce smooth outputs and gradients, SIREN is largely able to avoid the visual artifacts on ﬂat surfaces like walls that remain in reconstructions by F OURIER N ET and, to a lesser extent, G ABOR N ET .

Original: 4. 3D I NVERSE R ENDERING FOR V IEW S YNTHESIS
Revised: 4. 3D I NVERSE R ENDERING FOR V IEW S YNTHESIS

Original: [Table]
Revised: [Table]

Original: In this view synthesis task, we aim to reconstruct 3D representation from the observed 2D photographs.
Revised: In this view synthesis task, we aim to reconstruct 3D representation from the observed 2D photographs.

Original: Using the reconstructed 3D representation, we then render 2D images from new viewpoints.
Revised: Using the reconstructed 3D representation, we then render 2D images from new viewpoints.

Original: We use the “simpliﬁed NeRF” task on Lego dataset presented in Tancik et al.
Revised: We use the “simpliﬁed Neural Radiance Fields (NeRF)” task on Lego dataset presented in Tancik et al.

Original: The networks are trained to predict the color (in RGB format) and the volume density at a given 3D location of the viewpoint.
Revised: The networks are trained to predict the color (in RGB format) and the volume density at a given 3D location of the viewpoint.

Original: Volumetric rendering is then used to re-render the 2D image photograph at the viewpoint.
Revised: Volumetric rendering is then used to re-render the 2D image photograph at the viewpoint.

Original: The training loss is computed as the mean squared error between the rendered photograph and the actual 2D image observations.
Revised: The training loss is computed as the mean squared error between the rendered photograph and the actual 2D image observations.

Original: The results in Table 4 show that both MFNs perform competitively in the task.
Revised: The results in Table 4 show that both MFNs perform competitively in the task.

Original: G ABOR N ET has a slight advantage over the baselines and F OURIER N ET in terms of the overall PSNR metric.
Revised: G ABOR N ET has a slight advantage over the baselines and F OURIER N ET in terms of the overall PSNR metric.

Original: It also arguably produces a slightly better image rendering quality as shown in Figure 5b and other images in Appendix B.5.
Revised: It also arguably produces a slightly better image rendering quality as shown in Figure 5b and other images in Appendix B.5.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have introduced multiplicative ﬁlter networks (MFNs), a class of neural representation architectures that forego the usual compositional notion of network depth in favor of a similarly expressive multiplicative operation.
Revised: We have introduced multiplicative ﬁlter networks (MFNs), a class of neural representation architectures that forego the usual compositional notion of network depth in favor of a similarly expressive multiplicative operation.

Original: They also admit a natural signal processing interpretation, as in the two instantiations of MFNs, F OURIER N ET and G ABOR N ET , which are proven to be exactly equivalent to a linear combination of sinusoidal or Gabor wavelet bases, respectively.
Revised: They also admit a natural signal processing interpretation, as in the two instantiations of MFNs, F OURIER N ET and G ABOR N ET , which are proven to be exactly equivalent to a linear combination of sinusoidal or Gabor wavelet bases, respectively.

Original: In experiments, we show that, despite their simplicity relative to other deep architectures designed for implicit representation,
Revised: In experiments, we show that, despite their simplicity relative to other deep architectures designed for implicit representation,

Original: MFNs stand up to or surpass the previous state of the art on a battery of representation tasks.
Revised: MFNs stand up to or surpass the previous state of the art on a battery of representation tasks.