Original: M ULTIPLICATIVE F ILTER N ETWORKS
Revised: M ULTIPLICATIVE F ILTER N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although deep networks are typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as function approximators for low-dimensional-but-complex functions, such as representing images as a function of pixel coordinates, solving differential equations, or representing signed distance functions or neural radiance ﬁelds.
Revised: Although deep networks are typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as function approximators for low-dimensional-but-complex functions, such as representing images as a function of pixel coordinates, solving differential equations, or representing signed distance functions or neural radiance ﬁelds.

Original: Key to these recent successes has been the use of new elements such as sinusoidal nonlinearities or Fourier features in positional encodings, which vastly outperform simple
Revised: Key to these recent successes has been the use of new elements such as sinusoidal nonlinearities or Fourier features in positional encodings, which vastly outperform simple

Original: ReLU networks.
Revised: ReLU networks.

Original: In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative ﬁlter networks.
Revised: In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative ﬁlter networks.

Original: In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal or Gabor wavelet functions applied to the input.
Revised: In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal or Gabor wavelet functions applied to the input.

Original: This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over an exponential number of Fourier or Gabor basis functions, respectively.
Revised: This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over an exponential number of Fourier or Gabor basis functions, respectively.

Original: Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks or sinusoidal activation networks, we show that these multiplicative ﬁlter networks largely outperform or match the performance of these approaches on the domains highlighted in these past works.
Revised: Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks or sinusoidal activation networks, we show that these multiplicative ﬁlter networks largely outperform or match the performance of these approaches on the domains highlighted in these past works.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks are most commonly used to approximate functions over high-dimensional input spaces, such as functions that operate on images or long text sequences.
Revised: Neural networks are most commonly used to approximate functions over high-dimensional input spaces, such as functions that operate on images or long text sequences.

Original: However, there has been a recent growing interest in neural networks used to approximate low-dimensional-but-complex functions: for example, one could represent a continuous image as a function f : R 2 → R 3 where the input to this function speciﬁes ( x, y ) coordinates of a location in the image, and the output speciﬁes the RGB value of the pixel at that location.
Revised: However, there has been a recent growing interest in neural networks used to approximate low-dimensional-but-complex functions: for example, one could represent a continuous image as a function f : R 2 → R 3 where the input to this function speciﬁes ( x, y ) coordinates of a location in the image, and the output speciﬁes the RGB value of the pixel at that location.

Original: However, two recent papers in particular have argued that speciﬁc architectural changes are required to make (fully-connected) deep networks suitable to this task: Sitzmann et al.
Revised: However, two recent papers in particular have argued that speciﬁc architectural changes are required to make (fully-connected) deep networks suitable to this task: Sitzmann et al.

Original: (2020) employ sinusoidal activation functions within a multi-layer networks (called the SIREN architecture); and Tancik et al.
Revised: (2020) employ sinusoidal activation functions within a multi-layer networks (called the SIREN architecture); and Tancik et al.

Original: (2020) propose random Fourier features input to a traditional ReLU-based network.
Revised: (2020) propose random Fourier features input to a traditional ReLU-based network.

Original: Both papers show that the resulting networks can approximate these low-dimensional functions much better than simple feedforward ReLU networks, and achieve striking results in representing fairly complex functions (e.g. 3D signed distance ﬁelds or neural radiance ﬁelds) with a high degree of ﬁdelity.
Revised: Both papers show that the resulting networks can approximate these low-dimensional functions much better than simple feedforward ReLU networks, and achieve striking results in representing fairly complex functions (e.g. 3D signed distance ﬁelds or neural radiance ﬁelds) with a high degree of ﬁdelity.

Original: However, the precise beneﬁt of sinusoidal bases or a ﬁrst layer of Fourier features seems difﬁcult to characterize, and it remains unclear why such representations work well for these tasks.
Revised: However, the precise beneﬁt of sinusoidal bases or a ﬁrst layer of Fourier features seems difﬁcult to characterize, and it remains unclear why such representations work well for these tasks.

Original: In this paper, however, we argue and empirically demonstrate that an arguably simpler class of functions can work as well or better than these previously-proposed networks on this task.
Revised: In this paper, however, we argue and empirically demonstrate that an arguably simpler class of functions can work as well or better than these previously-proposed networks on this task.

Original: Speciﬁcally, we propose an architecture we call the multiplicative ﬁlter network (MFN).
Revised: Speciﬁcally, we propose an architecture we call the multiplicative ﬁlter network (MFN).

Original: Unlike a traditional multi-layer network that achieves representation power through compositional depth, the MFN instead simply repeatedly applies nonlinear ﬁlters (such as a sinusoid or a Gabor wavelet function) to the network’s input, then multiplies together linear functions of these features.
Revised: Unlike a traditional multi-layer network that achieves representation power through compositional depth, the MFN instead simply repeatedly applies nonlinear ﬁlters (such as a sinusoid or a Gabor wavelet function) to the network’s input, then multiplies together linear functions of these features.

Original: The notable advantage of this representation that, owing to the multiplicative properties of Fourier and Gabor ﬁlters, the entire function is ultimately just a linear function of (an exponential number of) these Fourier or Gabor features of the input.
Revised: The notable advantage of this representation that, owing to the multiplicative properties of Fourier and Gabor ﬁlters, the entire function is ultimately just a linear function of (an exponential number of) these Fourier or Gabor features of the input.

Original: Indeed, we can express the exact linear form of these MFNs, which can make their analysis considerably simpler than that for deep networks, where compositions of nonlinear activation’s make the entire function difﬁcult to characterize.
Revised: Indeed, we can express the exact linear form of these MFNs, which can make their analysis considerably simpler than that for deep networks, where compositions of nonlinear activation’s make the entire function difﬁcult to characterize.

Original: In this work, we show that despite this simplicity, the proposed networks often perform as well or better than the previously proposed SIREN or Fourier feature networks.
Revised: In this work, we show that despite this simplicity, the proposed networks often perform as well or better than the previously proposed SIREN or Fourier feature networks.

Original: Speciﬁcally, we compare our approach on networks with comparable numbers of parameters to the exact benchmarks proposed in the SIREN and Fourier features papers.
Revised: Speciﬁcally, we compare our approach on networks with comparable numbers of parameters to the exact benchmarks proposed in the SIREN and Fourier features papers.

Original: We show that MFNs achieve better performance deltas when increasing the depth or width of the networks.
Revised: We show that MFNs achieve better performance deltas when increasing the depth or width of the networks.

Original: Despite this, we do emphasize that
Revised: Despite this, we do emphasize that

Original: SIREN networks, in particular, appear to retain some notable advantages over MFNs, such as a bias towards smoother regions in the represented function and its gradients.
Revised: SIREN networks, in particular, appear to retain some notable advantages over MFNs, such as a bias towards smoother regions in the represented function and its gradients.

Original: However, especially given the fact that MFNs ultimately just correspond to a linear Fourier or Wavelet representation of a lowdimensional function, we believe they should be considered a standard benchmark for future work on such problems, to indicate where the compositional depth of typical deep networks can propose a substantial beneﬁt.
Revised: However, especially given the fact that MFNs ultimately just correspond to a linear Fourier or Wavelet representation of a lowdimensional function, we believe they should be considered a standard benchmark for future work on such problems, to indicate where the compositional depth of typical deep networks can propose a substantial beneﬁt.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Our approach is related to many previous works in Fourier and Wavelet transforms, random Fourier features, and implicit neural representations.
Revised: Our approach is related to many previous works in Fourier and Wavelet transforms, random Fourier features, and implicit neural representations.

Original: We explore the connection among the areas below.
Revised: We explore the connection among the areas below.

Original: Fourier and Wavelet transforms.
Revised: Fourier and Wavelet transforms.

Original: Transforming time or space domain signals to frequency domain using transforms such as Fourier and Wavelet transforms have been at the heart of many developments in image processing, signal processing, and computer vision.
Revised: Transforming time or space domain signals to frequency domain using transforms such as Fourier and Wavelet transforms have been at the heart of many developments in image processing, signal processing, and computer vision.

Original: In particular, the Fourier transform (Bracewell & Bracewell, 1986; Vetterli et al., 2014) and its various forms have found usage in myriad applications, such as spectroscopy, quantum mechanics, signal processing.
Revised: In particular, the Fourier transform (Bracewell & Bracewell, 1986; Vetterli et al., 2014) and its various forms have found usage in myriad applications, such as spectroscopy, quantum mechanics, signal processing.

Original: Wavelet transforms, which in particular aid in multi-scale analysis, have been found to be particularly useful in data compression, JPEG2000 (Rabbani, 2002) being one example.
Revised: Wavelet transforms, which in particular aid in multi-scale analysis, have been found to be particularly useful in data compression, JPEG2000 (Rabbani, 2002) being one example.

Original: Random Fourier features.
Revised: Random Fourier features.

Original: A seminal work by Rahimi & Recht (2008) demonstrates the power of
Revised: A seminal work by Rahimi & Recht (2008) demonstrates the power of

Original: Fourier transform in machine learning applications.
Revised: Fourier transform in machine learning applications.

Original: They show that simply projecting the original dataset into random Fourier bases vastly improves the expressiveness of models as it approximates kernel computations.
Revised: They show that simply projecting the original dataset into random Fourier bases vastly improves the expressiveness of models as it approximates kernel computations.

Original: Many subsequent works apply the Fourier features and variations (Rahimi &
Revised: Many subsequent works apply the Fourier features and variations (Rahimi &

Original: Recht, 2009; Le et al., 2013; Yu et al., 2016) to improve machine learning algorithm performance in many domain areas, including classiﬁcation (Sun et al., 2018; Rawat et al., 2019), regression (Avron et al., 2017; Brault et al., 2016), clustering (Chitta et al., 2012; Liu et al., 2019), online learning (Lin et al., 2014; Hu et al., 2015), and deep learning (Xue et al., 2019; Mehrkanoon & Suykens, 2018;
Revised: Recht, 2009; Le et al., 2013; Yu et al., 2016) to improve machine learning algorithm performance in many domain areas, including classiﬁcation (Sun et al., 2018; Rawat et al., 2019), regression (Avron et al., 2017; Brault et al., 2016), clustering (Chitta et al., 2012; Liu et al., 2019), online learning (Lin et al., 2014; Hu et al., 2015), and deep learning (Xue et al., 2019; Mehrkanoon & Suykens, 2018;

Original: Rick Chang et al., 2016; Mairal et al., 2014;
Revised: Rick Chang et al., 2016; Mairal et al., 2014;

Original: Jacot et al., 2018; Tancik et al., 2020).
Revised: Jacot et al., 2018; Tancik et al., 2020).

Original: Implicit neural representations.
Revised: Implicit neural representations.

Original: A recent line of work in representing signals as a continuous function parameterized by neural network (instead of using the traditional discrete representation) is gaining popularity.
Revised: A recent line of work in representing signals as a continuous function parameterized by neural network (instead of using the traditional discrete representation) is gaining popularity.

Original: This strategy has been used to represent different objects such as images (Nguyen et al., 2015; Stanley, 2007), shapes (Park et al., 2019; Genova et al., 2019; Chen & Zhang, 2019; Chabra et al., 2020), scenes (Mildenhall et al., 2020; Sitzmann et al., 2019; Jiang et al., 2020;
Revised: This strategy has been used to represent different objects such as images (Nguyen et al., 2015; Stanley, 2007), shapes (Park et al., 2019; Genova et al., 2019; Chen & Zhang, 2019; Chabra et al., 2020), scenes (Mildenhall et al., 2020; Sitzmann et al., 2019; Jiang et al., 2020;

Original: Niemeyer et al., 2020), and textures (Oechsle et al., 2019; Henzler et al., 2020).
Revised: Niemeyer et al., 2020), and textures (Oechsle et al., 2019; Henzler et al., 2020).

Original: In most of these applications, the standard neural networks architecture with multi-layer perceptrons and ReLU activation function is often used.
Revised: In most of these applications, the standard neural networks architecture with multi-layer perceptrons and ReLU activation function is often used.

Original: Recently, motivated by the success of Fourier transform in machine learning, a few papers have suggested architectural changes that integrate periodic nonlinearities into the network.
Revised: Recently, motivated by the success of Fourier transform in machine learning, a few papers have suggested architectural changes that integrate periodic nonlinearities into the network.

Original: Mildenhall et al.
Revised: Mildenhall et al.

Original: ; Zhong et al.
Revised: ; Zhong et al.

Original: ; Tancik et al.
Revised: ; Tancik et al.

Original: (2020) proposed the use of sinusoidal mapping of the input features (Rahimi & Recht, 2008) that uses positional encoding and
Revised: (2020) proposed the use of sinusoidal mapping of the input features (Rahimi & Recht, 2008) that uses positional encoding and

Original: Gaussian random distribution in the mapping.
Revised: Gaussian random distribution in the mapping.

Original: Others (Klocek et al., 2019; Sitzmann et al., 2020) have proposed the use of sinusoidal activation function within a multi-layer perceptron architecture.
Revised: Others (Klocek et al., 2019; Sitzmann et al., 2020) have proposed the use of sinusoidal activation function within a multi-layer perceptron architecture.

Original: Both of these strategies are demonstrated to vastly improve the results on many object representation tasks.
Revised: Both of these strategies are demonstrated to vastly improve the results on many object representation tasks.

Original:  M ULTIPLICATIVE FILTER NETWORKS
Revised:  M ULTIPLICATIVE FILTER NETWORKS

Original: A traditional k -layer deep network f : R n → R m is typically deﬁned by a recurrence such as:
Revised: A traditional k -layer deep network f : R n → R m is typically deﬁned by a recurrence such as:

Original: [Equation]
Revised: [Equation]

Original: where σ denotes a nonlinearity applied elementwise, W ( i ) ∈
Revised: where σ denotes a nonlinearity applied elementwise, W ( i ) ∈

Original: R d i +1 ×
Revised: R d i +1 ×

Original: d i and b ( i ) ∈
Revised: d i and b ( i ) ∈

Original: R d i +1 denote the weight and bias of the i th layer, and z ( i
Revised: R d i +1 denote the weight and bias of the i th layer, and z ( i

Original: ) ∈
Revised: ) ∈

Original: R d i denotes the hidden unit at layer i .
Revised: R d i denotes the hidden unit at layer i .

Original: We refer to these networks as compositional depth networks, because each nonlinearity is applied compositionally to outputs of the previous nonlinearity in order to achieve its representational complexity.
Revised: We refer to these networks as compositional depth networks, because each nonlinearity is applied compositionally to outputs of the previous nonlinearity in order to achieve its representational complexity.

Original: The SIREN or Fourier feature networks of (Sitzmann et al., 2020) and Tancik et al.
Revised: The SIREN or Fourier feature networks of (Sitzmann et al., 2020) and Tancik et al.

Original: (2020) respectively can be viewed as simple specializations of this structure.
Revised: (2020) respectively can be viewed as simple specializations of this structure.

Original: In a SIREN network, one uses the sinusoid σ ( x ) = sin( x ) as the nonlinearity, plus proper initialization of the weights and scaling of the input.
Revised: In a SIREN network, one uses the sinusoid σ ( x ) = sin( x ) as the nonlinearity, plus proper initialization of the weights and scaling of the input.

Original: In a Fourier features network, one replaces the input layer with
Revised: In a Fourier features network, one replaces the input layer with

Original: [Equation]
Revised: [Equation]

Original: where Ω ∈ R d 12 × n is matrix of random N (0 , τ 2 ) variables ( τ being a hyperparameter of the method), but with the typical ReLU nonlinearities σ ( x )
Revised: where Ω ∈ R d 12 × n is matrix of random N (0 , τ 2 ) variables ( τ being a hyperparameter of the method), but with the typical ReLU nonlinearities σ ( x )

Original: = ReLU( x ) .
Revised: = ReLU( x ) .

Original: Our proposed multiplicative ﬁlter network , in contrast, uses a different recursion that never results in composition of nonlinear functions.
Revised: Our proposed multiplicative ﬁlter network , in contrast, uses a different recursion that never results in composition of nonlinear functions.

Original: Speciﬁcally, an MFN is deﬁned via the following recursion
Revised: Speciﬁcally, an MFN is deﬁned via the following recursion

Original: [Equation]
Revised: [Equation]

Original: where ◦ denotes elementwise multiplication, W ( i ) , b ( i ) ,
Revised: where ◦ denotes elementwise multiplication, W ( i ) , b ( i ) ,

Original: z ( i ) are all deﬁned as above, but where g : R n → R d i is parameterized by parameters θ ( i ) (the size of θ ( i ) can vary to implicitly deﬁne the output dimensions
Revised: z ( i ) are all deﬁned as above, but where g : R n → R d i is parameterized by parameters θ ( i ) (the size of θ ( i ) can vary to implicitly deﬁne the output dimensions

Original: d i ) and denotes a nonlinear ﬁlter applied to the input directly.
Revised: d i ) and denotes a nonlinear ﬁlter applied to the input directly.

Original: Of immediate importance here is that in such a network, we never apply a nonlinearity to the output of a previous nonlinearity.
Revised: Of immediate importance here is that in such a network, we never apply a nonlinearity to the output of a previous nonlinearity.

Original: All the nonlinearity of the network occurs within the g functions; layers z ( i ) , after passing through a linear function, are simply multiplied by new ﬁlters of the input.
Revised: All the nonlinearity of the network occurs within the g functions; layers z ( i ) , after passing through a linear function, are simply multiplied by new ﬁlters of the input.

Original: This results in a considerably different type of function that is currently employed by most multi-layer networks, and indeed it is largely only by convention that we refer to such a function as a “network” at all.
Revised: This results in a considerably different type of function that is currently employed by most multi-layer networks, and indeed it is largely only by convention that we refer to such a function as a “network” at all.

Original: We now present two instantiations of the MFN, using sinusoids or a Gabor wavelet as the ﬁlter g ; we call these two networks the F OURIER N ET and G ABOR N ET respectively.
Revised: We now present two instantiations of the MFN, using sinusoids or a Gabor wavelet as the ﬁlter g ; we call these two networks the F OURIER N ET and G ABOR N ET respectively.

Original: As we show, the crucial property of a function f represented by a F OURIER N ET or G ABOR N ET is that the entire function f can also be written as a linear combination of sinusoids and Gabor wavelets of the input respectively (albeit an exponentially large number of such features, but of course also with a highly reduced space of allowable coefﬁcients on this exponential number of terms, since there are only a polynomial number of parameters that deﬁne the MFN).
Revised: As we show, the crucial property of a function f represented by a F OURIER N ET or G ABOR N ET is that the entire function f can also be written as a linear combination of sinusoids and Gabor wavelets of the input respectively (albeit an exponentially large number of such features, but of course also with a highly reduced space of allowable coefﬁcients on this exponential number of terms, since there are only a polynomial number of parameters that deﬁne the MFN).

Original: Thus, we would claim that the MFN really looks more like a (rich) Fourier or Wavelet representation of the underlying signal, just one that happens to have a similar parameterization as deep networks (and which can be tuned by typical gradient descent methods).
Revised: Thus, we would claim that the MFN really looks more like a (rich) Fourier or Wavelet representation of the underlying signal, just one that happens to have a similar parameterization as deep networks (and which can be tuned by typical gradient descent methods).

Original: 3. M ULTIPLICATIVE F OURIER N ETWORKS
Revised: 3. M ULTIPLICATIVE F OURIER N ETWORKS

Original: As our ﬁrst instantiation of the MFN, we consider using a simple sinusoidal ﬁlter
Revised: As our ﬁrst instantiation of the MFN, we consider using a simple sinusoidal ﬁlter

Original: [Equation]
Revised: [Equation]

Original: (4) with parameters θ ( i ) = { ω ( i ) ∈
Revised: with parameters θ ( i ) = { ω ( i ) ∈

Original: R d i × n , φ ( i )
Revised: R d i × n , φ ( i )

Original: ∈ R d i } .
Revised: ∈ R d i } .

Original: We term such a network the F OURIER N ET , as the sinusoidal activation (with arbitrary phase shifts, to represent sine or cosine functions equally) corresponds naturally to a Fourier random feature representation of the entire function.
Revised: We term such a network the F OURIER N ET , as the sinusoidal activation (with arbitrary phase shifts, to represent sine or cosine functions equally) corresponds naturally to a Fourier random feature representation of the entire function.

Original: An immediate and compelling feature of the F OURIER N ET , compared to networks based upon composition, is that its output can be directly viewed as a linear function of (an exponential number of)
Revised: An immediate and compelling feature of the F OURIER N ET , compared to networks based upon composition, is that its output can be directly viewed as a linear function of (an exponential number of)

Original: Fourier bases, with a low-rank set of coefﬁcients determined by the parameters of the network.
Revised: Fourier bases, with a low-rank set of coefﬁcients determined by the parameters of the network.

Original: This is conveyed by the following theorem.
Revised: This is conveyed by the following theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: The output of a Fourier Network is given by a linear combination of sinusoidal bases,
Revised: The output of a Fourier Network is given by a linear combination of sinusoidal bases,

Original: [Equation]
Revised: [Equation]

Original: for some coefﬁcients ¯ α 1: T , frequencies ¯ ω 1: T , phase offsets ¯ φ 1: T , and bias term ¯ b .
Revised: for some coefﬁcients ¯ α 1: T , frequencies ¯ ω 1: T , phase offsets ¯ φ 1: T , and bias term ¯ b .

Original: In other words, the F OURIER N ET represents its ﬁnal function as a linear combination of traditional
Revised: In other words, the F OURIER N ET represents its ﬁnal function as a linear combination of traditional

Original: Fourier bases, just as do “classical” random Fourier features, for instance.
Revised: Fourier bases, just as do “classical” random Fourier features, for instance.

Original: The key element of the proof (given in the appendix) is the fact that for two Fourier ﬁlters, with parameters ω, φ and τ, ψ respectively, their elementwise product can be transformed to a sum of the same type of ﬁlters
Revised: The key element of the proof (given in the appendix) is the fact that for two Fourier ﬁlters, with parameters ω, φ and τ, ψ respectively, their elementwise product can be transformed to a sum of the same type of ﬁlters

Original: [Equation]
Revised: [Equation]

Original: (note that the cosine can be expressed as a sine with a separate phase offset).
Revised: (note that the cosine can be expressed as a sine with a separate phase offset).

Original: Moreover, an inspection of the proof also lets us compute the exact coefﬁcients of the linear expansion, as a function of the network parameters.
Revised: Moreover, an inspection of the proof also lets us compute the exact coefﬁcients of the linear expansion, as a function of the network parameters.

Original: This is shown in the following corollary.
Revised: This is shown in the following corollary.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let i 1 , i 2 , . .
Revised: Let i 1 , i 2 , . .

Original: i k − 1 range over all (cid:81) k − 1 j =1 d
Revised: i k − 1 range over all (cid:81) k − 1 j =1 d

Original: j possible indices of each hidden unit of each layer of an MFN, and let s 2 , . .
Revised: j possible indices of each hidden unit of each layer of an MFN, and let s 2 , . .

Original: . , s k ∈ {− 1 , +1 } range over all 2 k − 1 possible binary signs; then the expansion of z ( k ) i k from (2) is given by all the terms
Revised: . , s k ∈ {− 1 , +1 } range over all 2 k − 1 possible binary signs; then the expansion of z ( k ) i k from (2) is given by all the terms

Original: [Equation]
Revised: [Equation]

Original: with a similar form for terms that begin at the i > 1 layer, multiplied by the corresponding b ( j ) i j term.
Revised: with a similar form for terms that begin at the i > 1 layer, multiplied by the corresponding b ( j ) i j term.

Original: This corollary follows simply by inspection of the proof in the appendix, noting that each additional multiplicative layer creates both a positive and negative combination of frequencies in the sinusoid terms, and a multiplication of the corresponding entries of W .
Revised: This corollary follows simply by inspection of the proof in the appendix, noting that each additional multiplicative layer creates both a positive and negative combination of frequencies in the sinusoid terms, and a multiplication of the corresponding entries of W .

Original: In other words, the multiplicative “depth” of the F OURIER N ET allows it to represent an exponential number of sinusoidal functions, but with the constraint that the actual number of coefﬁcients on these features is given by a “lowrank” tensor consisting mainly of the coefﬁcients in the W matrices.
Revised: In other words, the multiplicative “depth” of the F OURIER N ET allows it to represent an exponential number of sinusoidal functions, but with the constraint that the actual number of coefﬁcients on these features is given by a “lowrank” tensor consisting mainly of the coefﬁcients in the W matrices.

Original: This expansion also suggests a method for initializing parameters speciﬁc to this network in a manner that scales appropriately with the network size.
Revised: This expansion also suggests a method for initializing parameters speciﬁc to this network in a manner that scales appropriately with the network size.

Original: Speciﬁcally, however W ( i ) s are initialized (typically random uniform or
Revised: Speciﬁcally, however W ( i ) s are initialized (typically random uniform or

Original: Gaussian, though here with an additional scaling factor that depends on the relative scale of the input), one should divide these terms by √ k , to ensure that the variance of the ﬁnal frequency ω t is independent of the number of layers.
Revised: Gaussian, though here with an additional scaling factor that depends on the relative scale of the input), one should divide these terms by √ k , to ensure that the variance of the ﬁnal frequency ω t is independent of the number of layers.

Original: 3. M ULTIPLICATIVE G ABOR N ETWORKS
Revised: 3. M ULTIPLICATIVE G ABOR N ETWORKS

Original: A well-known deﬁciency of the pure Fourier bases is that they have global support, and thus may have difﬁculty representing more local features.
Revised: A well-known deﬁciency of the pure Fourier bases is that they have global support, and thus may have difﬁculty representing more local features.

Original: A common alternative to these bases is the use of
Revised: A common alternative to these bases is the use of

Original: Gabor ﬁlter to capture both a frequency and spatial locality component.
Revised: Gabor ﬁlter to capture both a frequency and spatial locality component.

Original: Speciﬁcally, we consider a
Revised: Speciﬁcally, we consider a

Original: Gabor ﬁlter of the form
Revised: Gabor ﬁlter of the form

Original: [Equation] 
Revised: [Equation] [Equation]

Original: with parametersθ
Revised: with parameters

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: (here

Original: [Equation]
Revised: [Equation]

Original: i ) 1: d i ∈ R n , φ ( i ) 1: d i ∈ R (cid:111) (here µ ( i ) j denotes the mean of the j th Gabor ﬁlter and γ ( i )
Revised: denotes the mean of the j th Gabor ﬁlter and γ ( i )

Original: j denotes the scale term), and where for simplicity we specify the functional form of each j = 1 . . .
Revised: j denotes the scale term), and where for simplicity we specify the functional form of each j = 1 . . .

Original: , d i coordinates of the function g : R n → R d i .
Revised: , d i coordinates of the function g : R n → R d i .

Original: We call the
Revised: We call the

Original: MFN using this ﬁlter the GaborNet .
Revised: MFN using this ﬁlter the GaborNet .

Original: As with the FourierNet, a compelling feature of the Gabor network is that the ﬁnal function f can be represented as a linear combination of Gabor ﬁlters.
Revised: As with the FourierNet, a compelling feature of the Gabor network is that the ﬁnal function f can be represented as a linear combination of Gabor ﬁlters.

Original: This is captured by the following theorem:
Revised: This is captured by the following theorem:

Original: Theorem 2.
Revised: Theorem 2.

Original: The output of a Gabor Network is given by a linear combination of Gabor bases,
Revised: The output of a Gabor Network is given by a linear combination of Gabor bases,

Original: [Equation]
Revised: [Equation]

Original:  The proof is given in the appendix, but the basic procedure is the same as the above: using the fact that just like Fourier ﬁlters, the product of Gabor ﬁlters is also a linear combination of (a different set of) Gabor ﬁlters.
Revised: for coefﬁcients ¯ α 1: T , scales ¯ γ 1: T , means ¯ µ 1: T , frequencies ¯ ω 1: T , phase offsets ¯ φ 1: T , and bias term ¯ b . The proof is given in the appendix, but the basic procedure is the same as the above: using the fact that just like Fourier ﬁlters, the product of Gabor ﬁlters is also a linear combination of (a different set of) Gabor ﬁlters.

Original: Likewise, we can also compute the explicit form of the coefﬁcients and for this linear basis expansion, with the explicit form again given in the appendix.
Revised: Likewise, we can also compute the explicit form of the coefﬁcients and for this linear basis expansion, with the explicit form again given in the appendix.

Original: One relevant point, though, is how we choose initializations for the γ and µ parameters.
Revised: One relevant point, though, is how we choose initializations for the γ and µ parameters.

Original: Since γ effectively acts as an inverse covariance term of a Gaussian, a Gamma( α, β ) random variable (the conjugate prior of the
Revised: Since γ effectively acts as an inverse covariance term of a Gaussian, a Gamma( α, β ) random variable (the conjugate prior of the

Original: Gaussian inverse covariance), is a reasonable choice for this parameter.
Revised: Gaussian inverse covariance), is a reasonable choice for this parameter.

Original: And since the ¯ γ functions in the ﬁnal linear expansion end up being a sum of the individual random γ ( i ) terms at each layer, we scale each layer’s α term by 1 /k to effectively control this parameter at the ﬁnal layer.
Revised: And since the ¯ γ functions in the ﬁnal linear expansion end up being a sum of the individual random γ ( i ) terms at each layer, we scale each layer’s α term by 1 /k to effectively control this parameter at the ﬁnal layer.

Original: We also simply choose each µ ( i ) to be uniformly distributed over the range of the allowable input space x .
Revised: We also simply choose each µ ( i ) to be uniformly distributed over the range of the allowable input space x .

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We test MFNs on a broad range of representation tasks, showing that the relative simplicity of MFNs improves upon the performance of existing neural representation methods.
Revised: We test MFNs on a broad range of representation tasks, showing that the relative simplicity of MFNs improves upon the performance of existing neural representation methods.

Original: Our set of experiments draws from those presented in Sitzmann et al.
Revised: Our set of experiments draws from those presented in Sitzmann et al.

Original: (2020) alongside SIREN (image representation, shape representation, and differential equation experiments) and in Tancik et al.
Revised: (2020) alongside SIREN (image representation, shape representation, and differential equation experiments) and in Tancik et al.

Original: (2020) alongside Fourier feature networks with Gaussian random features, which we call FF Gaussian (image generalization and 3D inverse rendering experiments).
Revised: (2020) alongside Fourier feature networks with Gaussian random features, which we call FF Gaussian (image generalization and 3D inverse rendering experiments).

Original: In each case, we compare against the set of models tested in the original experiment (generally, either SIREN or FF Gaussian, along with a basic ReLU MLP).
Revised: In each case, we compare against the set of models tested in the original experiment (generally, either SIREN or FF Gaussian, along with a basic ReLU MLP).

Original: Full details on hyperparameters and training speciﬁcations are available in the appendix.
Revised: A PyTorch implementation of MFN is available at https: //github.com/boschresearch/multiplicative-filter-networks , and full details on hyperparameters and training speciﬁcations are available in the appendix.

Original: 4. I MAGE R EPRESENTATION & G ENERALIZATION
Revised: 4. I MAGE R EPRESENTATION & G ENERALIZATION

Original:  We ﬁrst examine the ability of several networks architectures in the task of image representation as described in Section 1, where we ﬁt the network to a function f : R 2 → R c using a dataset where input coordinates ( x, y ) corresponding to the output pixel value at those coordinates (with c = 1 or 3 for grayscale or RGB images, respectively).
Revised: [Table] We ﬁrst examine the ability of several networks architectures in the task of image representation as described in Section 1, where we ﬁt the network to a function f : R 2 → R c using a dataset where input coordinates ( x, y ) corresponding to the output pixel value at those coordinates (with c = 1 or 3 for grayscale or RGB images, respectively).

Original: To demonstrate this, we construct such a dataset from a 256 × 256 pixel grayscale image and ﬁt various models using a simple mean squared error (MSE) loss, including SIREN, G ABOR N ET , F OURIER N ET , and ReLU MLPs with and without positional encoding (PE).
Revised: To demonstrate this, we construct such a dataset from a 256 × 256 pixel grayscale image and ﬁt various models using a simple mean squared error (MSE) loss, including SIREN, G ABOR N ET , F OURIER N ET , and ReLU MLPs with and without positional encoding (PE).

Original: Visual results and a plot of PSNR early in training are shown in (Figure 1).
Revised: Visual results and a plot of PSNR early in training are shown in (Figure 1).

Original: In particular, both F OURIER N ET and G ABOR N ET show quicker initial convergence than other architectures.
Revised: In particular, both F OURIER N ET and G ABORN ET show quicker initial convergence than other architectures.

Original: PSNRs after training (Table 1) show that SIREN eventually outperforms F OURIER N ET , while G ABOR N ET remains the best model throughout the training.
Revised: PSNRs after training (Table 1) show that SIREN eventually outperforms F OURIER N ET , while G ABOR N ET remains the best model throughout the training.

Original: Indeed, after only 1000 training iterations, G ABOR N ET performs reconstruction better than all other models trained for 10 times longer.
Revised: Indeed, after only 1000 training iterations, G ABOR N ET performs reconstruction better than all other models trained for 10 times longer.

Original: [Table] [Table] We can broaden the task above to represent video by appending a third dimension to the input: the output corresponding to the input ( x, y, t ) is the pixel value at ( x, y ) at frame t .
Revised: We can broaden the task above to represent video by appending a third dimension to the input: the output corresponding to the input ( x, y, t ) is the pixel value at ( x, y ) at frame t .

Original: We aim to represent a 300 frame color video with 512 × 512 resolution in this manner, testing all of the architectures used in the previous experiment.
Revised: We aim to represent a 300 frame color video with 512 × 512 resolution in this manner, testing all of the architectures used in the previous experiment.

Original: As shown in Figure 1, SIREN and G ABOR N ET are most capable of reproducing ﬁne details of the original video, such as whiskers and and eye color.
Revised: As shown in Figure 1, SIREN and G ABOR N ET are most capable of reproducing ﬁne details of the original video, such as whiskers and and eye color.

Original: This is reﬂected in the PSNR of these reconstructions (Table 1); SIREN performs best with a PSNR over 30 dB, while the G ABOR N ET reconstruction comes withindB of SIREN and exhibits much lower variation across frames than any other model.
Revised: This is reﬂected in the PSNR of these reconstructions (Table 1); SIREN performs best with a PSNR over 30 dB, while the G ABOR N ET reconstruction comes within 1 dB of SIREN and exhibits much lower variation across frames than any other model.

Original: [Table]
Revised: [Table]

Original: In addition to representing images, we demonstrate that the MFNs are able to generalize the representation to unseen pixels.
Revised: In addition to representing images, we demonstrate that the MFNs are able to generalize the representation to unseen pixels.

Original: We train the networks using only 25% of the image pixels (every other pixel in the width and height dimensions) and evaluate using the complete images.
Revised: We train the networks using only 25% of the image pixels (every other pixel in the width and height dimensions) and evaluate using the complete images.

Original: We compare the results of our methods with the
Revised: We compare the results of our methods with the

Original: Fourier feature networks on two datasets (natural and text images) presented in Tancik et al.
Revised: Fourier feature networks on two datasets (natural and text images) presented in Tancik et al.

Original: The peak signal-to-noise ratio (PSNR) metric is used to evaluate the performance.
Revised: The peak signal-to-noise ratio (PSNR) metric is used to evaluate the performance.

Original: As we can see from Table 2, both F OURIER N ET and G ABORN ET outperform all versions of the Fourier feature networks that uses basic, positional encoding, and random Gaussian features.
Revised: As we can see from Table 2, both F OURIER N ET and G ABORN ET outperform all versions of the Fourier feature networks that uses basic, positional encoding, and random Gaussian features.

Original: Some examples of the generated images are presented in Figure  2, with additional images in Appendix B.2.
Revised: Some examples of the generated images are presented in Figure [Table] [Figure] 2, with additional images in Appendix B.2.

Original: Visually, the MFNs’ generated images also have better quality over the baselines, particularly in Text datasets.
Revised: Visually, the MFNs’ generated images also have better quality over the baselines, particularly in Text datasets.

Original: Some parts of the text are missing in the baselines images (highlighted with red rectangles in Figure 2), whereas the MFNs completely generate all parts of the text in the images.[Figure]
Revised: Some parts of the text are missing in the baselines images (highlighted with red rectangles in Figure 2), whereas the MFNs completely generate all parts of the text in the images.

Original: 4. D IFFERENTIAL EQUATIONS
Revised: 4. D IFFERENTIAL EQUATIONS

Original: In this section, we aim to solve boundary value problems which are supervised by different forms of gradient information from the functional at hand.
Revised: In this section, we aim to solve boundary value problems which are supervised by different forms of gradient information from the functional at hand.

Original: We ﬁrst focus on the Poisson equation, where we demonstrate image reconstruction in two settings where the supervision for the model is brought about by gradients and Laplacians respectively.
Revised: We ﬁrst focus on the Poisson equation, where we demonstrate image reconstruction in two settings where the supervision for the model is brought about by gradients and Laplacians respectively.

Original: It is worth noting that the model is never presented with real function values.
Revised: It is worth noting that the model is never presented with real function values.

Original: We then focus on two 2 nd order differential equations, namely, the
Revised: We then focus on two 2 nd order differential equations, namely, the

Original: Helmholtz equation and the wave equation, where we solve for the wave ﬁeld, where the network is supervised by a known source function.
Revised: Helmholtz equation and the wave equation, where we solve for the wave ﬁeld, where the network is supervised by a known source function.

Original: We demonstrate image reconstruction using gradients and compare the performance of F OURIERN ET and G ABOR N ET with SIREN and ReLU MLP for the Poisson equation.
Revised: We demonstrate image reconstruction using gradients and compare the performance of F OURIERN ET and G ABOR N ET with SIREN and ReLU MLP for the Poisson equation.

Original: We use the same loss function as in (Sitzmann et al., 2020) in (5).
Revised: We use the same loss function as in (Sitzmann et al., 2020) in (5).

Original: Figures 3a and 3b show that the image is reconstructed successfully when the networks are trained while being supervised by gradients and Laplacians respectively, while ReLU fails spectacularly.
Revised: Figures 3a and 3b show that the image is reconstructed successfully when the networks are trained while being supervised by gradients and Laplacians respectively, while ReLU fails spectacularly.

Original: Table 3 depicts the losses of each method afteriterations, where it can be seen that G ABOR N ET beats other baselines in terms of performance.[Figure]
Revised: Table 3 depicts the losses of each method afteriterations, where it can be seen that G ABOR N ET beats other baselines in terms of performance.

Original: The Helmholtz and wave equations are related to the physical modeling of diffusion and waves and are closely related to a Fourier transform.
Revised: The Helmholtz and wave equations are related to the physical modeling of diffusion and waves and are closely related to a Fourier transform.

Original: Hence, we focus our attention on describing the Helmholtz equation.
Revised: Hence, we focus our attention on describing the Helmholtz equation.

Original: We aim to solve for the wave ﬁeld and compare the performance of F OURIER N ET and G ABOR N ET with SIREN and ReLU MLP.
Revised: We aim to solve for the wave ﬁeld and compare the performance of F OURIER N ET and G ABOR N ET with SIREN and ReLU MLP.

Original: To accommodate for complex-valued solutions, the network is conﬁgured to output two values which can be interpreted as the real and imaginary parts.
Revised: To accommodate for complex-valued solutions, the network is conﬁgured to output two values which can be interpreted as the real and imaginary parts.

Original: We use the same loss function as used in (Sitzmann et al., 2020)
Revised: We use the same loss function as used in (Sitzmann et al., 2020)

Original: (see Section 4. of (Sitzmann et al., 2020) for details).
Revised: (see Section 4.3 of (Sitzmann et al., 2020) for details).

Original: Figure 4 shows the magnitude and the phase of the reconstructed wave front for a single Gaussian source placed at the center of a medium with uniform wave propagation velocity.
Revised: Figure 4 shows the magnitude and the phase of the reconstructed wave front for a single Gaussian source placed at the center of a medium with uniform wave propagation velocity.

Original: Table 3 depicts the losses of each method after 50000 iterations, and shows that G ABOR N ET beats F OURIER N ET and SIREN in terms of performance, while, as previously shown in (Sitzmann et al., 2020), ReLU MLP fails miserably.
Revised: Table 3 depicts the losses of each method after 50000 iterations, and shows that G ABOR N ET beats F OURIER N ET and SIREN in terms of performance, while, as previously shown in (Sitzmann et al., 2020), ReLU MLP fails miserably.

Original: Details pertaining to the network and training are relegated to Appendix B.3.
Revised: Details pertaining to the network and training are relegated to Appendix B.3.

Original: 4. S HAPE R EPRESENTATION VIA SIGNED DISTANCE FIELDS
Revised: 4. S HAPE R EPRESENTATION VIA SIGNED DISTANCE FIELDS

Original:  Recent work (Park et al., 2019) has explored the problem of 3D shape representation with neural architectures, often with surprisingly effective results.
Revised: [Figure] Recent work (Park et al., 2019) has explored the problem of 3D shape representation with neural architectures, often with surprisingly effective results.

Original: This is done by training on raw geometric data: given an oriented point cloud, we seek to learn a function f : R 3 → R that takes points as input such that the zero level set { x | f ( x ) = 0 } of the network accurately represents the surfaces of the shape.
Revised: This is done by training on raw geometric data: given an oriented point cloud, we seek to learn a function f : R 3 → R that takes points as input such that the zero level set { x | f ( x ) = 0 } of the network accurately represents the surfaces of the shape.

Original: Effective training objectives for this task has been explored in considerable depth; we use the training loss presented by Park et al.
Revised: Effective training objectives for this task has been explored in considerable depth; we use the training loss presented by Park et al.

Original: and used in Sitzmann et al.
Revised: and used in Sitzmann et al.

Original: (2020), which includes terms involving the network’s output (penalties to encourage SDF values near and away from 0 for surface and off-surface points, respectively) as well as its gradients (a term encouraging the gradient to match the surface points’ normals, and a gradient norm penalty throughout the entire 3D space).
Revised: (2020), which includes terms involving the network’s output (penalties to encourage SDF values near and away from 0 for surface and offsurface points, respectively) as well as its gradients (a term encouraging the gradient to match the surface points’ normals, and a gradient norm penalty throughout the entire 3D space).

Original: A complete description of the loss function is left to the appendix.
Revised: A complete description of the loss function is left to the appendix.

Original: Figure 5a shows the results of our shape representation task on SIREN, a standard ReLU MLP, and both variants of MFN.
Revised: Figure 5a shows the results of our shape representation task on SIREN, a standard ReLU MLP, and both variants of MFN.

Original: As can be seen, the ReLU network’s fails to represent some features of the scene entirely, such as doorways, picture frames, and pillows.
Revised: As can be seen, the ReLU network’s fails to represent some features of the scene entirely, such as doorways, picture frames, and pillows.

Original: Both MFN architectures far outperform this baseline, and are able to reconstruct the room and objects within it to a recognizable degree.
Revised: Both MFN architectures far outperform this baseline, and are able to reconstruct the room and objects within it to a recognizable degree.

Original: However, likely owing to its strong ability to produce smooth outputs and gradients, SIREN is largely able to avoid the visual artifacts on ﬂat surfaces like walls that remain in reconstructions by F OURIER N ET and, to a lesser extent, G ABOR N ET .
Revised: However, likely owing to its strong ability to produce smooth outputs and gradients, SIREN is largely able to avoid the visual artifacts on ﬂat surfaces like walls that remain in reconstructions by F OURIER N ET and, to a lesser extent, G ABOR N ET .

Original: 4. 3D I NVERSE R ENDERING FOR V IEW S YNTHESIS
Revised: 4. 3D I NVERSE R ENDERING FOR V IEW S YNTHESIS

Original: [Table] In this view synthesis task, we aim to reconstruct 3D representation from the observed 2D photographs.
Revised: In this view synthesis task, we aim to reconstruct 3D representation from the observed 2D photographs.

Original: Using the reconstructed 3D representation, we then render 2D images from new viewpoints.
Revised: Using the reconstructed 3D representation, we then render 2D images from new viewpoints.

Original: We use the “simpliﬁed Neural Radiance Fields (NeRF)” task on Lego dataset presented in Tancik et al.
Revised: We use the “simpliﬁed Neural Radiance Fields (NeRF)” task on Lego dataset presented in Tancik et al.

Original: The networks are trained to predict the color (in RGB format) and the volume density at a given 3D location of the viewpoint.
Revised: The networks are trained to predict the color (in RGB format) and the volume density at a given 3D location of the viewpoint.

Original: Volumetric rendering is then used to re-render the 2D image photograph at the viewpoint.
Revised: Volumetric rendering is then used to re-render the 2D image photograph at the viewpoint.

Original: The training loss is computed as the mean squared error between the rendered photograph and the actual 2D image observations.
Revised: The training loss is computed as the mean squared error between the rendered photograph and the actual 2D image observations.

Original: The results in Table 4 show that both MFNs perform competitively in the task.
Revised: The results in Tableshow that both MFNs perform competitively in the task.

Original: G ABOR N ET has a slight advantage over the baselines and F OURIER N ET in terms of the overall PSNR metric.
Revised: G ABOR N ET has a slight advantage over the baselines and F OURIER N ET in terms of the overall PSNR metric.

Original: It also arguably produces a slightly better image rendering quality as shown in Figure 5b and other images in Appendix B.5.
Revised: It also arguably produces a slightly better image rendering quality as shown in Figure 5b and other images in Appendix B.5.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original:  We have introduced multiplicative ﬁlter networks (MFNs), a class of neural representation architectures that forego the usual compositional notion of network depth in favor of a similarly expressive multiplicative operation.
Revised: [Table] We have introduced multiplicative ﬁlter networks (MFNs), a class of neural representation architectures that forego the usual compositional notion of network depth in favor of a similarly expressive multiplicative operation.

Original: They also admit a natural signal processing interpretation, as in the two instantiations of MFNs, F OURIER N ET and G ABOR N ET , which are proven to be exactly equivalent to a linear combination of sinusoidal or Gabor wavelet bases, respectively.
Revised: They also admit a natural signal processing interpretation, as in the two instantiations of MFNs, F OURIER N ET and G ABOR N ET , which are proven to be exactly equivalent to a linear combination of sinusoidal or Gabor wavelet bases, respectively.

Original: In experiments, we show that, despite their simplicity relative to other deep architectures designed for implicit representation, 
Revised: In experiments, we show that, despite their simplicity relative to other deep architectures designed for implicit representation, MFNs stand up to or surpass the previous state of the art on a battery of representation tasks. A P ROOFS AND DERIVATIONS

Revised: We begin by stating the general condition under which an MFN can ultimately be expressed as a linear combination of the original ﬁlter function: the fact that elementwise multiplication of the bases results in a sum of the same bases with different parameters.

Revised: Deﬁnition 1.

Revised: A ﬁlter g ( x ; θ ) satisﬁes the multiplicative sum property if for some θ (1) and θ (2) we have that

Revised: [Equation]

Original: MFNs stand up to or surpass the previous state of  the art on a battery of representation tasks.
Revised: for some alternative set of parameters ¯ θ (1: N ) and coefﬁcients β 1: N . We now state the general theorem that if the ﬁlter used in an MFN satisﬁes the multiplicative sum property, then the function represented by the MFN can also be expressed as a linear combination of the ﬁlter bases applies to the input. Theorem 3. Let f by an MFN deﬁned by the recurrence

Revised: [Equation]

Revised: with a ﬁlter g that satisﬁes the multiplicative sum property.

Revised: Then f can be expressed as

Revised: [Equation]

Revised: where we use the notation g ( x ; τ ( t ) j ) to denote a scalar-valued application of the ﬁlter, speciﬁc to the j th coordinate.

Revised: Proof.

Revised: The proof follows by induction on i .

Revised: The base case of z (1) follows immediately from the deﬁnition (and in fact without any bias term b ).

Revised: Now suppose that for layer i − 1 , for each z

Revised: ( i − 1)

Revised: j we have

Revised: [Equation]

Revised: [Equation]

Revised: Then we have that

Revised: [Equation]

Revised: [Equation]

Revised: where the line (15) to (16) follows from the multiplicative sum instruction.

Revised: The ﬁnal output f ( x ) is simply a linear function of the ﬁnal hidden layer z ( k ) , and thus also satisﬁes the condition of the theorem.

Revised: We now prove the multiplicative sum property for both the Fourier and Gabor ﬁlters.

Revised: This, together with the theorem above, proves the results in the main text, stated for the FourierNet and GaborNet functions explicitly.

Revised: A. F OURIER N ETWORKS

Revised: Theorem 4.

Revised: The Fourier basis function

Revised: [Equation]

Revised: satisﬁes the multiplicative sum property with β and ¯ θ (1:2) =

Revised: (cid:8) ¯ ω (1:2) , ¯ φ (1:2) (cid:9) given bvy

Revised: [Equation]

Revised: Proof.

Revised: This fact follows from simple rules for sum of sinusoidal function

Revised: [Equation]

Revised: and was also highlighted in the main text

Revised: [Equation]

Revised: with the parameters deﬁned as in the theorem statement.

Revised: An inspection of the two theorems above directly leads to the explicit expansion of the terms in the sequence as well, which we restate here.

Revised: Corollary 2.

Revised: Let i 1 , i 2 , . .

Revised: i k − 1 range over all (cid:81) k − 1 j =1 d

Revised: j possible indices of each hidden unit of each layer of an MFN, and let s 2 , . .

Revised: . , s k ∈ {− 1 , +1 } range over all 2 k − 1 possible binary signs; then the expansion of z ( k ) i k given is given by all the terms

Revised: [Equation]

Revised: with a similar form for terms that begin at the i > 1 layer, multiplied by the corresponding b ( j ) i j term.

Revised: A. G ABOR N ETWORKS

Revised: We now prove similar properties for the Gabor Network.

Revised: Theorem 5.

Revised: The Gabor basis function

Revised: [Equation]

Revised: satisﬁes the multiplicative sum property with β and ¯ θ (1:2) =

Revised: (cid:8) ¯ γ (1:2) , ¯ µ (1:2) , ¯ ω (1:2) , ¯ φ (1:2) (cid:9) given by

Revised: [Equation]

Revised: Proof.

Revised: The sinusoidal terms in the Gabor ﬁlter proceed in the exact same manner as for the Fourier ﬁlter.

Revised: The exponential terms follow from the fact, well known from Gaussian distributions, that exp

Revised: [Equation]

Revised: The ﬁrst term is a constant, and so can be folded into the β terms, while the second is the exponential term of a Gabor ﬁlter with the parameters given above.

Revised: Finally, just as for the FourierNet, we can also provide precise characterizations of each term in the ﬁnal Gabor ﬁlter expansion.

Revised: This corollary follows immediately from the proof above, plus some algebraic simpliﬁcation.

Revised: Corollary 3.

Revised: Let i 1 , i 2 , . .

Revised: i k − 1 range over all (cid:81) k − 1 j =1 d

Revised: j possible indices of each hidden unit of each layer of an MFN, and let s 2 , . .

Revised: . , s k ∈ {− 1 , +1 } range over all 2 k − 1 possible binary signs; then the expansion of z ( k ) i k given is given by all the terms

Revised: [Equation]

Revised: with a similar form for terms that begin at the i > 1 layer, multiplied by the corresponding b ( j ) i j term.

Revised: B A DDITIONAL E XPERIMENTAL D ETAILS AND R ESULTS

Revised: [Figure]

Revised: In the image representation task, all models used were 3 layers with 256 hidden units per layer, and were trained for 10000 iterations.

Revised: All models were trained with the Adam optimizer, with a learning rate of 10 − 2 for the two MFNs and a learning rate of 10 − 4 for all other models.

Revised: Both MFN models used an input scaling factor of 256 .

Revised: The cat video used in the video representation task is available at the following url: https://www.pexels.com/video/the-full-facial-features-of-a-pet-catModels used for video representation were 3 layers with 1024 hidden units per layer, and were trained for 10000 iterations.

Revised: We performed an exponentially grid search over learning rates for each model, testing a learning rate of 10 − i for i ∈ { 2 , 3 , 4 , 5 , 6 } , and found the best performance when using 1 e − 5 for SIREN and 10 − 3 for all other models.

Revised: Both MFN architectures used a scaling factor γ of 256 .

Revised: B. I MAGE G ENERALIZATION

Revised: In the image generalization experiments, we use two datasets ( Natural and Text datasets); each contains 16 images with a size of 512 × 512 pixels.

Revised: We use 25% of the pixels resulting 256 ×images to train the models and use the full image size in the evaluation.

Revised: We construct and train the models separately for each image.

Revised: The models are trained to map the input of ( x, y ) coordinates to the corresponding RGB values.

Revised: We train all the models in 2000 epochs.

Revised: The ﬁnal result is computed by averaging the results over the 16 images in each dataset.

Revised: For each of the MFNs and Fourier feature networks, we construct a 4 hidden layer network with 256 nodes in each hidden layer.

Revised: For both F OURIER N ET and G ABOR N ET , we set the input scaling factor to 256, with the value of α in G ABOR N ET set to 3.0.

Revised: We use Adam optimizer with learning rate of 3e-3 and no weight decay.

Revised: For the Fourier feature models, we use the hyperparameters presented in the code repository published by the authors.

Revised: The optimizer used in the Fourier feature models is the Adam optimizer with learning rate of 1e-3 and no weight decay.

Revised: All the Fourier feature models use 256 as the embedding size for the random features.

Revised: The scaling factor for the Fourier feature networks with positional encoding is set to 6 for the Natural dataset and 5 for the Text dataset.

Revised: The scaling factor for the Fourier feature networks with Gaussian random features is set to 10 for the Natural dataset and 14 for the Text dataset.

Revised: As in the code implementation published by (Tancik et al., 2020), we implement our model in Python using JAX as the deep learning framework.

Revised: In addition to the experimental results presented in Section 4.1, we show additional generated images for Natural dataset in Figure 7 and Text dataset in Figure 8.

Revised: We can similarly see from the ﬁgure that the quality of generated images by the MFNs are superior to the baselines in Text dataset and competitive in the Natural dataset.

Revised: The Fourier feature networks (especially with basic and positional encoding) failed to generate some parts of the texts in the images.

Revised: The use of Gaussian random features improves the generated images, but still fails to generate small portions of the texts.

Revised: In contrast, both F OURIER N ET and G ABOR N ET correctly generate all parts of the texts.

Revised: Figurealso highlights the parts of the texts that are failed to be generated by the baselines methods.

Revised: In the

Revised: Natural dataset, all of the models except the Fourier feature networks using basic encoding performs equally well with very similar visual results.

Revised: Comparison with the traditional Fourier transform.

Revised: In addition to the experiments above, we also run the standard upsampling technique that performs FFT on the 256 × 256 pixels images and then performs inverse FFT on the extended frequency domains that outputs 512 × 512 pixels images.

Revised: We then measure the PSNR on the unseen pixels, similar to the experiments on the MFNs and Fourier feature networks.

Revised: This traditional FFT upsampling method performs quite well on both

Revised: Natural and Text dataset with 73 .

Revised: 80 ± 12 . 20 and 66 .

Revised: 78 ± 8 .

Revised: 14 PSNRs respectively.

Revised: However, the standard Fourier analysis on the traditional upsampling methods serves a different purpose compared to the one in the implicit neural representations (the MFNs, SIRENs, and Fourier feature networks).

Revised: The standard Fourier analysis does not scale well to higher dimensions, or it does not admit to backpropagation through more complex loss functions (e.g., in neural rendering ﬁelds).

Revised: The lack of scalability and non-differentiability hinders the usage of FFT based upsampling methods in an end-to-end manner and hence adds signiﬁcant barriers to such methods being employed in generic tasks.

Revised: In contrast, the MFNs (and other implicit neural representations) provides generic methods that apply to generic tasks where we can learn based upon derivatives, as shown in our paper and the related works (Sitzmann et al., 2020; Tancik et al., 2020; Mildenhall et al., 2020).

Revised: [Figure]

Revised: For the image reconstruction tasks using the supervision of gradients and Laplacians of the ground truth, we use 5 layer networks with a hidden size of 256 .

Revised: The networks were trained using Adam

Revised: Optimizer.

Revised: For the image reconstruction task using gradients, the learning rate chosen for F OURIERN ET and G ABOR N ET was 1e-3, which was chosen by grid search.

Revised: The batch size was chosen to be 16384 , as in SIREN.

Revised: For SIREN, we used the reference implementation of it, which also uses 5 layer networks with a hidden size of 256 .

Revised: The same architecture was also used for ReLU MLP.

Revised: All the networks were trained for 10000 iterations.

Revised: For the image reconstruction task using Laplacians, the learning rates chosen for F OURIER N ET and G ABOR N ET were 5e-3 and 3.5e-3 respectively, which was chosen by grid search.

Revised: The batch size was chosen to be 16384 , same as SIREN.

Revised: For SIREN, we used the reference implementation of it.

Revised: All the networks were trained for 10000 iterations.

Revised: B.3. H ELHMHOLTZ AND W AVE E QUATION

Revised: For the single source inversion task corresponding to the Helmholtz equation, we use 5 layer networks for both F OURIER N ET and G ABOR N ET with a hidden size of 256 .

Revised: The networks pertaining to SIREN, ReLU MLP, F OURIER N ET and G ABOR N ET are trained using randomly sampled points from a (cid:96) ∞ ball ∈ R 2 of radius 1 .

Revised: The networks were trained using Adam optimizer.

Revised: The learning rates chosen for F OURIER N ET and G ABOR N ET were 2.5e-4 and 5e-4 respectively, which was chosen by grid search.

Revised: The batch size was chosen to be 32 , same as SIREN.

Revised: For SIREN, we used the reference implementation of it, which also uses 5 layer networks with a hidden size of 256 .

Revised: The same architecture was also used for ReLU MLP.

Revised: All the networks were trained for 50000 iterations.

Revised: For the wave equation, we use 5 layer networks for both F OURIER N ET and G ABOR N ET with a hidden size of 512 .

Revised: The networks were trained using Adam optimizer.

Revised: The learning rates is chosen for F OURIER N ET and G ABOR N ET were 1e-4 and 2e-4 respectively, which was chosen by grid search.

Revised: The batch size was chosen to be 32 , same as SIREN.

Revised: For SIREN, we used the reference implementation of it, which also uses 5 layer networks with a hidden size of 512 .

Revised: All the networks were trained for 10000 iterations.

Revised: B. S HAPE R EPRESENTATION AND SDF F ITTING

Revised: The 3D room oriented point cloud used in our shape representation is freely available at http://www.turbosquid.com

Revised: The loss function used to train models on this task is identical to the one presented in Sitzmann et al.

Revised: (2020); namely, for a model N training on points x ∈ Ω in an oriented point cloud with surface Ω 0 , we use

Revised: [Equation]

Revised: where n ( x ) is the normal of the SDF f (i.e. n ( x ) = ∇ x f ( x ) , and we use the following prescribed values for the scaling factors of the various terms: λ 1 = 50 , λ 2 = 3000 , λ 3 = 100 , λ 4 = 3000 , and α = 100 .

Revised: Models used to ﬁt SDFs were 5 layers with 512 hidden units per layer, and were trained forepochs (roughly 175k iterations).

Revised: We performed a grid search to tune the learning rate for each model, searching over { a × 10 − i | a ∈ { 1 , 5 } , i ∈ { 2 , 3 , 4 , 5 , 6 }} .

Revised: This revealed best performance for F OURIER N ET at 10 − 4 , G ABOR N ET and ReLU at 5 × 10 − 4 , and SIREN at 10 − 5 .

Revised: Input scaling factors used for F OURIER N ET and G ABOR N ET were 128 and 256 , respectively.

Revised: B. 3D I NVERSE R ENDERING FOR V IEW S YNTHESIS

Revised: In the view synthesis task, we aim to reconstruct 3D representation from the observed 2D photographs.

Revised: The input to the models is a 3D coordinate ( x, y, z ) of a viewpoint.

Revised: The models need to predict the corresponding 4D vector ( r, g, b, v ) where ( r, g, b ) corresponds to the predicted color in

Revised: RGB format, and v corresponds to the volume density.

Revised: The outputs of the models are then used as the input for the volumetric rendering procedure, which outputs the ﬁnal rendered images from the viewpoint.

Revised: The loss is computed as the mean squared error between the generated images and the observed 2D photograph.

Revised: Since the rendering procedure is fully differentiable, all the training can be done end-to-end.

Revised: The Lego dataset in the “simpliﬁed NeRF” task contains 2D photographs from 120 viewpoints downsampled to 400 × 400 pixel resolution.

Revised: We split the datasets into 100 training images, 7 validation

Revised: GaborNet, images, and images.

Revised: All the results in Table 4, rendered images in Figure 5b, as well as additional rendered images in Figure 10 are computed from the test set.

Revised: In all models, we use ahidden layer network with 256 nodes in each hidden layer.

Revised: For both F OURIER N ET and G ABOR N ET , we set the input scaling factor to 180, with the value of α in G ABOR N ET set to 1.0.

Revised: We use Adam optimizer with learning rate of 1e-3 and no weight decay.

Revised: For the Fourier feature models, we use the hyperparameters presented in the code repository published by the authors.

Revised: The optimizer used in the Fourier feature models is Adam optimizer with learning rate of 5e-4 and no weight decay.

Revised: All the Fourier feature models use 256 as the embedding size for the random features.

Revised: All models use 2048 batches of rays, except G ABOR N ET that uses 1440 batches of rays due to memory restriction.