Periodic Signal Recovery with Regularized Sine Neural Networks
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
We consider the problem of learning a periodic one-dimensional signal with neural networks, and designing models that are able to extrapolate the signal well beyond the training window.
First, we show that multi-layer perceptrons with ReLU activations are provably unable to perform this extrapolation task, and lead to poor performance in practice even close to the training window.
Then, we propose a modified training procedure for two-layer architectures with sine activations with a more diverse feature initialization and well-chosen non-convex regularization, that is able to extrapolate the signal with low error well beyond the training window.
This procedure yields results several orders of magnitude better than its competitors for distant extrapolation (beyond 100 periods of the signal), while being able to accurately recover the frequency spectrum of the signal in a multi-tone setting.
1. Introduction
Real-world signals, such as weather forecasts, biological data or financial indicators, often exhibit periodic patterns.
These patterns are notoriously difficult to capture with standard neural networks such as ReLU networks (Parascandolo et al., 2017; Eger et al., 2019), thus making deep learning approaches rather unsuccessful for these prediction tasks.
While most of the literature focused on using activation functions that exhibit a periodic behavior (Parascandolo et al., 2017; Eger et al., 2019; Ziyin et al., 2020; Mehta et al., 2021), our analysis shows that this approach is, in essence, necessary yet insufficient for the purpose of distant extrapolation.
More precisely, we show that ReLU networks are fundamentally unable to learn periodic signals, and changing the activations to sine functions requires additional work to ensure proper training.
In this work, we propose a novel regularization for sine neural networks that is able to accurately learn the frequency spectrum of a target signal, and thus extrapolate it well beyond the training window. Sine-based architectures. Several architectures using sinusoids as activation have already been proposed in various contexts.
For the purpose of image representation, Tancik et al. (2020) introduced a sine-based mapping with static weights used as a preprocessing step to circumvent the issues of ReLU-based networks in learning high frequency components. In a relatively similar setting, but with learned frequency weights, Sitzmann et al. (2020) introduced a sine-based multi-layer architecture to encode various types of signals with high-frequency components.
Such changes in architecture are shown to empirically overcome the problem of learning high frequency components, typically encountered using standard ReLU-based networks (see for instance Jacot et al. (2018) for estimations of the speed at which each component is learned, or Rahaman et al. for a more targeted discussion of the spectral bias towards lower frequencies).
Contrary to the piecewise-linear functions learned with ReLU activations, such choices have the representation power sufficient to learn periodicity present in the data.
To investigate this possibility, we construct a one-dimensional well-defined problem where the underlying symmetry must be discovered during training. Problem setup. We consider the problem of recovering a periodic function given observations only on a bounded interval, where the period is unknown. Under the condition that at least one period is observed, this problem is always well-defined, however the recovery of the periodic function on the full real line requires one to uncover the underlying symmetry, which can be significantly harder than fitting the data on the observed interval.
For all the following, let T ∈ R ∗ + be a fixed but unknown period, and R ∈ R ∗ + be a fixed window size.
For all t ∈ R ∗ + , let F t ⊆
( R → R ) be the set of continuous t -periodic functions, i.e. satisfying ∀ x ∈ R , f ( x + t ) =
f ( x ) .
Note that these sets of functions are not disjoint.
We will assume in all the following that at least two periods have been observed, that is to say R ≥ 2 T .
Without this condition, recovering the entire function is clearly impossible, and this condition alone is sufficient to make the problem well defined:
If f ∗ ∈ F T with T ≤ R/ 2 , then the squared-error minimizer is unique (see Appendix A) for a proof argmin
[Equation]
The difficulty of this problem does not come from a plethora of indistinguishable minima, as is customary in machine learning, but from the unusual structure of the assumption.
Although each F t is a vector space, their union is not, for the sum of two periodic functions is not periodic in general.
It also lacks an easily exploitable convexity to treat it as an optimization problem that we could easily solve.
Moreover, ignoring the periodicity hypothesis and just fitting a function g : [0 , R ] → R to the segment, with a neural network for instance, renders the problem ill-defined, for the loss is supported only on the segment and does not specify how to extend the function to the entire real line.
2. Proposed architecture and training procedure 2.1. Regular Sine-based neural networks
We experiment with two-layer networks with sine as activation (sin : R → [ − 1 , +1]) .
For a width m ∈ N ∗ , we initialize independently at random ω
i ∼ N (0 , 1) , a i ∼ N (0 , 1 / √ m ) and b i ∼ N (0 , 1 / √ m ) , to get weights ( ω, a, b ) ∈ R m × R m × R m .
This ensures (cid:80) i
E [ a 2 i ]
= 1 = (cid:80)
i E [ b 2 i ] , so the initial response is bounded.
We train ( ω, a, b ) by gradient descent to minimize the empirical loss, using as prediction function F ω,a,b : x (cid:55)→ (cid:80) i ∈ [ m ] a i sin( ω i x ) + b i cos( ω i x ) .
2.2. Modified initialization and regularization
This minimal modification of the traditional multi-layer perceptron has had little success, because it does not seem to converge to approximations as good as the ReLU-based neural networks.
We show that by carefully altering the initialization procedure and regularization, this architecture is capable of distant extrapolation of periodic functions, beating the ReLUbased networks by several orders of magnitude.
To ensure diversity of the initial frequency weights, we initialize them at regular intervals ω i
= i δ , for i ∈ [ m ] ( δ = π/R in experiments, and there is no need for negative frequencies by symmetry).
We initialize the amplitude weights at zero ( a, b ) =
To promote sparsity in the amplitudes, and avoid interferences between very close but distinct frequencies, we use a non-convex sparsity-promoting penalty (with ε = 1e-20 a small constant for differentiability)
[Equation]
Note that we are not interested in finding the minimum to a different (regularized) loss function, we are merely trying to steer the trajectory of weights toward relatively sparser amplitudes during training.
For this reason, we instead use the time-dependent loss ( t, w, a, b ) (cid:55)→ L 0 ( w, a, b ) + e − t/τ R ( a, b ) (with τ = 10 in experiments), that is a weighted sum of the empirical loss L 0 and our regularizer R , but whose weight tends to zero during training, such that we truly converge to a minimum of the empirical loss alone when possible.
In the following sections, we explain the reasons that motivated these choices, starting with the need for a different periodicity-compatible activation function, and then how to fix the training problems that this change induces.
2.3. Limits of universal approximation on compacts, the need for sines
A historically strong argument in favor of the representation of functions as neural networks is the theorem of universal approximation on compacts (Cybenko, 1989; Barron, 1993; Leshno et al., 1993), stating that for any continuous function with a compact domain, there exists functions arbitrarily close to it that are representable as a neural network, provided only that the activation function is non-polynomial.
This property is not sufficient for periodic recovery, as the followng proposition shows.
Proposition
If a function f : R → R is representable as a multi-layer perceptron with
ReLU activations and finitely many weights, and if f is periodic, then f is constant.
Proof Such a function is piecewise linear with finitely many pieces (see e.g. Hanin and
Rolnick, 2019).
Thus there exists a constant B > 0 such that it is affine on [ B, + ∞ [ .
If it is both periodic and affine on that interval, then it must be constant on that interval, thus it is constant on R by periodicity.
This proposition does not constitute a problem in itself, because it remains possible that the learned function is not perfectly periodic, yet remains a very good approximation for a large number of periods after the observed interval, which would suffice in all practical applications.
However, our experiments demonstrate that such networks fail to extrapolate to even a single period outside the training interval.
2.4. Problems of the regular sine-based architecture
As presented in the previous section, the sine-based architecture is strikingly similar to a two-layer perceptron with only a modification in activation functions, and this is sufficient to fix the representation problem, all periodic functions are approximable on the whole real line by networks of this form.
However, experiments show that training this modified architecture as-is still fails to extrapolate outside the training interval.
We identify several problems and propose modifications to the training procedure to fix each of these issues.
Diagnostic of ill-convergence.
Figure 1 depicts the trajectories of neuron weights over time with two different initializations.
A weight ( w, a, b ) ∈
R m × 3 is depicted as a set of m points ( w i , a 2 i + b 2 i ) ∈ R × R + , together with the corresponding red trail for their evolution over time.
The green diamonds correspond to an optimal set of weights, exactly representing the signal.
As seen in the first case of Figure 1 (top left), independent initialization of frequency weights ( w ∈ R m ) with a normal distribution barely ever produces very high frequencies, which become harder for the network to learn (a long "distance" has to be traveled by the weights), a more diverse initialization appears preferable. This problem of absence of high frequencies at initialization is common to all network initializing features weights independently with too low a variance, typically reducing the variance with the number of neurons.
{+For instance, to deal with this issue, SIREN architectures using an initialization of w i ∼ U ( − (cid:112) 6 /m, + (cid:112) 6 /m ) with m ∈ N ∗ the number of neurons, have to resort to a hardcoded constant ω 0 = 
+}
(Sitzmann et al., 2020, section 3.2) in the activation x (cid:55)→ sin( ω 0 x ) , tuned to the considered applications and chosen network sizes, to ensure that the higher frequencies are present at initalization.
Additional regularization to avoid interferences.
The trajectories of gradient descent have several times been linked with a form of " ℓ 2 -inertia", see for instance Gunasekar et al.
Indeed in Figure 1, we see that many neurons have a small but non-zero amplitude.
While this may not be a problem in the training interval, it tends to cause unwanted wiggling outside the training interval, because these frequencies are usually not rationally linked and thus do not produce exactly periodic functions but only functions that look approximately periodic on the training interval, see for instance learned signals in Figure 3.
This issue can be fixed by promoting sparsity in the amplitudes, but some care must be taken to not introduce a phase bias by the choice of regularization.
Indeed, a change of phase in the signal corresponds to a rotation in the ( a, b ) plane.
Therefore, to avoid the phase-bias, the regularization chosen must depend on r 2 i = a 2 i + b 2 i but not a i and b i individually.
As shown in Fig.
1, an ℓ 1 regularization (cid:80) i
r
i does induce sparsity, but does not distinguish between two very similar neurons, which causes issues on large extrapolation intervals, which are of particular interest in these experiments (see loss values in Fig. 1).
To avoid these interferences, we use the non-convex penalty R ( a, b )
= − (cid:80)
i exp( − r i ) , which even for ω i =
ω j , forces either r i or r j to be null, where the ℓ 1 penalty allows any combination with constant sum. We emphasize that this problem is very different from the previous point.
While nonindependent initialization is used to ensure presence of high frequencies in the learned signal, this regularization ensures frequency localization after training.
The effect can be observed in Figure 1. On the top right panel, there is a diverse initialization, thus higher frequency components are present, but the support is spread out, many frequencies have non-zero amplitude. As a result, the training error L 0 is very low, but the extrapolation error Lremains high, because the learned function is not periodic. See Appendix B.1 for a discussion of why.
In contrast, on the bottom panels, the learned signal is much more localized in frequency, corresponding to the desired periodic behavior.
3. Experiments
Evaluation methods.
We use R = 10 as window size for our experiments.
For each experiment, we select a period T ∈ [0 , R ] , a T -periodic function f ∗ ∈ F T , a number of observed samples n ∈ N ∗ .
We then sample independently uniformly at random on [0 , R ] a collection of n samples ( x i ) i ∈ [ n ] ∈ [0 , R ] n .
Along with these inputs, algorithms are provided with the noiseless response ( y i =
f ∗ ( x i ))
i ∈ [ n ] ∈ R n .
After training, we evaluate the empirical error
L 0 : f
(cid:55)→ 1 n (cid:80) i
( f ( x i ) − y i ) 2 , and the k -windows expected error L k : f
(cid:55)→ E x
[( f ( x ) − f ∗ ( x )) 2 ] where x ∼ U (0 , k R ) , for various values of k ∈ N ∗ .
In similar settings in machine learning, the empirical error L 0 is referred to as "train loss", and the 1 -window expected error L 1 as the "test loss", for its underlying distribution corresponds to that of the samples.
However, the performance we are truly interested in is the k -windows expected error L k for large values of k (typically k ≥ 10 at the very least for simple signals), for which we can really claim that the periodicity assumption has been exploited.
Baseline architectures evaluated
We experiment with multi-layer perceptrons with
ReLU activations (( · ) + : x ∈ R (cid:55)→ max(0 , x )) .
For a number of hidden layers L ∈ N ∗ , and a width m ∈ N ∗ , we initialize independently at random θ
i ∼ N (0 , 1) for i ∈ [ m ] for the first layer, θ ki,j ∼ N (0 , 1) for ( i, j ) ∈ [ m ] ×
[ m ] for layer k ∈ { 1 , · · · , L − 1 } , and θ Li ∼ N (0 , 1 / √ m ) for i ∈ [ m ] for the last layer.
For an input x ∈ R , we set the initial activation z
= x , then recursively z k +1 i = max(0 , θ ki · z k ) for k < L , and use as prediction function F θ ( x ) =
(cid:80)
i θ L i z
L i ∈ R .
We then train the weights θ ∈ R m × ( R m × m )
L − 1 × R m by gradient descent (Adam optimizer, step size 1e-4) to minimize the empirical loss.
We also perform experiments with other non-linearities such as tanh , and the snake x
(cid:55)→ x + 1 a sin 2 ( ax ) activation (Ziyin et al., 2020) with a = 27 .
5 (authors recommend a ∈ [5 , 50] ).
Single-tone periodic recovery
We set ω ∗ = 2 and signal f ∗ : x (cid:55)→ cos( ω ∗ x + π/ 12) .
We use n = 1000 samples.
Figure 3 shows the performance of each method, along with the function learned.
All methods achieve reasonably low error on the training set, and comparable "test" error L 1 (which is almost identical because a thousand-point average on an interval of length 10 is a very good estimate).
However, the ReLU architecture fails to uncover the symmetry and produces linear extrapolations outside the training interval.
Multi-tone periodic recovery
With n = 1000 samples as previously, we set ω ∗ = 2 ,
K = 25 and a k = ( − 0 .
8) k for k < K , to construct the more complicated periodic target signal f ∗ : x
(cid:55)→ (cid:80) k<K a k cos(( k +
) ω ∗ x + π/ 12) .
The results are depicted in Figure 4.
The relative ordering of methods by reconstruction accuracy is essentially unchanged. The more complicated signal induces higher average errors even on the observed interval (the in-interval "test" error
L 1 is order of magnitudes larger than for the simple sine wave of Fig. 3), and the errors are even larger when more periods are observed. One interesting particularity in this experiment is that we observe increasing errors even for the regularized sine network (albeit at a different rate). This seems to be caused by the inexactitudes in the learned harmonics. Roughly speaking, we would want ω
k = k ω 1 essentially for a perfectly periodic signal, whereas the learned ω k ≈ k ω 1 yield small deviations from this periodic behavior which are nearly invisible in the first few periods but start to show when more periods are observed. Frequency estimation.
In Figure 5, we compare the frequency estimation capability of our method with periodogram maximization (Kootsookos, 1993) for n = 100 and n =samples on a window of size R = 10 , with independent identically distributed noise (normal distribution with mean zero and standard deviation σ for an amplitude of 1.0, yielding a signal-to-noise ratio of 1 / 2 σ 2 ).
For our method, we estimate the whole signal in 10iterations, then return the frequency ω k associated with the maximum final amplitude k = argmax
i ( a 2 i + b 2 i ) .
Note that the computational cost of the algorithms are hardly comparable, ours requires solving an optimization problem every time, whereas periodogram maximization is nearly-instantaneous here.
However this may still be useable in a regime where very few data points are available.
We depict the mean root squared error overindependent runs with true frequency sampled uniformly in [ ω 0 , 15] for each run, where ω 0 =
R/ 2 π ≈ 1 .
59 is the minimal frequency to ensure that at least one period has been observed.
Error bars depict the minimum and maximum value of the root squared error.
Regarding orders of magnitudes, each frequency estimation with Sine-NN in the context of
Figure 5 took around 13 minutes in our experiments for n = 100 samples and 20 minutes with n = 1000 samples, compared to under a second with periodogram maximizer.
We regard these performance considerations as outside the scope of this paper, proper evaluations in controlled environments should be conducted before any conclusions can be drawn.
We presented these experiments as simple counterexamples to conjectures in theoretical machine learning, we claim only that they demonstrate plausible usability for the purpose of theorists, not that this architecture and training procedure are usable as-is in realistic tasks.
4. Conclusion
We have shown with simple and well-defined examples that neural networks with sine activations, often disregarded in favor of more complicated solutions resembling ReLU activations, require only minimal modifications to yield trainable architectures with competitive performance on the task of periodic signal recovery.
While the direct applicability of this architecture to more realistic signals and partially-periodic datasets remains as future work, these experiments provide good sanity checks and baselines for architectures designed to learn periodic trends in larger settings in the future. Appendix A. Uniqueness of mean squared error minimizer Let us prove Equation 1 (reproduced below). argmin
[Equation]
The proof of unique minimizer is almost immediate on F T because it is a vector space.
The only difficulty in extending it to ∪ t ≤ R/ 2 F t is to avoid picking t , then f ∈ F t a minimizer and try to show t = T , which does not hold in general, because the sets ( F u ) u are not disjoint. Proof Let R ∈ R ∗ + . Let f ∗ ∈ F T be the target, with period T ∈ R ∗ + such that T ≤ R/(i.e. at least two periods have been observed).
Define L the mean squared error on [0 , R ] :
[Equation]
We will show that f ∗ is the unique minimizer of L in ∪ t ≤ R/ 2 F t .
First, note that L ( f ∗ ) = 0 , and L has only non-negative values, therefore f ∗ is a minimizer of L , and f ∗ ∈ F T ⊆ ∪ t ≤ R/ 2 F t because T ≤ R/ 2 by assumption.
It remains to show that f ∗ is the unique such minimizer.
For the first part, note that if f ∈ F T , and L ( f )
= 0 , then f = f ∗ , because ( f − f ∗ ) ∈ F T is a continuous T -periodic function null almost-everywhere in [0 , R ] , therefore null on [0 , R ] by continuity, thus null on R by T -periodicity since T ≤ R .
Let us proceed to show that all minimizers are in F T .
Let f ∈ ∪ t ≤ R/ 2 F t be such that f / ∈ F T , and let us show that L ( f ) ̸ = 0 . The assumption f / ∈ F T implies that there exists x ∈ R such that f ( x ) ̸ = f ( x + T ) . By continuity, there exists an open interval I ⊆ R containing x such that ∀ u ∈
I, f ( u ) ̸
= f ( u + T ) .
However, since we have f ∈ ∪ t ≤ R/ 2 F t , there exists t ∈ ]0 , R/ 2] such that f ∈ F t , therefore we can assume without loss of generality that x ∈ [0 , R/ 2] (otherwise let v = x − ⌊ x/t ⌋ t , satisfying v ∈ [0 , t [ ⊆ [0 , R/ 2[ , and observe f ( v ) =
f ( x ) ̸ = f ( x + T ) =
f ( v + T ) by t -periodicity of f ).
Observe now that ( x + T ) ≤ R because x ≤ R/ 2 and T ≤ R/ 2 .
Therefore (up to shrinkage of I to ensure I + T ⊆ [0 , R ] and I ∩ ( I + T ) = ∅ ), there is an open interval I of ]0 , R [
such that ∀ u ∈
I, f ( u + T ) ̸
= f ( u ) and ∀ u ∈ I, u + T ∈ [0 , R ] \ I .
To conclude, use the identity ( a − b ) 2 + ( a − c ) 2 ≥ 12 ( b − c ) 2 , which holds for all ( a, b, c ) ∈ R 3 , where a = f ∗ ( u ) =
f ∗ ( u + T ) by T -periodicity of f ∗ , to get
[Equation]
A
.1. One observed period is not sufficient to learn the period
Appendix B. Ablation study B.1. Effect of regularization on frequency localization
Learning a two-layer sine network representation for the function f ∗ : x (cid:55)→ sin( ω ∗ x + φ ) would be trivial if we had access to values on R , we would just compute its Fourier transform.
Since it is composed of a single sine wave in input-space, this would lead to a dirac in frequency-space, located at the sine frequency ω ∗ .
However, we have access only to values on the interval [0 , R ] , thus for all practical purposes, f ∗ is indistinguishable from the truncated function ( g : x (cid:55)→ 1 ( x ∈ [0 , R ]) sin( ω ∗ x + φ )) , identical to f ∗ on [0 , R ] , but zero outside.
The Fourier transform of g is a convolution of the Fourier transform of f ∗ with a cardinal sine filter corresponding to the truncation to the interval.
Thus, while the Fourier transform of f ∗ is perfectly localized in frequency because it corresponds to a periodic function in input-space, the Fourier transform of g is "spread out", with many non-zero amplitudes around ω ∗ corresponding to the convolution with the filter.
Without regularization, the network tends to learn g rather than f ∗ (both would lead to zero error, but the ℓ 2 norm of amplitudes for g is smaller than that for f ∗ ).
This can be observed in Figure 1 (top right) in the Fourier half-plane, and in Figure 7 in input-space: as the gap between frequencies at initialization shrinks, the learned function gets closer to g , with values of zero outside the training interval.