Reproducibility Report: Neural Networks Fail to Learn Periodic Functions and How to Fix It
Reproducibility SummaryScope of Reproducibility
Neural Networks Fail to Learn Periodic Functions and How to Fix It [15] demonstrates experimentally that standardactivations such as ReLU , tanh , sigmoid and their variants all fail to learn to extrapolate simple periodic functions.
The original paper goes on to propose a new activation, which is named the snake function.
The central claims of the paper are two-fold.
The properties of the activation functions are carried over to theneural networks.
A tanh network will be smooth and extrapolates to a constant function, while ReLU extrapolates in alinear way.
Standard neural networks with conventional activation functions are insufﬁcient for extrapolating periodicfunctions.
The proposed activation function manages to learn periodic functions while being able to optimize aswell as conventional activation functions.
While both experimental proof and theoretical justiﬁcations are provided forthe claims, we shall only be concerned with testing the claims via experimental means.
Methodology
While the author was contacted to clarify certain difﬁculties, the reproduction of all experiments was completed usingonly the information provided in the original paper itself.
With one exception, the link to all datasets used was alsoprovided in the paper itself.
This allowed us to implement most experiments from scratch.
Results
We were able to successfully replicate experiments supporting the central claim of the paper, that the proposed snakenon-linearity can learn periodic functions.
We also analyze the suitability of the snake activation for other tasks likegenerative modeling and sentiment analysis.
What was easy
Many experiments included descriptions of the neural network architectures and graphs showcasing performance,giving us a clear benchmark to compare our results against.
Links to datasets for all experiments, barring one, were alsoincluded in the paper itself.
What was difﬁcult
Data for the human body temperature experiment was not available.
Proper implementation details were not given forinitializing the weights in neural networks with snake and using snake with RNNs.
Communication with original authors
One author, Liu Ziyin was contacted to provide the dataset used for the human body temperature experiment, elaborateupon the implementation of variance correction during weight initialization and provide his implementation of RNNusing snake .
He provided the GitHub link to his code for the human body temperature, market index, and extrapolationexperiments.
He also provided an explanation on how to implement variance correction.
While the code for the RNNimplementation using the snake activation was not made public, he provided a screenshot of the same. Submitted to
Introduction
Deep neural networks are playing an increasing prominent role in ﬁelds as diverse as computer vision [4], speechrecognition [2], and language modeling [5].
However, while neural networks are excellent tools for interpolatingbetween existing data, standard versions of these networks are not suited for extrapolation beyond the training range.
This causes them to struggle at making predictions in problems with a periodic component.
Previous attempts at addressing neural networks’ inability to learn periodic functions have included using periodicactivation functions [11, 14].
For example, using sin( x ) as the activation function for implicit neural representationshas been successful at representing complex natural signals and their derivatives [12].
However in more general cases,experimental results suggest that using sin as the activation function cannot compete against ReLU -based activationfunctions [10, 6, 1, 13] on standard tasks [7].
The original paper: (1) studies the extrapolation properties of a neural network beyond a bounded region; (2) showsthat neural networks with standard activation functions are insufﬁcient to learn periodic functions outside the boundedregion where data points are present; (3) proposes a solution for this problem in the form of a novel activation functionand its variants, and showcases its performance on toy examples and real-world tasks.
We have tested the claims madein the original paper, replicating both the experiments displaying the failure of standard activation functions to learnperiodic functions as well as the results of the novel activation function on toy and real-world tasks.
We have alsoconducted experiments of our own to understand how viable the proposed activation function is at replacing existingstandards such as ReLU and tanh .
Scope of reproducibility
The authors make two key claims:
• Standard neural networks with standard activation functions are insufﬁcient to learn periodic functions outside thebounded region where data points are present.•
The proposed novel activation function can learn periodic functions while maintaining the favorable optimizationproperty of the ReLU -based activations.
The novel activation is dubbed “ snake ”:
[Equation]
where a is treated as a ﬁxed parameter in initial experiments, and as a learnable parameter in a few experiments.
Snake is shown to outperform standard activation functions ReLU , tanh , LeakyReLU
[6], as well as more recentlyproposed functions such as swish [10], and sin [12, 7].
Due to the broad and far-reaching consequences of the two claims, the original paper supports them via both theoreticaljustiﬁcation and an extensive list of experiments which range from testing performance on toy datasets to real worldapplications.
We have exhaustively replicated the original list of experiments, and have conducted a few additionalexperiments of our own, using the proposed activation function in a Deep Convolutional Generative Adversarial Network (DCGAN) to generate images of handwritten digits and in a Long Short Term Memory (LSTM) network forsentiment analysis.
Methodology
The code used by the authors had not been made public at the time we started working on re-implementing the paper.
That meant we reproduced all the results in the paper from scratch relying on the descriptions of the neural networkarchitecture and a link or description of the dataset.
The descriptions were brief but sufﬁcient such as “feedforwardneural network with 2 hidden layers, both with 64 neurons” for the Body Temperature Prediction experiment and “4-layerfeedforward network with 1 → 64 → 64 → 1 hidden neurons” for the Financial Data Prediction experiment.
In thecase of experiments that utilized large standard networks such as ResNet18, the PyTorch library implementation of themodel was used, with snake substituted in place of the default activation functions.
Besides the model implementations,we were also required to make a a learnable parameter in snake for a few experiments.
3.1 Model descriptions
Models used in the original paper included fully-connected, feed-forward neural networks with different architecturesfor the various experiments.
Larger standard models such as ResNet18 were also used.
The authors of the originalpaper had initially not made their code available and we had to implement most models ourselves.
3.2 Datasets
The data used in the extrapolation experiments are directly sampled from periodic functions such as sin( x ) .
Someexperiments dealt with standard datasets such MNIST and CIFAR-10.
Data for the real-life datasets had to bedownloaded:
• Daily data from 1995-1-1 to 2020-1-31 of Wilshire 5000 Total Market Full Cap Index: Downloaded from link
provided in the original paper:
https://www.wilshire.com/indexes
• Average weekly temperature evolution in Minamitorishima, an island south of Tokyo (longitude: 153.98,latitude: 24.28) after April 2008:
Downloaded from link provided in the original paper: https://join.
juelich .de/ access
• Patient body temperature: Made available by the authors upon request
• IMDB Reviews Dataset used for our additional sentiment analysis experiment: Downloaded from https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews
3.3 Hyperparameters
Different experiments included varying levels of detail with respect to hyperparameters.
Many experiments provided anoverview of the neural network architecture (e.g. “4-layer fully connected neural network”) but not other hyperparameters, such as batch size, loss function, or learning rate.
In cases where information was missing, assumptions had to bemade, with some trial-and-error required to obtain a close approximation of the original result.
This trial-and-errorinvolved a grid search over the architecture (number of layers, number of neurons in each layer), number of epochs(100 to 5000), batch size (16 to 512), optimizer (Adam, SGD, RMSProp), learning rate (0.001 to 0.1) and value of a innetworks with the snake activation (1 to 30).
3.4 Experimental setup
The entire codebase has been uploaded to GitHub and is publicly available:
https://github.com/mayurak47/
Reproducibility_Challenge .
The experiments were run locally as well as on GPU enabled sessions on Google
Colab.
All the models and experiments were coded using the PyTorch library.
3.5 Computational requirements
Many of the experiments, particularly those relating to regressing different functions and datasets, could be run locallyon a MacBook Air with an Intel i5 CPU and 8 GB of RAM, not requiring more than a few minutes to train.
The moredemanding experiments required the use of GPUs.
Training a ResNet18 on CIFAR-10 with six activation functions for100 epochs took roughly 12 hours on a Tesla T4 GPU on Google Colab.
Our additional experiments on training a GANand an LSTM required roughly 2 hours each on the same hardware.
Results
Wherever possible, the claims of the original papers were tested and in each case, we were able to reproduce the originalresults.
The list of experiments that we reproduced is listed below.
4.1 Extrapolation experiments on analytic functions
Neural networks with a single hidden layer consisting of 512 neurons are trained on data sampled from four differentanalytic functions using the ReLU and tanh activation functions.
The training data is obtained by sampling from [-5, 5]with a gap in [-1, 1].
It is observed in Fig.
1 that the extrapolation of neural networks depends on the activation functionused.
When ReLU is used, the extrapolation diverges to ±∞ .
When tanh is used, the extrapolation levels off.
Theauthors formally prove these observations and conclude that neural networks using these activation functions cannotlearn to extrapolate periodic functions.
4.2 Applicability of proposed method
It is ﬁrst demonstrated that the snake activation function is easier to optimize than other commonly used baselineperiodic activation functions like sin( x ) and x + sin( x ) .
Fully-connected neural networks with 3 hidden layers (512neurons each) are trained on the MNIST dataset.
This is a 10-way classiﬁcation problem, and the training cross-entropylosses for the different networks can be observed in Fig.
2, with the snake network achieving the lowest training loss.
It is then shown that snake is able to regress the periodic function sin( x ) .
While all activation functions learn thetraining data (Fig. 1), only snake is able to capture the periodic behavior of sin( x ) (Fig. 3).
The extrapolation divergesfrom the underlying sin function due to the limited training data used.
4.3 Applications
[Figure]
Multiple experiments are conducted to illustrate the performance of snake on a range of tasks.
[Figure]
different non-
ResNet18 [3], with 10M parameters, is trained on the CIFAR-10 dataset.
Thisis a 10-way image classiﬁcation task.
The ReLU layers in the architectureare replaced with the speciﬁed activation, and the network is trained for 100epochs.
The LaProp optimizer 1 [16] is used; the learning rate is 4 × 10 − 4 forthe ﬁrst 50 epochs and 4 × 10 − 5 for the next 50.
A test accuracy of 93-94% isachieved by the snake network (Fig. 4), in line with that of the other standardactivation functions.
This suggests that snake is suitable for large-scale imageclassiﬁcation problems, and may be used as an straightforward alternative toother activation functions.
The core utility of snake is shown via two real-life problems.
The two tasksare predicting the evolution of temperature in Minamitorishima island in Japan(Fig. 5), and the modeling the body temperature of a patient (Fig. 6).
Thearchitectures used are 1 → 100 → 100 → 1 and 1 → 64 → 64 → 1respectively, as in the original paper.
In the Minamitorishima experiment,the parameters a were made learnable; in the body temperature experiment,a = 30 .
In both cases, snake is the only activation function that makesmeaningful extrapolation and predictions.
It can also be seen in Fig.
5b thatsnake is the only activation function that is able to learn the training data - theother non-linearities are unable to ﬁt the training points, irrespective of thenumber of epochs the models are trained for.
[Figure]
The snake network correctly learns the periodicity of the atmospheric temperature dataset, even though the amplitude isslightly off, and correctly infers that body temperature is roughly 37°C.
Another regression problem the authors used to demonstrate the working of snake is that of ﬁnancial data prediction(Fig. 7).
The data used is from the Wilshire 5000 Total Market Full Cap Index, considered representative of theworldwide economic trend.
The snake network ( 1 → 64 → 64 → 1 , a = 30 ), which was trained using data from1995 to 2020-1-31, before COVID-19 impacted the world economy, predicted an economic slowdown in 2020.
Thismight be due to the cyclic nature of world markets, which the model was able to capture.
As in the previous regressionexperiments, snake performs better than conventional non-linearities (Table 1).
[Table]
The authors, in an additional experiment described inthe appendix, use this dataset to gain insights into howthe snake activation function learns (Fig.
Observingthe predictions made at various points in the trainingprocess, we notice that at ﬁrst, the features learned aremostly linear, low frequency features are then learned, andhigh-frequency features are learned at the later stages oftraining.
The performance of a snake feedforward network (two hidden layers of 64 neurons each, a = 30 ) and a recurrent neuralnetwork (single recurrent layer, 64 features in hidden state), typically used for time-series prediction, are compared in
Fig.
The task is to learn the function sin(0 .
1 x ) , with Gaussian noise σ added, for T = 300 timesteps.
The ﬁrst 200are used for training, while the last 100 are used for testing.
It is seen that because of the noisy training data, even the predictions of the RNN are noisy, with a high generalizationloss.
The feedforward network, on the other hand, almost perfectly learns the underlying function with the rightfrequency and amplitude.
Further, RNNs learn by backpropagation through time (BPTT), which has a prohibitively high computation cost, andcan result in the exploding/vanishing gradient problem [8].
As a result the time taken by the snake network to regressthe function is roughly 2 orders of magnitude lower than the time taken by the RNN (Fig. 10).
This suggests that snakenetworks may be more effective in modeling data that is known beforehand to be periodic in nature.
4.4 Effect of a
In a series of experiments, the authors depict the effect the parameter a has on the learning process.
We reproduce oneof these experiments for brevity.
Simple neural networks ( 1 → 64 → 64 → 1 ) are trained on the sinusoidal functionsin( x ) + sin(4 x ) / 4 .
It is seen in Fig.11 that larger a encourages the model to learn features with higher frequency.
With a = 1 , the higherfrequency modulation is considered noise, while the a = 16 model learns both the signals.
This tendency can be takeninto account while working with data known to be periodic, with a well-chosen a speeding up training.
4.5 Results beyond original paper
The original paper demonstrated the ability of neural networks with the snake activation function to learn periodicfunctions and that the performance on everyday tasks like image classiﬁcation is similar to that of conventional activationfunctions.
We extend this study to more sub-ﬁelds of deep learning.
We train a deep convolutional generative adversarial network (DCGAN) 2 [9] to generate samples of the MNIST dataset.
All the activations in the generator and discriminator sub-networks are replaced with the speciﬁed non-linearity.
We seethat while the initial training is slow for the snake GAN (Fig. 13a), it eventually generates realistic samples (Fig.
12a),which are qualitatively indistinguishable from those output by a typical GAN using the LeakyReLU non-linearity (Fig.12b).
a was a learnable parameter in this experiment.
[Figure] [Figure]
Finally, we use the snake activation function in a Long Short Term Memory (LSTM) network for sentiment analysis 3on the IMDB movie reviews dataset.
This is a binary classiﬁcation problem, attempting to predict whether a moviereview is positive or negative.
The typical tanh activation used to output the value h t = o t ∗ tanh( C t ) in an LSTMis replaced by the snake activation, so that h t = o t ∗ snake( C t ) .
We observed that the snake LSTM network did notperform very well in this task (Fig. 14) and convergence was much more gradual.
A single epoch of training the snake
LSTM took twice as long as training the tanh LSTM.
Also, in many cases, the snake network got stuck in local minima,necessitating a restart of training.
A possible explanation for this is that the snake function is not bounded like tanh , causing an increase in the values ofh t .
The results of the experiment do not mean that snake cannot be used in sequence models, only that the application isnot as straightforward as in the previous experiments, and further modiﬁcations in the architectures might be necessary.
[Figure]
Discussion
As the authors had not initially made their code available and only included brief descriptions of the network architecturesused in their experiments, exact replication of their experimental results was not possible.
However, the qualitativenature of the paper meant that only the relative performance of snake in comparison to other activation functions on thespeciﬁed problems was of interest, as opposed to the exact architectural details or loss values achieved.
For example,the losses observed in Table 1 and Fig.
5b are orders of magnitude different from those in the original paper, likely dueto varying normalization techniques and hyperparameters, even though the overall results observed in Fig.
5a and Fig.
7are similar to those observed in the original paper.
We were able to uphold the claim that neural networks with standardactivation functions are insufﬁcient to learn periodic functions outside the training range.
We were also able to verifythat the proposed activation function performs as well as standard activation functions, ReLU , tanh , LeakyReLU , overa wide range of tasks (with the exception of the LSTM experiment), by replicating the experiments in the original paperand conducting some additional ones ourselves.
Future work could focus upon providing theoretical justiﬁcations forthe behavior of snake and developing more suitable optimization algorithms.
5.1 What was easy
A detailed description of the neural network architectures used for experiments such as training on the MNIST datasetand human body temperature was provided, allowing us to replicate the experiments closely.
Links to datasets forall experiments, barring one, were also included in the paper itself.
An extensive appendix sections listed additionalexperiments comparing the performance of snake with different a .
Every experiment was supported by graphsshowcasing the performance of snake with other activation functions, giving us a clear metric against which we couldcompare the results of our reproductions.
5.2 What was difﬁcult
The original source code was not provided initially and we had to rely on the descriptions of architectures andhyperparameters (which were absent in many cases) and educated guesswork while attempting to replicate the results.
Data for the human body temperature experiment was not available.
Theoretical justiﬁcation for variance correctionand the results of this variance correction using ResNet101 on CIFAR-10 were provided, but implementation detailswere not included.
The section on Comparison with RNN on Regressing a Simple Periodic Function simply states thatsnake was deployed on a feedforward network, without any additional details of the hyperparameters used.
The datasetfor the experiment had to be inferred from the graphs of the results, and since white noise had been added to the data,exact replication of the experimental setup was not possible.
5.3 Communication with original authors
Liu Ziyin, one of the authors, was contacted to provide the dataset used for the human body temperature experiment,elaborate upon the implementation of variance correction and provide the implementation of RNNs using snake .
On being contacted, he provided the GitHub link to his code 4 for the human body temperature, market index, andextrapolation experiments.
He also provided an explanation on how to implement variance correction.
While thecode for the RNN implementation using snake activation was not made public, he provided a screenshot of the same.
The provided code was incomplete and not fully documented but was nonetheless valuable in giving us a rough ideaabout the hyperparameters used.
The provided repository also contains the human body temperature dataset within thecodebase, which is not available in the original paper.