Paper:
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation networks, an improvement to modern neural networks in which each neuron learns the particular nonlinear activation function that it requires. We show that enabling each neuron with the ability to learn its own activation function results in a more compact network capable of achieving comperable, if not superior performance when compared to much larger networks. We also showcase the capability of a differential equation neuron to learn behaviors, such as oscillation, currently only obtainable by a large group of neurons. The ability of differential equation networks to essentially compress a large neural network, without loss of overall performance makes them suitable for on-device applications, where predictions must be computed locally. Our experimental evaluation of realworld and toy datasets show that differential equation networks outperform fixed activatoin networks in several areas.

Review:
This paper proposes the use of structured activations based on ordinary differential equations, as an activation in neural network architectures.
There are validations that the approach discovers different activations, and comparisons to a variety of other architectures with fixed activations. In general, I would like to see additional contextualization of your work with other approaches which learn activation functions. How does your approach differ from max-out and Agostinelli? References to Network in Network, Lin et al and even Hypernetworks, Ha et al would also be helpful. Some of these papers are cited, but only the comparison to Ramachadran et. al. is directly discussed and the methodological difference there is not true for the case of Maxout.
The paper indicates that the ODE should be solved before beginning to use the network. Should this process be performed once per network design? Once per dataset? What happens if instead of solving f1 and f2, these are set randomly? How is this actually solved in the case of the MNIST convnet?
The experiments were useful in demonstrating the proposed method. However, some discussion and comparison to other learned activation functions would be helpful (for instance, one could perform similar experiments as in Maxout). Performance on larger datasets, as seen in Swish, would make the results more compelling. 
The MNIST experiments shown are also pretty far from standard baselines. See for example the benchmark performance in Maxout, which also references an architecture from Jarrett et. al., 2009 which is quite similar to the baseline architecture, but ~.5% error. It isn't necessary to get the absolute best performance with a new activation, just show that the proposed doesn't actively hurt and enables new interpretation or direction. But as it stands, it isn't possible to tell from the experiments if the proposed method has serious limitations, because the baselines on MNIST are below where they should be.
In general, a larger test suite that compares on more standard datasets is necessary to really prove out this idea, or see if there are problems with cases on larger datasets. CIFAR10 at a minimum would be a key addition as well as other datasets (besides the diabetes dataset shown) where there can be direct comparison to existing work. Currently, only MNIST is filling that role. Many of the cited / compared work (PReLU, Swish, LReLU, SELU) has a broad suite of benchmarks, all on datasets with existing numbers tuned by the authors of respective past papers.
What are the downsides of this method? Tradeoffs in memory and training time should be discussed in detail if application to low power hardware is a real application area, perhaps along with an inclusion of the time / effort required for solving the ODEs in Maple. A difference of 2x+ in parameter count may not be a difference if the computational time is much worse. Can you consider a case where "normal" architectures don't fit in memory, but one based on ODE activations will? The paper directly discusses mobile and low footprint deployments, but without discussion of the computational overhead and complexity it is speculative, especially when there are also numerous methods for compressing or distilling very large architectures to much smaller sizes, as well as small models which directly achieve high performance. A few relevant methods are linked below.
MobileNets - https://arxiv.org/abs/1704.04861
ENet - https://arxiv.org/abs/1606.02147
In the Network Compression section, the paper fails to discuss a number of successful foundational and modern network compression techniques that would improve the argument, including: 
Optimal brain damage  - “removing unimportant weights can actually improve performance” - https://papers.nips.cc/paper/250-optimal-brain-damage
Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding -  pruned state-of-the-art CNN models with no loss of accuracy - https://arxiv.org/abs/1510.00149
Bayesian Compression for Deep Learning - https://arxiv.org/abs/1705.08665 
Practical Variational Inference for Neural Networks - http://papers.nips.cc/paper/4329-practical-variational-inference-for-neural-networks 
There are repeated claims of first use of ODE in neural networks, which is frankly false. Though the specific use proposed here may be new, neural networks and ODEs have been used together many times. Clarifying what particular usage of ODE inside this setting is novel would be much better than a broad claim such as "While the presented model, algorithms and results in this paper are the first application of ODEs in neural networks...". Much of this work has been about controlling or solving ODEs, but particularly the setting of Meade Jr. et. al. strongly resembles a "neuron" in this architecture, so a discussion of the relevant differences would be useful. In addition Neural Ordinary Differential Equations allows the end-to-end training of ODEs in larger models, which also closely resembles the use of ODEs here.
Artificial Neural Networks for Solving Ordinary and Partial Differential Equations - https://ieeexplore.ieee.org/document/712178/
Solution of nonlinear ordinary differential equations by feedforward neural networks - https://www.sciencedirect.com/science/article/pii/089571779400160X
Neural Ordinary Differential Equations - https://arxiv.org/abs/1806.07366
Overall, a stronger focus on empirical results on comparable datasets would be beneficial, especially larger tasks. If larger tasks are not possible, a description of what it may take to "scale up" would be useful. The written focus on novelty detracts from the presentation, and a discussion of neural ODE methods (whether acting as activations, or solvers) would serve as good background material. If compute / performance in low footprints or mobile hardware is a focus, it should be described and tested. If lower parameter count is a perceived benefit, a more direct exploration and discussion of parameter count settings for this architecture and baselines would also be useful. Particularly, hyperparameters become very important in small architectures, so "Dropout probability,
batch size, epochs and learning rate were consistent across all networks" is not a positive (presuming the authors have likely tuned toward their own architecture). Baselines should be given equal treatment and tuning in order to compare "best-on-best" performance.
The description of universal approximation, visualization of the adaptivity of the method, and background are all very nice. My concerns come primarily to relation to prior and relevant work, strength of relevant experimentation, and claims of application and novelty / "first past the post".
——
Minor Nitpicks: 
Page 1: In the sentence - “researchers have introduced highly effective network structures such as convolutional neural networks”, it seems inconsistent to cite a foundational paper for CNNs and not RNNs. 
Page 2: It seems like there is a word missing here - “The size of a neural network is delineated its number of hidden neurons and their interconnections, which together determine the network’s complexity”
There seems to be a missing word in “3.3 DIFEN IS UNIVERSAL APPROXIMATOR” . 
Numerous spelling errors should be corrected - 
3.1 differentiatial
4.1 challnging
Figure 1 - fucntion