We show that for neural network functions that have width less or equal to the input dimension all connected components of decision regions are unbounded. The result holds for continuous and strictly monotonic activation functions as well as for ReLU activation. This complements recent results on approximation capabilities of Hanin and Sellke (2017) and connectivity of decision regions of Nguyen et al. (2018) for such narrow neural networks. Further, we give an example that negatively answers the question posed in Nguyen et al. (2018) whether one of their main results still holds for ReLU activation. Our results are illustrated by means of numerical experiments.
REF shows that the decision regions are open for DNNs with width less than or equal to input dimensions.
On decision regions of narrow deep neural networks
ArXiv
ArXiv
Computer Science Mathematics