  We propose a complexity measure of a neural network mapping function based on
the diversity of the set of tangent spaces from different inputs. Treating each
tangent space as a linear PAC concept we use an entropy-based measure of the
bundle of concepts in order to estimate the conceptual capacity of the network.
The theoretical maximal capacity of a ReLU network is equivalent to the number
of its neurons. In practice however, due to correlations between neuron
activities within the network, the actual capacity can be remarkably small,
even for very big networks. Empirical evaluations show that this new measure is
correlated with the complexity of the mapping function and thus the
generalisation capabilities of the corresponding network. It captures the
effective, as oppose to the theoretical, complexity of the network function. We
also showcase some uses of the proposed measure for analysis and comparison of
trained neural network models.
