Paper:
A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.

Review:
First, I believe that the acknowledgements in the manuscript give identifying information which could stand in conflict with a double blind review process. I’ll leave it to the area chairs/program chairs to make a decision on this. The following review will be contingent on the fact that the authors did not break the submission rules.
This paper aims to give new insights into deep neural networks by presenting a number of approaches to analyse the linear regions in such networks. The authors define a linear region around a point x* as the intersection between a number of half spaces that are defined through linear approximations of a DNN (tangents) around that point x*. The authors show that points within these regions can be found using convex optimization with a number of linear constraints that are equal or less than the number of nodes in a DNN. In experiments with a fully connected network the authors analyse different properties of these linear regions: (1) How big is the biggest sphere that we can fit in a linear region? (2) How much do the hyperplanes that define a region correlate with each other? (3) How reliably does a linear region represent a single class? And (4), How does a linear region interact with neighbouring regions? In their presentation, the authors focus on comparing these properties between models that were either trained without regularisation, with batch normalisation, or with dropout and with different learning rates. This allows them to draw insightful conclusions about the difference between linear regions in these models. The authors hope that their work will enable new ways of analysing DDNs that will inspire new architectures and optimization techniques.
I vote to accept this paper. The authors present a large array of methods to analyse the linear regions of DNNs. Their insights into the differences of BN and Dropout are useful (figure 1 & 2) and sensible (figure 3). The implications of linear regions on adversarial robustness can have an impact in the future. Because the paper relies on geometrical reasoning, I wished there would be more visualisations that guide the reader.
Here are a number of comments and questions that I have on the manuscript:
- Figure 1 Top: What do the different colour represent in the linear regions plot?
- Section 2.1: maybe add a toy graph that visualises the depth-wise ‘exclusion’ process of feasible “neighbours” of x*?
- Eq. (2) & (3): Explain where these equations come from.
- Sec 3.2, first sentence. The authors claim that inspheres of linear regions are highly relate to the expressivity of DNN. Can they elaborate on that claim? Is this claim a result of their experiments?
- What is the relationship between the number of constraints in eq. (5) and the radius of an insphere? Does the insphere size decrease with more constraints? What implications would that have on deeper networks than the one that was presented?
- Why should distortion be a good measure of the size of a linear region?
- The authors claim that it is expensive to run their approach, and that they will aim to improve speed in the future. Can the authors give a more concrete example of runtimes in their current approach?