Paper:
While much of the work in the design of convolutional networks over the last five years has revolved around the empirical investigation of the importance of depth, filter sizes, and number of feature channels, recent studies have shown that branching, i.e., splitting the computation along parallel but distinct threads and then aggregating their outputs, represents a new promising dimension for significant improvements in performance. To combat the complexity of design choices in multi-branch architectures, prior work has adopted simple strategies, such as a fixed branching factor, the same input being fed to all parallel branches, and an additive combination of the outputs produced by all branches at aggregation points. In this work we remove these predefined choices and propose an algorithm to learn the connections between branches in the network. Instead of being chosen a priori by the human designer, the multi-branch connectivity is learned simultaneously with the weights of the network by optimizing a single loss function defined with respect to the end task. We demonstrate our approach on the problem of multi-class image classification using four different datasets where it yields consistently higher accuracy compared to the state-of-the-art "ResNeXt" multi-branch network given the same learning capacity.
While much of the work in the design of convolutional networks over the last 1 five years has revolved around the empirical investigation of the importance of 2 depth, filter sizes, and number of feature channels, recent studies have shown that 3 branching, i.e., splitting the computation along parallel but distinct threads and then 4 aggregating their outputs, represents a new promising dimension for significant 5 improvements in performance. To combat the complexity of design choices in 6 multi-branch architectures, prior work has adopted simple strategies, such as a 7 fixed branching factor, the same input being fed to all parallel branches, and an 8 additive combination of the outputs produced by all branches at aggregation points.

Review:
The authors extend the ResNeXt architecture. They substitute the simple add operation with a selection operation for each input in the residual module. The selection of the inputs happens through gate weights, which are sampled at train time. At test time, the gates with the highest values are kept on, while the other ones are shut. The authors fix the number of the allowed gates to K out of C possible inputs (C is the multi-branch factor in the ResNeXt modules). They show results on CIFAR-100 and ImageNet (as well as mini ImageNet). They ablate the choice of K, the binary nature of the gate weights.
Pros:
(+) The paper is well written and the method is well explained
(+) The authors ablate and experiment on large scale datasets
Cons:
(-) The proposed method is a simple extension of ResNeXt 
(-) The gains are reasonable, yet not SOTA, and come at a price of more complex training protocols (see below)
(-) Generalization to other tasks not shown
The authors do a great job walking us through the formulation and intutition of their proposed approach. They describe their training procedure and their sampling approach for the gate weights. However, the training protocol gets complicated with the introduction of gate weights. In order to train the gate weights along with the network parameters, the authors need to train the parameters jointly followed by the training of only the network parameters while keeping the gates frozen. This makes training of such networks cumbersome.
In addition, the authors report a loss in performance when the gates are not discretized to {0,1}. This means that a liner combination with the real-valued learned gate parameters is suboptimal. Could this be a result of suboptimal, possibly compromised training? 
While the CIFAR-100 results look promising, the ImageNet-1k results are less impressive. The gains from introducing gate weights in the input of the residual modules vanish when increasing the network size. 
Last, the impact of ResNeXt/ResNet lies in their ability to generalize to other tasks. Have the authors experimented with other tasks, e.g. object detection, to verify that their approach leads to better performance in a more diverse set of problems?