Paper:
Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectified linear function-like activations (e.g. ReLU) as building blocks. Previous work pointed out that learning feature channels with equal magnitudes is important for a CNN to achieve good generalization ability. However, the above "Norm+ReLU-like" basic block often learns inhibited channels that has small magnitude (i.e. contributes little to the feature representation), impeding both learning and generalization ability of CNNs. This problem is seldom explored in the literature. To mitigate the inhibited channels and encourage channels to contribute equally to the feature representation, we propose a new building block, Channel Equilibrium (CE), which is able to prevent inhibited channels in both experiments and theory. CE has several appealing properties. First, CE can be stacked after many different normalization methods such as BN and Group Normalization (GN), as well as integrated into many advanced CNN architectures such as ResNet and MobileNet V2 to form a series of CE networks (CENets), outperforming existing network architectures. Second, CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. Third, extensive experiments show that CE achieves state-of-the-art results on various challenging benchmarks such as ImageNet and COCO. The models and codes will be released.

Review:
This paper proposes a new block that gives diminishing 
Pros)
(+) This paper is well-written and the idea looks interesting.
(+) The provided experiments look quite extensive. I like the experiments with both heavy and light network architectures to show the effectiveness of the proposed method.
(+) The authors provided plausible and sufficient backups for the claim.
(+) Connection with Nash equilibrium seems to be slightly overclaimed, but the attempt is novel and interesting.
Cons)
(-) The overall paper flow could be organized much better. I think some of the materials in the appendix needs to be placed in the main paper.
(-) Some of the notations in Section 3 need to be rewritten for better clarity. 
(-) The terminology such as channel-level sparsity, channel equalization, and equivalent lambda looks vague. I recommend the authors define them for clarity.
Comments) 
- I think the relationship between diminishing the sparsity on channels and improving the evaluation performance should be more addressed. If we change ReLU after Conv with other nonlinear functions such as ELU or PReLU, then the output hardly becomes zero, but as you know the performance usually gets worse. Then, how do you explain the necessity of reducing sparsities? 
- How did you measure the sparsity in a plain network in Figure 1 and Figure 3? Did you pick a single layer or gather all the layers' sparsity? Please specify the way of measuring it.  
- How did you plot the graphs in Figure 3.(a) and 3.(b). Did you pick a single channel randomly?
- To support the authors' claim, it would be better to show the sparsity ratio for a model (ResNet18, MobileNetV2, and so on) which shows better performance than each original one. 
- How did you determine the transformation of F in the AII branch? Please provide any intuitions why the authors choose the transformation like in eq.(9). 
- If CE-block actually conducts well right after a BN layer, then why CE block is attached only after the final BN layer in a bottleneck module? It would be better to provide any results by doing some studies when dealing with the intermediate BNs in a bottleneck module.
- I am wondering whether adaptive normalization techniques such as Instance-Batchnormalziation or Switchable Normalization methods could also give less sparsified features. Please clarify this by comparing with the proposed method.
- It seems that CE would have some extra computational costs compared with SE's. Please clarify why CE-module does have small computational costs.
- Please specify what MobileNetV2 has used. Looks like MobileNetV2x1.0 would be used.
- Why do you think the Top-5 accuracy of MobileNetV2 has not been improved?
- Can your method combine with ResNet with SE such as ResNet50SE? 
- Please clarify the way of measuring the sparsity ratio in Figure 3d.   Why did you consider the sparseness of a feature by measuring the gamma in Figure 4? There may exist a channel that contains large magnitude values then gamma cannot make the output close to zero.
About overall rating)
The theoretical backups for the authors' claim look sound, and all the experiments including the performance improvement with several models seem to support the effectiveness of the idea very well. Specifically, I like all the analyses and the connection with Nash equilibrium, which are very intuitive and may provide further insights to researchers in this field.  I think this paper is clearly above the standard of ICLR. If the authors could address all the concerns above and refine the paper with better readability, then I could increase the score.