  Weight decay is a widely used technique for training Deep Neural
Networks(DNN). It greatly affects generalization performance but the underlying
mechanisms are not fully understood. Recent works show that for layers followed
by normalizations, weight decay mainly affects the effective learning rate.
However, despite normalizations have been extensively adopted in modern DNNs,
layers such as the final fully-connected layer do not satisfy this
precondition. For these layers, the effects of weight decay are still unclear.
In this paper, we comprehensively investigate the mechanisms of weight decay
and find that except for influencing effective learning rate, weight decay has
another distinct mechanism that is equally important: affecting generalization
performance by controlling cross-boundary risk. These two mechanisms together
give a more comprehensive explanation for the effects of weight decay. Based on
this discovery, we propose a new training method called FixNorm, which discards
weight decay and directly controls the two mechanisms. We also propose a simple
yet effective method to tune hyperparameters of FixNorm, which can find
near-optimal solutions in a few trials. On ImageNet classification task,
training EfficientNet-B0 with FixNorm achieves 77.7%, which outperforms the
original baseline by a clear margin. Surprisingly, when scaling MobileNetV2 to
the same FLOPS and applying the same tricks with EfficientNet-B0, training with
FixNorm achieves 77.4%, which is only 0.3% lower. A series of SOTA results show
the importance of well-tuned training procedures, and further verify the
effectiveness of our approach. We set up more well-tuned baselines using
FixNorm, to facilitate fair comparisons in the community.
