Convergence Analyses on On-Line Weight Noise Injection-Based Training Algorithms for MLPs
Injecting weight noise during training is a simple technique that has been proposed for almost two decades. However, little is known about its convergence behavior. This paper studies the convergence of two weight noise injection-based training algorithms, multiplicative weight noise injection with weight decay and additive weight noise injection with weight decay. We consider that they are applied to multilayer perceptrons either with linear or sigmoid output nodes. Let w(t) be the weight vector, let V(w) be the corresponding objective function of the training algorithm, let alpha > 0 be the weight decay constant, and let mu(t) be the step size. We show that if mu(t) -> 0, then with probability one E[parallel to w(t)parallel to(2)(2)] is bound and lim(t ->infinity) parallel to w(t)parallel to(2) exists. Based on these two properties, we show that if mu(t) -> 0, Sigma(t)mu(t) = infinity, and Sigma(t)mu(t)(2) < infinity, then with probability one these algorithms converge. Moreover, w(t) converges with probability one to a point where del V-w(w) = 0.
convergence weight noise injection multilayer perceptron additive noise multiplicative noise