Two Strategies to Avoid Overfitting in Feedforward Networks
We present a new network topology to avoid overfitting in two-layered feedforward networks. We use two additional linear layers and principal component analysis to reduce the dimension of both inputs and internal representations and to transmit the essential information. Thereby neurons with small variance in the output are removed, which results in better generalization properties. Our network and learning rules can also be seen as a procedure to reduce the number of free parameters without using second order information of the error function. As a second strategy we derive a penalty term, which drives the network to keep the variances of the hidden layer outputs small. Experimental results show that thereby the transmitted information is limited, which reduces the noise and gives better generalization. The variances of the outputs of the hidden neurons are used again as a pruning criterion.  1997 Elsevier Science Ltd. All Rights Reserved.
feedforward networks information penalty terms pruning mechanisms principle component analysis theory overfitting.