Abstract-We investigate the use of inforination from all second order derivatives of the error function to perform network pruning (i.e., renioving unimportant weights fi*om a trained network) in order to improve generalization, siniplify networks, reduce hardware or storage requirements, increase the speed of further training. and in soiiie cases enable rule extraction. Our method, Optimal Brain Surgeon (OBS), is significantly better than magnitude-based methods and Optinial Brain Damage, which often remove the wrong weights. OBS permits pruning of more weights than other methods (for the same error on the training set), and thus yields better generalizatioii on test data. Crucial to OBS is a recursion relation for calculating the inverse Hessian matrix H-' from training data and structural information of the net. OBS permits a 76%, a 62%, and a 90% reduction in weights over backpropagation with weight decay on three benchmark MONK'S problems. Of OBS, Optimal Brain Damage, and a magnitude-based method, only OBS deletes the correct weights from a trained XOR network in every case. Finally, whereas Sejnowski and Rosenberg used 18,000 weights in their NETtalk network, we used OBS to prune a network to just 1,560 weights, yielding better generalization.
One of the earliest successes is the optimal brain surgeon approach REF .
Optimal Brain Surgeon and general network pruning
IEEE International Conference on Neural Networks
IEEE International Conference on Neural Networks
Computer Science