Optimal Brain Surgeon Algorithm - CAE Users.DOC

Optimal Brain Surgeon Algorithm ECE 539 Final Project Mark Slosarek 9014829809Introduction: For this project I am performing the optimal brain surgery algorithm. This algorithm is a pruning algorithm that reduces the connection between layers thus making it more efficient. A pruned network has several advantages. First, it needs smaller storage space. Second, it spends shorter time in calculation. Reducing a neural networks complexity improves the ability of the network to generalize future examples. Work Performed: The Optimal Brain Surgery Algorithm is based on the MLP Feed-forward Model similar to that described in Lecture 9 MLP (I): Feed-forward Model of professor Hu’s notes. Figure 1: A Three Layer Feed-Forward Multi-layer Perceptron In implementing the Optimal Brain Surgeon Algorithm, I implemented six steps as described in Neural Networks: A comprehensive Foundation by Simon Haykin. Train the given multilayer perceptron to minimum mean-square error Use the multilayer perceptron model with two hidden layers and one output neutron (three layer feed-forward multilayer perceptron) to compute the vectorwhere is the input-output mapping realized by the multilayer perceptron with an overall weight vector , and is the input vector. Use the recursion to calculate the inverse Hessian Find the that corresponds to the smallest saliency:where is the element of . If the saliency is much smaller than the mean square, , then delete synaptic weight , and proceed to step 4. Otherwise, go to 5. Update all the synaptic weights in the network by applying the adjustment:Go to step 2. Stop the computation when no more weights can be deleted from the network without a large increase in the mean-square error. To truly implement this solution many different loops had to be implemented. These loops ended up being very cost effective on the computer and so I implemented a weight removal system that applied an algorithm that removed more weights. Although this was more ef