November16,2010.ppt

Neural Networks Lecture 17: Self-Organizing Maps November 16, 2010 Neural Networks Lecture 17: Self-Organizing Maps * About Assignment #3 Two approaches to backpropagation learning: 1. “Per-pattern” learning: Update weights after every exemplar presentation. 2. “Per-epoch” (batch-mode) learning: Update weights after every epoch. During epoch, compute the sum of required changes for each weight across all exemplars. After epoch, update each weight using the respective sum. November 16, 2010 Neural Networks Lecture 17: Self-Organizing Maps * About Assignment #3 Per-pattern learning often approaches near-optimal network error quickly, but may then take longer to reach the error minimum. During per-pattern learning, it is important to present the exemplars in random order. Reducing the learning rate between epochs usually leads to better results. November 16, 2010 Neural Networks Lecture 17: Self-Organizing Maps * About Assignment #3 Per-epoch learning involves less frequent weight updates, which makes the initial approach to the error minimum rather slow. However, per-epoch learning computes the actual network error and its gradient for each weight so that the network can make more informed decisions about weight updates. Two of the most effective algorithms that exploit this information are Quickprop and Rprop. November 16, 2010 Neural Networks Lecture 17: Self-Organizing Maps * The Quickprop Learning Algorithm The assumption underlying Quickprop is that the network error as a function of each individual weight can be approximated by a paraboloid. Based on this assumption, whenever we find that the gradient for a given weight switched its sign between successive epochs, we should fit a paraboloid th