[工学]Ch11_backpropagation.ppt

Multilayer Perceptron Example Elementary Decision Boundaries Elementary Decision Boundaries Total Network Function Approximation Example Nominal Response Parameter Variations Multilayer Network Performance Index Chain Rule Gradient Calculation Approximate Steepest Descent Jacobian Matrix Backpropagation (Sensitivities) Initialization of sM (Last Layer) Summary Example: Function Approximation Network Initial Conditions Forward Propagation Transfer Function Derivatives Backpropagation Weight Update Some issues about BP implementation Choice of network architecture Network convergence and generalization Improvement of algorithm Choice of Architecture Choice of Network Architecture Convergence Generalization * 11 Z.C Huang BME.CSU Backpropagation R – S1 – S2 – S3 Network Nominal Parameter Values Several comments about the curve features: The centers of the steps occur where the net input to a neuron in the first layer is zero. The steepness of each step can be adjusted by changing the network weights. It has been shown that two-layer networks, with sigmoid transfer function in the hidden layer and linear transfer function in the output layer, can approximate virtually any function of interest to any degree of accuracy, provided sufficient hidden units are available. nnd11nf Training Set Mean Square Error Vector Case Approximate Mean Square Error (Single Sample) Approximate Steepest Descent Example Application to Gradient Calculation Sensitivity Gradient Next Step: Compute the Sensitivities (Backpropagation of sensitivities) s m F ? ? n m ? - - - - - - - - - - o F ? ? n 1 m ? - - - - - - - - - F ? ? n 2 m ? - - - - - - - - - ? F ? ? n S m m ? - - - - - - - - - - - = . . Sm+1?Sm The sensitivities are computed by starting at the last layer, and then propagating backwards through the network to the first layer. Forward Propagation Backpropagation Weight Update 1-2-1 Network + - t a e p -2 ? p ? 2 1-2-1 Network a p nnd11bc 1-3-1 Network i = 1 i = 2 i = 4 i = 8 1-5-1 1-2-1