Decision Tree Learning - 北大计算机研究所.ppt

Source of the embedded image: /~tom/mlbook-chapter-slides.html source of the embedded image in this page: /~tom/mlbook-chapter-slides.html Why Multilayer? Single layer network has quite a limited representation power, equivalent to a linear classifier. Ex. Consider the problem of recognition 1 of 10 vowel sounds occurring in the context “h_d” (e.g. had, hid), with input of two parameters obtained from a spectral analysis of sounds, then the decision surface might become quite complicated: * Why Multilayer (2) * A Differentiable Threshold Unit How to choose the type of unit as the basis for constructing multilayer networks? Linear unit not good enough Perceptron not differentiable Modify perceptron to make it differentiable! One natural choice is sigmoid function * A Differentiable Threshold Unit (2) We can derive gradient descent rule to train one sigmoid unit Multilayer network of sigmoid unit leads to backpropagation algorithm * Error Gradient for one Activation Unit * Backpropagation Algorithm Since for multilayer network, we must consider multiple outputs instead, thus we redefine E as the sum of errors over all of the network outputs: * Backpropagation Algorithm (2) Consider network with one hidden layer, and sigmoid as unit function, then incremental gradient descent version of the algorithm becomes: Assume input and weight from unit i into j are denoted by xij and wij, respectively. Initialize all weights to small random number Until the termination condition is met, Do For each training example (x, t), Do Propagate the input forward through the network Computer the output of ou every unit u in the network Propagate the error backward through the network For each output unit k, calculate error For each hidden unit k, calculate error Update each network weight * Backpropagation Algorithm (3) Eq 2 is identical in form to the delta training rule, however, Eq 3 is somehow different, b/c we can’t compute the target outputs at hidden units, bu