决策树学习算法ID3DecisionTreeLearningAlgorithm.PPT

史忠植 高级人工智能 高级人工智能 决策树 决策树 决策树 决策树Decision Trees Tree structure Node = query on attribute Link = attribute value Leaf = class Recursively separate data into sub-populations Prediction: Traverse path, yield most probable class CLS算法 决策树 决策树 决策树 决策树 决策树 信息熵 Informative Establishes Good decision trees Entropy Measure how informative is a node Definition: P=(p1,p2,…,pn) Then Entropy of P is : I(P)= -(p1*log(p1) + p2*log(p2) +…+ pn*log(pn) ) Golf Example 信息熵 Entropy For example: If P=(0.5,0.5) ? I(P)=1 If P=(2/3,1/3) ? I(P)=0.92 If P= (1,0) ? I(P)=0 What do u see ? ? Entropy of previous golf example I(P)=I(9/14,5/14)=-(9/14*log(9/14) + 5/14*log(5/14)) = 0.94 信息熵 Entropy of an attribute For example: I(Outlook,T)= 5/14 * I(2/5,3/5) + 4/14 * I(4/4,0) + 5/14 * I)(3/5,2/5) = 0.694 While I(windy,T)=0.892 信息增益 Information Gain Gain(T, A) = I(T) – I(T,A) I(T) = expected information for distinguishing classes = -(p/(p+n)log2(p/(p+n))+n/(p+n)log2(n/(p+n))) I(T, A) = expected information of tree with A as root = ?i(pi+ni)/(p+n)*I(Ti) p, n: number of positive/negative training data pi, ni: number of positive/negative training data in the training data Ti partitioned by the attribute value Ai Select an attribute with highest information gain Prefer an attribute A with smallest I(T,A) i.e., Prefer the attribute that makes non-uniform distribution 信息增益 The initial information I(T) = I(9/14, 5/14) = 0.94 The information for Outlook attribute Sunny, Overcast, Rain I(T,Outlook) = 5/14*I(2/5,3/5) + 4/14*I(4/4,0/4) + 5/14*I(3/5,2/5) = 0.694 Gain(T,Outlook) = 0.94-0.694 = 0.246 The information for Windy attribute True, False I(T,Windy) = 6/14*I(3/6,3/6) + 8/14*I(6/8,2/8) = 0.892 Gain(T,Windy) = 0.94-0.892 = 0.048 select Outlook attribute for the first partition 信息增益 Entropy of an attribute Definition of Gain() Gain(X,T) = I(T) – I(X,T) Gain(Outlook,T) = I(T) – I(