Lecture10Tree-basedmethods,neutralnetworks.pptVIP

Tree-based methods Why decision trees Compact representation of data Possibility to predict outcome of new observations Tree structure Root Nodes Leaves (terminal nodes) Parent-child relationship Condition Label is assigned to a leaf Example How to build a decision tree: Hunt’s algorithm Proc Hunt(Dt,t) Given Data set Dt={(X1i,..Xpi, Yi), i=1..n}, t-curr.node If all Yi are equal mark t as leaf with label Yi If not, use the test condition to split into Dt1…Dtn, create children t1…tn and run Hunt(Dt1,t1),…, Hunt(Dtn,tn) Hunt’s algorithm example Hunt’s algorithmWhat if some combinations of attributes are missing? CART: Classification and regression trees Regression trees Given Dt={(X1i,..Xpi, Yi), i=1..n}, Y – continuous, build a tree that will fit the data best Classification trees Given Dt={(X1i,..Xpi, Yi), i=1..n}, Y – categorical, build a tree that will classify the observations best A CART algorithm: Regression trees Post-pruning CART: Classification trees For each node define proportions Define measure of impurity Design issues of decision tree induction How to split: CART How to split: C4.5 Constructing decision trees Expressing attribute test conditions Characteristics of decision tree induction Neural networks Principal components regression (PCR) Extract principal components (transformation of the inputs) as derived features, and then model the target (response) as a linear function of these features Neural networks with a single target Extract linear combinations of the inputs as derived features, and then model the target (response) as a linear function of a sigmoid function of these features Artificial neural networks Introduction from biology: Neurons Axons Dendrites Synapse Capabilities of neural networks: Memorization (noise stable, fragmentary stable!) Classification Terminology Feed-forward neural network Input layer [Hidden layer(s)] Output layer Terminology Feed-forward network Nodes in one layer are connected to the nodes in next laye