TT. Poggio, R. Rifkin, S. Mukherjee, P. Niyogi General.pptVIP

T. Poggio, R. Rifkin, S. Mukherjee, P. Niyogi:General Conditions for Predictivity in Learning Theory Michael Pfeiffer pfeiffer@igi.tugraz.at 25.11.2004 Motivation Supervised Learning learn functional relationships from a finite set of labelled training examples Generalization How well does the learned function perform on unseen test examples? Central question in supervised learning What you will hear New Idea: Stability implies predictivity learning algorithm is stable if small pertubations of training set do not change hypothesis much Conditions for generalization on learning map rather than hypothesis space in contrast to VC-analysis Agenda Introduction Problem Definition Classical Results Stability Criteria Conclusion Some Definitions 1/2 Training Data: S = {z1=(x1,y1), ..., zn=(xn, yn)} Z = X ? Y Unknown Distribution ?(x, y) Hypothesis Space: H Hypothesis fS ? H: X ? Y Learning Algorithm: Regression: fS is real-valued / Classification: fS is binary symmetric learning algorithm (ordering irrelevant) Some Definitions 2/2 Loss Function: V(f, z) e.g. V(f, z) = (f(x) – y)2 Assume that V is bounded Empirical Error (Training Error) Expected Error (True Error) Generalization and Consistency Convergence in Probability Generalization Performance on training examples must be a good indicator of performance on future examples Consistency Expected error converges to most accurate one in H Agenda Introduction Problem Definition Classical Results Stability Criteria Conclusion Empirical Risk Minimization (ERM) Focus of classical learning theory research exact and almost ERM Minimize training error over H: take best hypothesis on training data For ERM: Generalization ? Consistency What algorithms are ERM? All these belong to class of ERM algorithms Least Squares Regression Decision Trees ANN Backpropagation (?) ... Are all learning algorithms ERM? NO! Support Vector Machines k-Nearest Neighbour Bagging, Boosting Regularization ... Vapnik asked Class