mlps(mono-layerpolynomialsandmulti-layerperceptrons.docVIP

MLPs (Mono-Layer Polynomials and Multi-Layer Perceptrons)for Nonlinear Modeling Isabelle Rivals isabelle.rivals@espci.fr Léon Personnaz leon.personnaz@espci.fr équipe de Statistique Appliquée école Supérieure de Physique et de Chimie Industrielles Paris, F75005, FRANCE Editors: Isabelle Guyon and André Elisseeff Abstract This paper presents a model selection procedure which stresses the importance of the classic polynomial models as tools for evaluating the complexity of a given modeling problem, and for removing non-significant input variables. If the complexity of the problem makes a neural network necessary, the selection among neural candidates can be performed in two phases. In an additive phase, the most important one, candidate neural networks with an increasing number of hidden neurons are trained. The addition of hidden neurons is stopped when the effect of the round-off errors becomes significant, so that, for instance, confidence intervals cannot be accurately estimated. This phase leads to a set of approved candidate networks. In a subsequent subtractive phase, a selection among approved networks is performed using statistical Fisher tests. The series of tests starts from a possibly too large unbiased network (the full network), and ends with the smallest unbiased network whose input variables and hidden neurons all have a significant contribution to the regression estimate. This method was successfully tested against the real-world regression problems proposed at the NIPS2000 Unlabeled Data Supervised Learning Competition; two of them are included here as illustrative examples. Keywords: additive procedure, approximate leave-one-out scores, confidence intervals, input variable selection, Jacobian matrix conditioning, model approval, model selection, neural networks, orthogonalization procedure, overfitting avoidance, polynomials, statistical tests. 1. Introduction Many researchers in the neural network community devote a large part of their work to t