ParticleSwarmModelSelection.PDF

Journal of Machine Learning Research 10 (2009) 405-440 Submitted 1/07; Revised 8/08; Published 2/09 Particle Swarm Model Selection Hugo Jair Escalante HUGOJAIR@CCC.INAOEP.MX Manuel Montes MMONTESG@INAOEP.MX Luis Enrique Sucar ESUCAR@INAOEP.MX Department of Computational Sciences National Institute of Astrophysics, Optics and Electronics Puebla, Mexico,′ 72840 Editor: Isabelle Guyon and Amir Saffari Abstract This paper proposes the application of particle swarm optimization (PSO) to the problem of full model selection, FMS, for classi?cation tasks. FMS is de?ned as follows: given a pool of pre- processing methods, feature selection and learning algorithms, to select the combination of these that obtains the lowest classi?cation error for a given data set; the task also includes the selection of hyperparameters for the considered methods. This problem generates a vast search space to be explored, well suited for stochastic optimization techniques. FMS can be applied to any classi?- cation domain as it does not require domain knowledge. Different model types and a variety of algorithms can be considered under this formulation. Furthermore, competitive yet simple models can be obtained with FMS. We adopt PSO for the search because of its proven performance in different problems and because of its simplicity, since neither expensive computations nor com- plicated operations are needed. Interestingly, the way the search is guided allows PSO to avoid over?tting to some extend. Experimental results on benchmark data sets give evidence that the proposed approach is very effective, despite its simplicity. Furthermore, results obtained in the framework of a model selection challenge show the c