intelligent document classification.pdfVIP

Intelligent document classification Rafael A. Calvo, H. A. Ceccatto ´ Instituto de Fisica Rosario (CONICET-UNR) 27 de Febrero 210bis 2000 Rosario, Argentina rafa@.ar May 26, 2000 Abstract In this work we investigate some technical questions related to the applica- tion of neural networks in document classification. First, we discuss the effects of different averaging protocols for the  2 statistic used to remove non-informative terms. This is an especially relevant issue for the neural network technique, which requires an aggressive dimensionality reduction to be feasible. Second, we es- timate the importance of performance fluctuations due to inherent randomness in the training process of a neural network, a point not properly addressed in previous works. Finally, we compare the neural network results with those obtained using the best methods for this application. For this we optimize the network architecture by evaluating much larger nets than previously considered in similar studies in the literature. KEYWORDS: text classification, statistical learning, neural networks, knowledge management. 1 1 Introduction The content explosion in the information age has produced a major challenge to sci- ence and technology: How can we use efficiently all the numerical data, text, sound and video produced? A clear sign of the problems to come is that the contents avail- able on the Internet are growing at a much faster pace than the possibility of finding particular pieces of this information. The number of documents available on the web on almost any subject, and the ambiguity in user queries sent to search engines like Altavista (about 2 terms per query), created a gr