n-grambased text categorization（n-grambased文本分类）.pdfVIP

N-Gram-Based Text Categorization William B. Cavnar and John M. Trenkle Environmental Research Institute of Michigan P.O. Box 134001 Ann Arbor MI 48113-4001 Abstract training set, and less than 4K bytes for an indi- vidual document. Using N-gram frequency profiles provides a Text categorization is a fundamental task in doc- simple and reliable way to categorize documents ument processing, allowing the automated han- in a wide range of classification tasks. dling of enormous streams of documents in electronic form. One difficulty in handling some classes of documents is the presence of different 1.0 Introduction kinds of textual errors, such as spelling and grammatical errors in email, and character rec- ognition errors in documents that come through Electronic documents come from a wide variety OCR. Text categorization must work reliably on of sources. Many are generated with various all input, and thus must tolerate some level of word processing software packages, and are sub- these kinds of problems. jected to various kinds of automatic scrutiny, We describe here an N-gram-based approach e.g., spelling checkers, as well as to manual edit- to text categorization that is tolerant of textual