机器学习方法在基于内容的垃圾邮件过滤中的研究-计算机软件与理论专业论文.docxVIP

东北大学硕士学位论文 东北大学硕士学位论文 摘 要 机器学习方法在基于内容的 垃圾邮件过滤中的研究 摘 要 高精度的垃圾邮件过滤技术是最近自然语言处理的研究热点。在这 篇论文中我们在有指导机器学习(Supervised Machine Leaming)的框架 下使用统计的方法研究垃圾邮件过滤问题。本文系统地研究了不同统计 特征选取的方法和特征集大小对常用分类算法·陛能的影响。特别地，本 文观察到特征选取函数的重要性随分类器的不同差别很大。以常用的朴 素贝叶斯(Naive Bayes)分类器为例，我们观察到其性能在小特征集上 受特征选取函数影响很大。支撵向量机(Support Vector Machine)， AdaBoost和最大熵模型在邮件分类上取得了很好的效果：对特征选取 函数不敏感，很容易扩展到高维特征空间，在不同测试集上的分类效果 稳定。此外，试验还表明简单的“Bag ofWords”过滤模型对垃圾邮件 过滤有很好的效果，尤其拥有与语言无关的特性 一我们对中文语料进 行了简单的预处理(分词)之后就取得了很好的分类效果。试验中还发 现邮件信头中的特征对分类效果有显著影响。在仅使用邮件信头中的特 征的情况下，我们在四个测试语料上取得了98％的分类准确率。这意味 着邮件信头的特征可以被用作有效垃圾邮件过滤特征。 关键词 文本分类垃圾邮件过滤机器学习 东北大学硕士学位论文 东北大学硕士学位论文 ABSTRACT Content·-based Anti·-Spam Filtering： the Machine Learning Approach ABSTRACT High performance Anti—Spam filtering techniques have been the focus of recent Natural Language Processing research In this thesis，we addressed the problem of content—based Anti-Spam filtering under the machine—learning framework． In particular,we studied the 1mpact of different feature pruning methods．and various feature set sizes on several statistical classifiers’performance using several public available COrpOVa We observed that the significance of feature selection varies greatly from classifier to classifier．Naive Bayes，a commonly used classifier in spam filtering，was found to be sensitive to feature selection methods on small feature set SVM．AdaBoost and Maximum Entropy Model performed as good as fine tuned Naive Bayes leamer,sharing similar characteristics：not sensitive to feature selection strategy，easily scalable to very high feature dimension and stable performances across different data sets．Furthermore．we concluded that the simple、’bag of words” filtering model can be effective in spam filtering task，with the language independence of this model being verified on a Chinese sparta COmUS A rather striking finding is the effect of mail headers on spare filtering，which was often ignored in previo