文本挖掘简介.pptVIP

Introduction Why？ Text mining ≈ Web mining How？ Classification or Clustering Retrieval 文本分类一般过程 预处理 将文档集表示成易于计算机处理的形式 特征表示与选择、降维 根据适宜的权重计算方法表示文档中各项的重要性 学习建模 构建分类器 文本分类预处理 去标点、多余空格、数字（可选） 大小写统一 去停用词（stop words） 没有实际含义的词，比如and, you, have等等 统一词根 PorterStemmer 分词 英文？中文 特征表示 向量空间模型 以词项为特征组成高维特征向量 TF/IDF得到权值 TF-IDF TF（Term Frequency） 表示词项频率 IDF（Inverse Document Frequency） 逆文档频率 TF*IDF值 Measurement Edit distance Short text, words For personal text Jaccard distance Long text, ignoring the word similarity For government text Near Duplicate Data Similarity Join Tokenize: Each record is a set of tokens from a finite universe. Suppose each record is a single text document x = “yes as soon as possible” y = “as soon as possible please” x = {A, B, C, D, E} y = {B, C, D, E, F} 参考文献 Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu. Efficient Similarity Joins for Near Duplicate Detection. WWW 2008. Guoliang Li, Dong Deng, Jiannan Wang, Jianhua Feng. Pass-Join: A Partition based Method for Similarity Joins. VLDB 2012. 文本挖掘简介 邹权 博士，助理教授 Outline Introduction TF-IDF Similarity * Similarity Applications Many Web-mining problems can be expressed as finding “similar” sets: Plagiarism/Mirror Pages/Articles from the Same Source/Duplication Remove Collaborative Filtering as a Similar-Sets Problem Recommend to users items that were liked by other users who have exhibited smilar tastes Microsoft Academic Search PK /Author/2037349.aspx /Author/3054641.aspx Real-world Data is Rather Dirty！? Kenneth De Jong Kenneth Dejong * Trie-Join @ VLDB2010 */38 Typo in “author” Typo in “title” relaxed related Argyrios Zymnis Argyris Zymnis DBLP Complete Search * Real-world Data is Rather Dirty！? Trie-Join @ VLDB2010 */38 The similarity join is an essential operation for data integration and cleaning Perform a similarity join on Name attribute (find all record pairs whose Name attributes are similar) Output: (2037349, 3054641), … Similarity Joins R Id Name Univ. 2037349