向量空间模型knn算法.ppt

聚类的质量评价 指标：纯度（Purity）和F值（F-measure） 标准答案：一般是人工分好类的文档集合 纯度 F值 F值 大作业选题 词义空间的聚类算法实现及优化 空间降维与合理变换 噪声识别与处理方案 Robust PCA 授课视频中ppt图像首帧定位算法 三维空间物体碰撞及运动模拟 引文网络中文章潜在影响力预测与分析 博弈算法优化及实现 密码及校验码及其他自选专题 本次词义聚类大作业要求 按中文词汇的上下文信息对词汇按句法功能聚类 或 ：其他类文本聚类的应用 可选目标：近义词挖掘、词类的上下文特征挖掘 选大作业：发信给hujf@. 提交时间： 聚类算法实现及优化 11月14号 语言学词类本质探讨 11月21号 * * 中国 日本 文化 -留学生 site:/ Baidu report 38 results Google report 908 results * incidence matrix 关联矩阵 * Linked lists generally preferred to arrays Dynamic space allocation Insertion of terms into documents easy Space overhead of pointers * How to build inverted index, 1M*1K terms document collection, There are 1G postings, can’t do it in memory. 讨论? * Queries using AND, OR and NOT together with query terms Views each document as a set of words Is precise: document matches condition or not. Primary commercial retrieval tool for 3 decades. Professional searchers (e.g., Lawyers) still like Boolean queries: You know exactly what you’re getting. 00/uhtbin/cgisirsi/nLEYr4CC23/%E5%8C%97%E5%A4%A7%E4%B8%AD%E5%BF%83%E9%A6%86/201190538/2/1000 * Thus far, our queries have all been Boolean Docs either match or not OK for expert users with precise understanding of their needs and the corpus Not good for (the majority of) users with poor Boolean formulation of their needs Most users don’t want to wade through 1000’s of results – cf. use of web search engines * Document is a vector: a column below Bag of words model WARNING: In a lot of IR literature, “frequency” is used to mean “count” Thus term frequency in IR literature is used to mean number of occurrences in a doc Not divided by document length (which would actually make it a frequency) We will conform to this misnomer In saying term frequency we mean the number of occurrences of a term in a document. * Thus the doc John is quicker than Mary. is indistinguishable from the doc Mary is quicker than John. Bag of words view of a doc * Document/collection frequency weighting is on