数据挖掘和分析9W9A精要.pptVIP

聚类分析 2014.12.30 第十九周 簇的确认 Cluster Validity 如何验证和评价聚类分析的结果？ “goodness” of the resulting clusters? But “clusters are in the eye of the beholder”! 为何评价聚类分析的结果? 避免发现噪声产生的模式 比较不同的聚类算法 比较不同的簇集合 簇之间的比较 随机数据中被发现的簇 Clusters found in Random Data 簇确认的重要问题 Cluster Validation 确定数据集的聚类趋势 clustering tendency ，即是否存在非随机结构 确定正确的簇的个数. 评估聚类分析结果对数据的拟合情况 - Use only the data 将聚类分析的结果跟已知的客观结果（如，外部提供的类标号）比较 比较不同的聚类方法的优劣. 1,2,3 非监督 3,4,5 进一步区分是评估整个聚类还是单个簇 簇确认的度量 Measures of Cluster Validity 三种度量方式 外部指标 External Index: 监督的 Used to measure the extent to which cluster labels match externally supplied class labels. Entropy 熵 内部指标 Internal Index: 非监督的 Used to measure the goodness of a clustering structure without respect to external information. Sum of Squared Error (SSE) 相对指标 Relative Index: Used to compare two different clusterings or clusters. Often an external or internal index is used for this function, e.g., SSE or entropy Sometimes these are referred to as criteria instead of indices However, sometimes criterion is the general strategy and index is the numerical measure that implements the criterion. 非监督簇评估：通过相关 Via Correlation 两个矩阵 Two matrices 邻近性矩阵 Proximity Matrix 理想的邻近性矩阵 “Incidence” Matrix 每个数据点对应一行一列 矩阵中每项对应的两点如果是同簇，为1 否则为 0 计算两个矩阵的相关性 Since the matrices are symmetric, only the correlation between n(n-1) / 2 entries needs to be calculated. 高相关--簇中的点相近. Not a good measure for some density or contiguity based clusters. 非监督簇评估：通过相关 Correlation of incidence and proximity matrices for the K-means clusterings of the following two data sets. 通过相似性矩阵 根据簇对数据排序后的相似性矩阵 Order the similarity matrix with respect to cluster labels and inspect visually. 通过相似性矩阵 Clusters in random data are not so crisp 通过相似性矩阵 Clusters in random data are not so crisp 通过相似性矩阵 Clusters in random data are not so crisp 通过相似性矩阵 非监督的 Internal Measures: SSE Clusters in more complicated figures aren’t well separated 内部指标 Internal Index: 不需要外部信息Used to measure the goodness of a clustering structure wi