WJ-CH10-模式识别-聚类算法-06 英文版 教学课件.pptVIP

CLUSTER VALIDITY Clustering tendency Facts Most clustering algorithms impose a clustering structure to the data set X at hand. However, X may not possess a clustering structure. Before we apply any clustering algorithm on X, it must first be verified that X possesses a clustering structure. This is known as clustering tendency. Clustering tendency is heavily based on hypothesis testing. Specifically, it is based on testing the randomness (null) hypothesis (H0) against the regularity (H1) hypothesis and the clustering (H2) hypothesis . Randomness hypothesis (H0): “The vectors of Χ are randomly distributed, according to the uniform distribution in the sampling window (the compact convex support set for the underlying distribution of the vectors of the data set X) of X”. Regularity hypothesis (H1): “The vectors of X are regularly spaced (that is they are not too close to each other) in the sampling window”. Clustering hypothesis (H2): “The vectors of X form clusters”. p(q|H0), p(q|H1) and p(q|H2) are estimated via Monte Carlo simulations Some tests for spatial randomness, when the input space dimensionality greater than or equal to 2 are: Tests based on structural graphs Test that utilizes the idea of the minimum spanning tree (MST) Tests based on nearest neighbor distances The Hopkins test The Cox-Lewis test A method based on sparse decomposition. Cluster validity In the sequel it is assumed that the clustering tendency procedure  indicated the existence of a clustering structure in X. Applying a clustering algorithm on X, with inappropriate values of the involved parameters, poor results may be obtained. Hence the need for further evaluation of clustering results is apparent. Cluster validity: a task that evaluates quantitatively the results of a clustering algorithm. A clustering structure C, resulting from an algorithm may be either A hierarchy of clusterings or A single clustering. Cluster validity may be approached in three possible directions: