COMP 578Discovering Clusters in Databases.ppt

Data Warehouse Management Handbook COMP 578Discovering Clusters in Databases Keith C.C. Chan Department of Computing The Hong Kong Polytechnic University Discovering Clusters Introduction to Clustering Problem: Given A database of records. Each characterized by a set of attributes. To Group similar records together based on their attributes. Solution: Defines similarity/dissimilarity measure. Partition database into clusters according to similarity. An Example of Clustering:Analysis of Insomnia Analysis of Insomnia (2) Applications of clustering Psychiatry To refine or even redefine current diagnostic categories. Medicine Sub-classification of patients with a particular syndrome. Social services To identify groups with particular requirements or which are particularly isolated. So that social services could be economically and effectively allocated. Education Clustering teachers into distinct styles on the basis of teaching behaviour. Similarity and Dissimilarity (1) Many clustering techniques begin with a similarity matrix. Numbers in matrix indicate degree of similarity between two records. Similarity between two records ri and rj is some function of their attribute values, i.e. sij = f(ri, rj) Where ri = [ai1, ai2, …, aip] and rj = [aj1, aj2, …, ajp] are the attributes values for ri and rj. Similarity and Dissimilarity (2) Most similarity measures are: Symmetric, i.e., sij = sji. Non negative. Scaled so as to have an upper limit of unity. Dissimilarity measure can be: dij = 1 - sij Also symmetric and non negative. dij + dik ? djk for all i, j ,k. Also called distance measure. The most commonly used distance measure is Euclidean distance. Some common dissimilarity measures Euclidean distance: City block: ‘Canberra’ metric: Angular separation: Examples of a similarity /dissimilarity matrix Hierarchical clustering techniques Clustering consists of a series of partitions/merging. May run from a single cluster containing all records to n clusters each