Fast approximate hierarchical clustering using similarity heuristics.docVIP

BioData Mining BioMedCentral Research Open Access Fast approximate hierarchical clustering using similarity heuristics Meelis Kull*1,2 and Jaak Vilo*1,2 Address: 1Institute of Computer Science, University of Tartu, Liivi 2, 50409 Tartu, Estonia and 2Quretec Ltd. ülikooli 6a, 51003 Tartu, Estonia Email: Meelis Kull* - Meelis.Kull@ut.ee; Jaak Vilo* - Jaak.Vilo@ut.ee * Corresponding authors Published: 22 September 2008 Received: 8 April 2008 Accepted: 22 September 2008 BioData Mining 2008, 1:9 doi:10.1186/1756-0381-1-9 This article is available from: /content/1/1/9 ? 2008 Kull and Vilo; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Background: Agglomerative hierarchical clustering (AHC) is a common unsupervised data analysis technique used in several biological applications. Standard AHC methods require that all pairwise distances between data objects must be known. With ever-increasing data sizes this quadratic complexity poses problems that cannot be overcome by simply waiting for faster computers. Results: We propose an approximate AHC algorithm HappieClust which can output a biologically meaningful clustering of a large dataset more than an order of magnitude faster than full AHC algorithms. The key to the algorithm is to limit the number of calculated pairwise distances to a carefully chosen subset of all possible distances. We choose distances using a similarity heuristic based on a small set of pivot objects. The heuristic efficiently finds pairs of similar objects and these help to mimic the greedy choices of full AHC. Quality of approximate AHC as compared to full AHC is studied with three measures. The first measure evaluates the global quality o