Clustering Applications at Yahoo!-2009外文电子书.ppt

Clustering Applications at Yahoo! Deepayan Chakrabarti (deepay@) Outline Clustering, in itself, is often not the primary problem Exploratory analysis is rarely needed Methods are often tied to the “final” goal Outline Advertiser-keyword graph(+ social networks) Graph Partitioning CTR predictions for ads on webpages Co-clustering Query refinement and suggestions Local search methods Conclusions Outline Advertiser-keyword graph(+ social networks) Graph Partitioning CTR predictions for ads on webpages Co-clustering Query refinement and suggestions Local search methods Conclusions Graph Partitioning (Applications) Find clusters of advertisers and keywords Keyword suggestions Running experiments on some “natural” clusters Similar: Y! Answers Flickr Graph Partitioning (Applications) Find clusters of IM users Targeted advertising Exploratory analysis Clusters of the Web Graph Distributed pagerank computation Graph Partitioning (Methods) Basic “Global” spectral partitioning [Ng+/01] Find 2nd eigenvector of the graph Laplacian This embeds all nodes on the real line Split the line in two, to get two clusters Can approximate the optimal conductance cut For more clusters: Use k eigenvectors (for known k), or Split in two, and recurse on each cluster Graph Partitioning (Methods) However, this has problems [Lang/05, Leskovec+/08]: Min. conductance or quotient cuts lead to “small chunks” Graph Partitioning (Methods) However, this has problems: Min. conductance or quotient cuts lead to “small chunks” Balance is not very strongly encouraged Recursive partitioning takes too long Each eigenvector computation yields only a small cluster being broken off Two alternatives: More balanced cuts ? recursion is faster Unbalanced cuts, but much faster computation Graph Partitioning (Methods) Achieving better balance Combine algorithms (e.g., [Anderson+/08]) METIS (more balanced cuts), followed by Flow-based improvement (conductance) Stronger balance constraints Graph Partitioning (Me