minhash聚类的讲解资料.pdfVIP

049011: Algorithms for Large Data Sets Spring 2005 Lecture 11 (01/06/05) Sketching Instructor: Ziv Bar-Yossef Scribe: Uri Schonfeld 1 Introduction In this lecture we will get to know several techniques that can be grouped by the general de?nition of sketching. When using the sketching technique each element is replaced by a more compact representation of itself. An alternative algorithm is run on the more compact representations. Finally, one has to show that this algorithm gives the same result as the original algorithm with high probability. This technique is shown through two example problems: ? Syntactic Clustering of the Web: Given a large set of documents, output them grouped by “similarity groups.” ? Calculating Hamming Distance: Given two strings, calculate the range of the number of bits in which they di?er. Through these examples we will become familiar with the idea of sketching algorithms and get to know such important concepts as shingles. 2 Syntactic Clustering The problem of syntactic clustering of documents is, given a large document set, identify groups of documents that are very similar in content. The motivation for the problem is to eliminate near duplicate documents. It turns out there are a lot of near-duplicate documents on the web. Examples include mirror sites, manuals, di?erent versions of the same document and even plagiarism. The reason we want to eliminate duplicate documents