PODS 2002 Invited Talk Stanford CS Theory荚2002邀请谈斯坦福政务司司长理论.pptVIP

CS 361A stanfordstreamdatamanager CS 361A (Advanced Data Structures and Algorithms) Lecture 18 (Nov 30, 2005) Fingerprints, Min-Hashing, and Document Similarity Rajeev Motwani Game Plan for Week Fingerprints Document Similarity Shingling Min-Hashing Min-Wise Independent Permutations Fingerprints W – set of large objects (e.g., URLs) Goal avoid storing large objects explicitly quick-and-dirty equality-testing Fingerprints? Short tags for objects Distinct fingerprints ? distinct objects Distinct objects ? probably distinct fingerprints Formalization Fingerprint length k ? fingerprint space size N=2k Fingerprint function family F = { f : W?{0,1}k } Random f eR F ? f(A) 1 f(B) ? A 1 B Collisions: P[ f(A) = f(B) | A 1 B ] ? 0 (ideally 2O(-k)) Typical Application Adversarial object-set S with |S| = n 2k Goal – |f(S)| = |S| with high probability n2 pair-wise collisions possible ? need 2k n2 (to avoid Birthday Paradox) Example – URL Fingerprints Search Engines Manage large numbers of URL strings Long, variable strings (embedded objects/database-queries) Desiderata small/fixed-length encodings – hopefully, unique Some scenarios Exact string irrelevant Only need ability to distinguish distinct URLs Even otherwise, unique IDs useful for indexing Numbers? 4 billion webpages ? n=232 N ? n2 ? k=64 Fingerprints ? 8-byte representation Fingerprinting vs Hashing Hashing h: W ? {0,1}k Set Membership testing for set S of size n Desire uniform distribution over bin address {0,1}k Minimize collisions per bin – reduce lookup time Minimize hash table size ? n ? N=2k Fingerprinting f : W ?{0,1}k Object Equality testing over set S of size n Distribution over {0,1}k is irrelevant Avoid collisions altogether Tolerate larger k – typically N n2 Fingerprinting Strings Typical Application – but techniques extend to combinatorial objects (database tuples, trees/graphs) Obvious techniques Checksum – no worst-case collisio