tight lower bounds for the distinct elements problem.pptVIP

Tight Lower Bounds for the Distinct Elements Problem David Woodruff MIT dpwood@mit.edu Joint work with Piotr Indyk The Problem Stream of elements a1, …, an each in {1, …, m} Want F0 = # of distinct elements Elements in adversarial order Algorithms given one pass over stream Goal: Minimum-space algorithm A Trivial Algorithm Negative Results Any algorithm computing F0 exactly must use ?(m) space [AMS96] Any deterministic alg. that outputs x with |F0 – x| ?F0 must use ?(m) space [AMS96] What about randomized approximation algorithms? Rand. Approx. Algorithms for F0 O(log log m/?2 + log m log 1/?) alg. outputs x with Pr[| F0 – x| ?F0 ] ? [BJKST02] Lots of hashing tricks Is this optimal? Previous lower bounds ?(log m) [AMS96] ?(1/?) [Bar-Yossef] Open Problem of [BJKST02]: GAP: 1/? 1/?2 Idea Behind Lower Bounds Randomized 1-way comm. complexity Boolean function f: X ￡ Y ! {0,1} Alice has x 2 X, Bob y 2 Y. Bob wants f(x,y) Only 1 message sent: must be from Alice to Bob Comm. cost of protocol = expected length of longest message sent over all inputs. ? -error randomized 1-way comm. complexity of f, R?(f), is comm. cost of optimal protocol computing f w.p. ? 1-? The VC Dimension [KNR] F = {f : X ! {0,1}} family of Boolean functions f 2 F is length-|X | “bit string” For S μ X, shatter coefficient SC(fS) of S is |{f |S}f 2 F| = # distinct bit strings when F restricted to S SC(F, p) = maxS 2 X, |S| = p SC(fS) If SC(fS) = 2|S|, S shattered by F VC Dimension of F, VCD(F), = size of largest S shattered by F Shatter Coefficient Theorem Notation: For f: X ￡ Y ! {0,1}, define: fX = { fx(y) : Y ! {0,1} | x 2 X }, where fx(y) = f(x,y) Theorem [BJKS]: For every f: X ￡ Y ! {0,1}, every p ? VCD( fX ), R1/4(f) = ?(log(SC(fX, p))) The ?(1/?) Lower Bound [Bar-Yossef] Alice has x 2R {0,1}m, wt(x) = m/2 Bob has y 2R {0,1}m, wt(y) = ?m and: Either wt(x ? y) = 0 OR wt(x ? y) = ?m f(x,y) = 0 f(x,y) = 1 R1/