的压缩方法.pptxVIP

吴金龙 2011/04/07;Contents;剪枝（Pruning）;聚类（Clustering）;量化（Quantization）;紧凑的数据结构;随机化方法（ Randomized LMs ）;Log-frequency Bloom filter;Randomized LMs via Perfect Hash Functions; For example，;Encoding n-grams in the model All addresses in A are initialized to zero. The procedure we use to ensure g(xi) = v(xi) for all xi ∈ S updates a single, unique location in A for each n-gram xi To ensure that once an n-gram xi has been encoded in the model, these k locations are not subsequently changed since this would invalidate the encoding To find a suitable order in which to enter n-grams in the model and, for each n-gram xi, determine which of the k hash functions, say hj , can be used to update A without invalidating previous entries. Given this ordering of the n-grams and the choice of hash function hj for each xi ∈ S,;Finding an Ordered Matching For example， matched result = [(a, 1), (b, 2), (d, 4), (c, 5)];False positive The ?ngerprint f(xi) is assumed to be distributed uniformly at random (u.a.r.) in [0,B ? 1], g(xi) is also u.a.r. in [0,B?1] for xi ∈ U\S. The probability that xi ∈ U \ S is assigned a value in v ∈ V: Define error bits: 8-bit quantization + 12 error bits are enough (8+12)×1.23 = 24.6 bits = 3.08 bytes per n-gram Further improvement quantized probabilities using 2.26 bytes per n-gram See David Guthrie et al., Efficient Minimal Perfect Hash Language Models, 2009 ;Review;The End