Efficient Similarity Joins for Near Duplicate Detection近重复检测的有效相似度.pptVIP

Efficient Similarity Joins for Near Duplicate Detection Efficient Similarity Joins for Near Duplicate Detection Chuan Xiao The University of New South Wales, Australia Joint Work: Wei Wang (UNSW), Xuemin Lin (UNSW), Jeffrey Xu Yu (CUHK) Outline Introduction Algorithms Experiments Conclusion Near Duplicate Data Applications For Web search engines: Perform focused crawling Increase the quality and diversity of query results Identify spams. For Web mining: Perform document clustering Find replicate Web collections Detect plagiarism Similarity Join near duplicates = pairs of objects with high similarity similarity - quantitative way - similarity function Given a collection of records, the similarity join problem is to find all pairs of records, x,y, such that sim(x,y)=t Tokenize: Each record is a set of tokens from a finite universe. Suppose each record is a single text document x = “yes as soon as possible” y = “as soon as possible please” x = {A, B, C, D, E} y = {B, C, D, E, F} Similarity Function Common similarity functions: Jaccard: Cosine: Overlap: Jaccard can be equivalently converted to Overlap Na?ve and Index-Based Algorithms Na?ve Algorithm: Compare every pair of objects - O(n2) time complexity Index-based Algorithm [MIR, SIGMOD04]: Index-Based Algorithm Index-based Algorithm Example Suppose sim(x,y) = O(x,y) = t = 3 Result: w,x, w,y Prefix Filter [ICDE06, WWW07] Sort the tokens by a global ordering increasing order of document frequency Index the first few tokens (prefix) for each record Example: suppose sim(x,y) = O(x,y) = t = 4 x = y = Must share at least one token in prefix to be a candidate pair Prefix Filter [ICDE06, WWW07] O(x,y) = t ? prefix length = |x| - t + 1 J(x,y) = t ? O(x,y) = t |x| ? prefix length = (1-t) |x| + 1 Example: suppose sim(x,y) = J(x,y) = t = 0.8 w = {C, D, E, F} x = {B, C, D, E, F} y = {A, B, C, D, F} z = {G, A, B, E, F} Prefix + Positional Information We use prefix filter (All-Pairs [w