DEPARTMENT OF COMPUTER SCIENCE GLIMPSE（计算机科学部门,一瞥）.pdfVIP

Finding Similar Files in a Large File System Udi Manber TR 93-33 October 1993 DEPARTMENT OF COMPUTER SCIENCE To appear in the 1994 Winter USENIX Technical Conference FINDING SIMILAR FILES IN A LARGE FILE SYSTEM Udi Manber1 Department of Computer Science University of Arizona Tucson, AZ 85721 udi@ ABSTRACT We present a tool, called sif, for finding all similar files in a large file system. Files are considered similar if they have significant number of common pieces, even if they are very different otherwise. For example, one file may be contained, possibly with some changes, in another file, or a file may be a reorganization of another file. The run- ning time for finding all groups of similar files, even for as little as 25% similarity, is on the order of 500MB to 1GB an hour. The amount of similarity and several other customized parameters can be determined by the user at a post-processing stage, which is very fast. Sif can also be used to very quickly identify all similar files to a query file using a preprocessed index. Application of sif can be found in file management, information collecting (to remove duplicates), program reuse, file synchronization, data compression, and maybe even plagiarism detection. 1. Introduction Our goal is to identify files that came from the same source or contain parts that came f