爬虫url去重（Crawler URL de weight）.docVIP

爬虫url去重（Crawler URL de weight） In the process of starting the crawler, we dont want the same page to be downloaded many times, because repeated downloading not only wastes CPU, but also adds load to the search engine system. In order to control this repetitive download, we should consider the hyperlinks to download. As long as we can control the URL to be downloaded without duplication, we can basically solve the problem of repeated downloading of the same web page. Its easy to think of building a global search engine in the search engine system that will be used to check if a URL corresponding to a web page has been downloaded. Thats the solution, URL. The next thing to consider is how to make the crawler work more efficiently, or, to be exact, to make it more productive. If the implementation is heavy, must be the establishment of a URL repository, and have downloaded the URL in the detection time, to be loaded into memory, were detected in the memory will be much faster than the speed of reading directly from disk. Lets start with the simplest situation, and then gradually optimize, and finally get a very good solution. First, disk based sequential storage. Here is to store each downloaded URL sequentially. You can save all of the downloaded URL into the disk Notepad file. Every time there is a crawler to get a thread to start the download task URL before, through to retrieve the files on the disk, if not, will this new URL Notepad write the last line, or give up the URL download. This approach is hardly thought of, but the idea of such an inspection is very intuitive. Just imagine, if you have downloaded 10 billion pages, then the corresponding 10 billion links, namely the check whether URL will repeat the notepad file storage of the 10 billion URL, and URL, a lot of the length of the string is not small, the storage space does not say, super low search efficiency, this plan will certainly give up. Second, storage based on Hash algorithm. For each given URL, it