网络爬虫源代码（Web crawler source code）.docVIP

网络爬虫源代码（Web crawler source code） Web crawler source code.Txt, and then a few decades, we come to meet, sent to the crematorium, all burned into ashes, you pile, I pile, who do not know who, all sent to the countryside to do fertilizer. Public, class, Spider, implements, Runnable { Private ArrayList URLs; //URL list Private HashMap indexedURLs; / / URL has retrieved the list Private int threads; / / initialize thread number Public, static, void, main (String, argv[]) throws Exception { If (argv[0] = null) { System.out.println (Missing required argument: [Sit URL]); Return; } Spider Spider = new Spider (argv[0]); Spider.go (); } Public Spider (String strURL) { URLs = new, ArrayList (); Threads = 10; Urls.add (strURL); ThreadList = new, ArrayList (); IndexedURLs = new, HashMap (); (if) (urls.size = 0) Throw new IllegalArgumentException (Missing required argument: -u [start url]); If (threads 1) (Invalid, number, of, threads: +) Threads); } Public void go (String strURL) throws Exception { Each entry point URL / / index Long start = System.currentTimeMillis (); For (int i = 0; I threads; i++) { Thread, t = new, Thread (this, Spide) + (i+1); T.start (); ThreadList.add (t); } While (threadList.size () , 0) { Thread child = (Thread) threadList.remove (0); Child.join (); } Long elapsed = System.currentTimeMillis () - start; } Public, void, run () { String url; {try While ((url = dequeueURL ()) = = null) { IndexURL (URL); } }catch (Exception, e) { Logger.info (e.getMessage ()); } } Detection of URL / / list container have not been resolved if there is URL, URL is returned by the thread to continue Public, synchronized, String, dequeueURL (), throws, Exception { While (true) { If (urls.size () ) 0 { Return (String) urls.remove (0); } {else Threads--; If (threads 0) { Wait (); Threads++; } Else { NotifyAll (); Return null; } } } } * * add URL and current URL progressions, and wake up sleep threads * / 公共使用同步enqueueurl（网址字符串、整型） { 如果（indexedurls。得到（URL）= = null） { 添加（URL）； i