Network Resource Search System Applying Cluster Architecture.docVIP

Network Resource Search System Applying Cluster Architecture 　　Key words: Web data mining, parallel gripper, incremental updating strategy 　　Introduction 　　Network resource search system is the front end of Web search engine and collects the information resources needed for Web. The characteristics that cluster models and computing nodes are combined to improve processing capacity assigns update tasks to appropriate computing nodes for operation, which can process the update of network resources. And adding and reducing computing nodes can automatically make task allocation adjustment, which can improve extensibility, based on which the paper introduces incremental updating technology to make the system estimate the updating rate of web page, which not only reduces work load of network resource search system updating web pages, but also improves the freshness of pages. 　　Multithreading Searcher 　　Collection efficiency is a very important performance index of crawlers. There are at least thousands of crawlers processing pages, so how to improve the performance and efficiency of program is very important. It is very common to take multithreading technology to improve collection efficiency in system implementation. 　　Thread synchronization must be considered under multithreading condition. In the system, URL queue to be accessed is multithreaded operation. While the thread accesses the queue, the other thresholds can’t insert, extract and modify the queue. Figure 1 shows concrete steps of thread synchronization of URL queue to crawl. 　　Crawlers are multithreaded, so one method is needed to assign the tasks in different threads. The job is completed by Spider Worker object. The object maintains thread pool, which can undertake the task of Spider creating and destroying thread objects. 　　Defined function of Spider Worker object: 　　Public class Spider Worker extends Thread 　　It indicates that Spider Worker object subclasses Thread object. Each execution call