蜘蛛简介（Spider profile）.docVIP

蜘蛛简介（Spider profile） Spider (Spider) is a very useful Internet program, using search engine spiders will collect Web pages to the database, using the enterprise spider program monitor competitorssites and tracking changes in individual users with the spider to download the Web page for offline use, developers use a spider to scan your Web check invalid link Spider programs have different uses for different users. So how exactly does spider work work? A spider is a semi-automatic program, just as the actual spider travels on its Web (spider web), and spider programs travel online in a similar manner on Web links. The spider is semi-automatic, because it always requires an initial link (starting point), but since then the situation will be decided by its own, the spider will scan the start page contains links, then access the link to the page, and then analysis and tracking of those pages contain links. In theory, the ultimate spider program accesses every page on Internet because almost every page on the Internet is always referenced by other, more or less pages. This article describes how to use C# language to construct a spider program, it can download the content of the entire web site to a specified directory, the programs running interface, as shown in figure 1. You can easily use the core classes provided in this article to construct your own spider program. Figure 1 C# is particularly suited for constructing spider programs, This is because it has built-in HTTP access and multi threading capabilities, and these two capabilities are very critical for spider programs. Here are the key issues to be solved for constructing a spider program: The HTML analysis: need a HTML parser to analyze every page spider encounter. The page processing: the need to handle each download page. The downloaded content may be saved to disk or further analyzed. The multi thread: only has the ability of multithreading, the spider can truly effective. How to determine when completed: do