爬虫工具汇总（Crawler tools summary）.docVIP

爬虫工具汇总（Crawler tools summary） Heritrix Heritrix is an open source, scalable web crawler project. Heritrix is designed to strictly follow the robots.txt files removal instructions and the META robots tag. Http:/// WebSPHINX WebSPHINX is an interactive development environment for Java class packages and Web crawlers. Web crawlers (also known as robots or spiders) are programs that automatically browse and process Web pages. WebSPHINX consists of two parts: the crawler, the workbench, and the WebSPHINX class package. Http:///~rcm/websphinx/ WebLech WebLech is a powerful tool for downloading and mirroring Web sites. It supports downloading web sites by functional requirements and is able to mimic the behavior of standard Web browsers as much as possible. WebLech has a function console and uses multithreaded operations. Http:/// Arale Arale is designed primarily for personal use, rather than focusing on the page index like any other crawler. Arale can download the entire web site or some resources from the web site. Arale is also able to map dynamic pages into static pages. Http://web.tiscali.it/_flat/arale.jsp.html J-Spider J-Spider: is a fully configurable and customizable Web Spider engine. You can use it to check the site errors (internal server error), the site of internal and external link checking, structure analysis of the website (which can create a site map), download the Web site, you can also write a plug-in to JSpider extend the functionality you need. Http:/// Spindle Spindle is a Web index / search tool built on top of the Lucene toolkit. It includes a HTTP spider for creating indexes and a search class for searching these indexes. The spindle project provides a set of JSP tag libraries that enable JSP based sites to increase search functionality without having to develop any Java class. Http:///projects/spindle/ Arachnid Arachnid: is a web based on spider Java framework. It contains a simple HTML parser to analyze the input stream containing HTML content.