爬虫工具汇总(国外英语资料).doc

爬虫工具汇总(国外英语资料) Heritrix Heritrix is an open source, extensible web crawler project. The Heritrix is designed to strictly follow the exclusion instructions and META robots tags of the robots.txt file. / WebSPHINX WebSPHINX is an interactive development environment for Java class packages and Web crawlers. Web crawlers (also called robots or spiders) are programs that automatically browse and process Web pages. WebSPHINX is made up of two parts: the crawler platform and the WebSPHINX package. /~ rcm/websphinx/ WebLech WebLech is a powerful Web site download and image tool. It supports downloading web sites by functional requirements and is able to mimic the behavior of standard web browsers as much as possible. WebLech has a functional console and USES multithreaded operations. / Arale Arale is designed primarily for personal use, and unlike other crawlers, it is focused on page indexing. Arale can download the entire web site or some resources from the web site. Arale can also map dynamic pages to static pages. http://web.tiscali.it/_flat/arale.jsp.html J - spiders J - spiders: is a fully configurable and customizable Web spiders engine. You can use it to check the website of error (internal server error, etc.), website, internal and external links, analysis of the structure of the site (you can create a site map), download the entire Web site, you can also write a JSpider plug-in to extend functionality you need. At / spindle The spindle is a Web index/search tool built on top of the Lucene toolkit, which includes an HTTP spider for creating indexes and a search class for searching these indexes. The project provides a set of JSP tag libraries that enable JSPS to add search functionality without developing any Java classes. /projects/spindle/ Arachnid Arachnid: is a java-based web spiders framework. It contains a simple HTML parser can analyze the input stream that contain HTML. By implementing the Arachnid subclasses can develop a simple web spiders and every page