nutch introdution开源爬虫软件.pdfVIP

NUTCH CHEAT SHEET What is NUTCH you ask? Nutch is a very popular open source JAVA based search engine built on top of L ucene which is translated to C, C++, C#, Python, Perl and Ruby. It provides all of the tools you need to run your very own search engine. Current version of nutch (as of October 2010) is: 1.2 Some important gotchas on nutch: • Founded in 2003 by Doug Cutting, the Lucene creator, and Mike Cafarella • Apache project since 2004 (sub-project of Lucene) • Spin-offs: • Map-Reduce and distributed FS → Hadoop • Content type detection and parsing → Tika • Many installations in operation, mostly vertical search • Collections typically 1 mln - 200 mln documents Usage: nutch [-core] COMMAND where COMMAND is one of the commands from following table or CLASSNAME the class named CLASSNAME most commands print help when invoked w/o parameters. Expert: -core option is for developers only. It avoids building the job jar, instead it simply includes classes compiled with ant compile-core. [ NOTE: this works only for jobs executed in local mode ] Comma Descriptio Usage Example nd n Crawl urlDir [-dir d] [-threads n] [-depth i] [-topN N] [- bin/nutch one-step solr solrURL] crawl urls crawl crawler for -dir crawl intranets -depth 3 -topN 50 readdb read / dump CrawlDbReader crawldb (-stats | -dump out_dir | -topN 1. bin/nutch crawl d nnnn out_dir [min] | -url url)