Nutchcrawler工作流程.docVIP

Nutch Crawler1¤×÷á÷3ì?°???t??ê??ê??·??? CrawleroíSearcherá?2?·?±???ê?·??a?C???÷òa??μ?ê??aá?ê1á???2?·??éò?2?μ??????úó2?t??ì¨é??Càyè?CrawleroíSearcher·?±e±?·????úá????÷?úé??C?a?ù?éò???′óμ?ìá??áé??D?oíD??ü?Bò??A×üì??ééü?F??? 1?A?è×￠è???×óurlsμ?crawldb??? 2?A?-?·?F generate ′ócrawldb?Déú3éò???urlμ?×ó?ˉó?óú×￥è? fetch ×￥è?é?ò?D?μ?urléú3éò?????segment parse ·???ò?×￥è?segmentμ??úèY update °?ò?×￥è?μ?êy?Y?üD?μ??-?èμ?crawldb ??? 3?A′óò?×￥è?μ?segments?D·???3?linkμ?í???? 4?A?÷òysegment??±??°inlink?a??±??t?A?à1?μ?êy?Y?á11?F??? Crawl DB?ü CrawlDb ê?ò???°üo?è????á11êy?Yμ????t:??? ??? ??? URL, CrawlDatum?ü CrawlDatum:??? ??? status, date, interval, failures, linkCount, ...?ü Status:??? ??? ? {db_unfetched, db_fetched, db_gone,linked,??? ??? ??? fetch_success, fetch_fail, fetch_gone}?@?@?à3?Crawler?F??? Crawlerμ?1¤×÷á÷3ì°üà¨á?????nutchμ??ùóD2??è?|?|injector,generator,fetcher,parseSegment, updateCrawleDB,Invert links, Index ,DeleteDuplicates, IndexMerger??? Crawleré??°μ?êy?Y???toí??ê?oío?ò??Coíò?é?μ??÷??2??è?à1?μ????t·?±e±?′?·??ú??àíéè±?é?μ?ò??????????t?Dà??Ccrawldb,segments,indexes,linkdb,index???????t?Dà??B??? ???′?÷??2??èoíá÷3ìê????′?C?÷?????t?Dà?ó?ê?·?×?ê2?′???H??? 1?2ìCrawleràà?éò??aμà?üμ?á÷3ì./nutch crawl urls -dir ~/crawl -depth 4 -threads 10 -topN 2000??? 1?AInjector injector = new Injector(conf);??? ??? ??? Usage: Injector crawldb url_dir??? ??? ê×?èê??¨á￠?eê?url?ˉ?C????url???-1yURLNormalizers?AfilteroíscoreFilterèy ??1y3ì2￠±ê??×′ì??Bê×?è?-1ynormalizer plugin,°?url??DD±ê×??ˉ?C±èè?basic?@nomalizerμ?×÷ó?óD°?′óD′μ?url±ê×??ˉ?aD?D′?C°?????è￥3yμèμè?Bè?oó?ù?-1yμ?plugin ê?filter?C?éò??ù?Y??D′μ??y?ò±í′?ê?°???òaμ?urlá???à′?B?-1yá???2??èoó?Cè?oó?íê?°??a??url??DD×′ì?±ê???C????url????ó|×?ò??? CrawlDatum,?a??àà??ó|×?????url?ú?ùóDéú?ü?ü?ú?úμ?ò??D×′ì??B???úé??1óD?a??url′|àíμ?ê±??oí3?ê?ê±μ?·??μ?B?@?@í?ê±?C?ú?a??2??èà??C?á?ú???t?μí3à?éú3é?@è??????t?@crawlDB/current/part-00000?@?@?a?????t?Dà??1óD.data.crc , .index.crc, data, index???????t?ü MapReduce1: °?ê?è?μ???