nutch摘要及网页快照分析.doc

Nutch二次开发总结（一） 2009年06月06日 星期六 16:06 通过一系列的离线活动（对于查询用户而言）的开展，Nutch检索系统相对而言变得简单了许多。在二次开发的时候，需要重点对Nutch的界面及界面显示数据进行适当的调整。1 摘要提取1.1 摘要提取源码分析 **????? * Low level api to get the most relevant (formatted) sections of the document.???? * 底层API，获取文档中最相关的（格式化）部分????? * This method has been made public to allow visibility of score information held in TextFragment objects.????? * Thanks to Jason Calabrese for help in redefining the interface.????? * @param tokenStream????? * @param text????? * @param maxNumFragments????? * @param mergeContiguousFragments????? * @throws IOException????? */???? public final TextFragment[] getBestTextFragments(???????? TokenStream tokenStream,???????? String text,???????? boolean mergeContiguousFragments,???????? int maxNumFragments)???????? throws IOException????? {???????? ArrayList docFrags = new ArrayList();???????? StringBuffer newText=new StringBuffer();???????? TextFragment currentFrag =??? new TextFragment(newText,newText.length(), docFrags.size());???????? fragmentScorer.startFragment(currentFrag);???????? docFrags.add(currentFrag);???????? FragmentQueue fragQueue = new FragmentQueue(maxNumFragments);???????? try????????? {???????????? org.apache.lucene.analysis.Token token;???????????? String tokenText;???????????? int startOffset;???????????? int endOffset;???????????? int lastEndOffset = 0;???????????? textFragmenter.start(text);???????????? TokenGroup tokenGroup=new TokenGroup();???????????? token = tokenStream.next();???????????? while ((token!= null)(token.startOffset()maxDocBytesToAnalyze))????????????? {???????????????? if((tokenGroup.numTokens0)(tokenGroup.isDistinct(token)))????????????????? {???????????????????? //the current token is distinct from previous tokens -???????????????????? // markup the cached token group info???????????????????? startOffset = tokenGroup.matchStartOffset;???????????????????? endOffset = tokenGroup.matchEndOffset;???????????????????? tokenText = text.