Web信息体系结构 Text Processing.pptVIP

Text Processing Indexing Goal Converting text to indexing terms Goal: produce a set of indexing terms that make the best use of resources, good representing the text that will accurately match user query terms Manual vs. Automatic Indexing Manual or human indexing: Indexers decide which keywords to assign to document based on controlled vocabulary e.g. OPAC(图书查询检索系统 ), sohu/yahoo/ODP Significant cost Automatic indexing: Indexing program decides which words, phrases or other features to use from text of document Indexing speeds range widely Recent 40GB/hour processing speed at TREC Manual vs. Automatic Indexing Manual vs. Automatic Indexing Experimental evidence is that retrieval effectiveness using automatic indexing can be at least as effective as manual indexing with controlled vocabularies original results were from the Cranfield experiments in the 60s considered counter-intuitive other results since then have supported this conclusion broadly accepted at this point Experiments have also shown that using both manual and automatic indexing improves performance “combination of evidence” Basic Automatic Indexing Parse documents to recognize structure e.g. title, date, other fields maybe very difficult for web pages, e.g. what can be searched? (what to look for? ↓) Scan for word tokens numbers, special characters, hyphenation, capitalization, etc. languages like Chinese need segmentation record positional information for proximity operators Stopword removal based on short list of common words such as “the,”to”,”or” saves storage overhead of very long indexes can be dangerous (e.g. “to be or not to be” ) used in SE? not in indexing time but in query processing (why?) Basic Automatic Indexing Stem words morphological processing to group word variants such as plurals better than string matching (e.g. comput*) can make mistakes but generally preferred not done by most Web search engines (why?) Weight words want more “important” words to have higher weight using freq