4-信息检索.ppt

* * * * * * * Text pre-processing Word (term) extraction: easy 中文处理要复杂很多：切词 Stopwords removal Stemming Frequency counts and computing TF-IDF term weights. Stopwords removal Many of the most frequently used words in English are useless in IR and text mining – these words are called stop words. the, of, and, to, …. Typically about 400 to 500 such words For an application, an additional domain specific stopwords list may be constructed Why do we need to remove stopwords? Reduce indexing (or data) file size stopwords accounts 20-30% of total word counts. Improve efficiency and effectiveness stopwords are not useful for searching or text mining they may also confuse the retrieval system. Stemming Techniques used to find out the root/stem of a word. E.g., user engineering users engineered used engineer using stem: use engineer Usefulness: improving effectiveness of IR and text mining matching similar words Mainly improve recall reducing indexing size combing words with same roots may reduce indexing size as much as 40-50%. Given a query: Are all retrieved documents relevant? Have all the relevant documents been retrieved? Measures for system performance: The first question is about the precision of the search The second is about the completeness (recall) of the search. false negative (false drop) - some relevant documents may not be retrieved. false positive - some irrelevant documents may be retrieved. Evaluation: Precision and Recall General form of precision/recall Precision change w.r.t. Recall (not a fixed point) Systems cannot compare at one Precision/Recall point Average precision (on 11 points of recall: 0.0, 0.1, …, 1.0) Precision-recall curve Compare different retrieval algorithms Concept-Based Querying Approach For each word, determine the concept it represents from context Use one or more ontologies: Hierarchical structure showing