信息检索7-西北工业大学.ppt

These associations were likely to happen because the non-doctor words shown here are very common and therefore likely to co-occur with any noun. Un-Interesting Associations with “Doctor” Keyword Correlation Keyword Correlation Factor: Keyword Score: * * Chen Qun School of Computer, NWPU 11/2016 Information Retrieval: Theory and Practice IR Components: Document Processing Today Components of IR systems Content Analysis Stemming Statistical Properties of Document collections Statistical Dependence Word Associations Content Analysis Automated Transformation of raw text into a form that represent some aspect(s) of its meaning Including, but not limited to: Automated Thesaurus Generation Phrase Detection Categorization Clustering Summarization Techniques for Content Analysis Statistical Single Document Full Collection Linguistic Syntactic Semantic Knowledge-Based (Artificial Intelligence) Hybrid (Combinations) Text Processing Standard Steps: Recognize document structure titles, sections, paragraphs, etc. Break into tokens usually space and punctuation delineated special issues with Asian languages Stemming/morphological analysis Store in inverted index Document Processing Steps Statistical Properties of Text Token occurrences in text are not uniformly distributed They are also not normally distributed They do exhibit a Zipf distribution Plotting Word Frequency by Rank Main idea: count How many tokens occur 1 time How many tokens occur 2 times How many tokens occur 3 times … Now rank these according to how of they occur. This is called the rank. Plotting Word Frequency by Rank Say for a text with 100 tokens Count How many tokens occur 1 time (50) How many tokens occur 2 times (20) … How many tokens occur 7 times (10) … How many tokens occur 12 times (1) How many tokens occur 14 times (1) So things that occur the most often share the highest rank (rank 1). Things that occur the fewest times have the lowest rank (rank n). Many similar distributions… Words in a