龙星计划课程信检索 Natural Language Processing.pptVIP

龙星计划课程:信息检索?? Natural Language Processing for IR Elements of Text Info Management Technologies Overview of NLP for IR Most work has focused on improving the indexing unit Syntactic phrases Word sense disambiguation Anaphora resolution Dependency parsing Sentiment retrieval Improving the Indexing Unit The Central Problem in IR Why is IR hard? IR is hard because natural language is so rich (among other reasons) What are the issues? Tokenization Morphological Variation Synonymy Polysemy Paraphrase Ambiguity Anaphora Possible Solutions Vary the unit of indexing Strings and segments Tokens and words Phrases and entities Senses and concepts Manipulate queries and results Term expansion Post-processing of results Tokenization What’s a word? First try: words are separated by spaces What about clitics? What about languages without spaces? Word-Level Issues Morphological variation = different forms of the same concept Inflectional morphology: same part of speech Derivational morphology: different parts of speech Synonymy = different words, same meaning Polysemy = same word, different meanings (e.g., root) Paraphrase Language provides different ways of saying the same thing Ambiguity What exactly do you mean? Why don’t we have problems (most of the time)? Ambiguity in Action Different documents with the same keywords may have different meanings… Anaphora Language provides different ways of referring to the same entity More Anaphora Terminology Anaphor = an expression that refers to another Anaphora = the phenomenon Other different types of referring expressions: Anaphora resolution can be hard! What can we do? Here are the some of the problems: Tokenization Morphological variation, synonymy, polysemy Paraphrase, ambiguity Anaphora General approaches: Vary the unit of indexing Manipulate queries and results What do we index? In information retrieval, we are after the concepts represented in the documents … but we can only index strings So what’s the best