A Data Driven Approach to Query Expansion in Question.ppt

A Data Driven Approach to Query Expansion in Question Answering Leon Derczynski, Robert Gaizauskas, Mark Greenwood and Jun Wang Natural Language Processing Group Department of Computer Science University of Sheffield, UK Summary Introduce a system for QA Find that its IR component limits system performance Explore alternative IR components Identify which questions cause IR to stumble Using answer lists, find extension words that make these questions easier Show how knowledge of these words can make rapidly accelerate the development of query expansion methods Show why one simple relevance feedback technique cannot improve IR for QA How we do QA Question answering system follows a linear procedure to get from question to answers Pre-processing Text retrieval Answer Extraction Performance at each stage affects later results Measuring QA Performance Overall metrics Coverage Redundancy TREC provides answers Regular expressions for matching text IDs of documents deemed helpful Ways of assessing correctness Lenient: the document text contains an answer Strict: further, the document ID is listed by TREC Assessing IR Performance Low initial system performance Analysed each component in the system Question pre-processing correct Coverage and redundancy checked in IR part IR component issues Only 65% of questions generate any text to be prepared for answer extraction IR failings cap the entire system performance Need to balance the amount of information retrieved for AE Retrieving more text boosts coverage, but also introduces excess noise Initial performance Lucene statistics Using strict matching, at paragraph level Potential performance inhibitors IR Engine Is Lucene causing problems? Profile some alternative engines Difficult questions Identify which questions cause problems Examine these: Common factors How can they be made approachable? Information Retrieval Engines AnswerFinder uses a modular framework, including an IR plugin for Lucene Indri and