前言_43525.pptVIP

Desiderata for Annotating Data to Train and Evaluate Bootstrapping Algorithms Outline The Bootstrapping Era Why Bootstrapping? Additional Benefits of Bootstrapping Outline Automatic Annotation with Seeding Seeding strategies Seed Nouns for Semantic Class Bootstrapping Seed Words for Word Sense Disambiguation [Yarowsky, 1995] Seed Patterns for Extraction Pattern Bootstrapping [Yangarber et al. 2000]  Seed Patterns for Relevant Region Classifier Bootstrapping [Patwardhan Riloff, 2007] Seed Rules for Named Entity Recognition Bootstrapping [Collins Singer, 1999] Seed Heuristics for Coreference Classifiers [Bean Riloff, 2005; Bean Riloff, 1999] Example Coreference Seeding Heuristics Seed Classifiers for Subjectivity Bootstrapping [Wiebe Riloff, 2005] Outline The Importance of Good Seeding General Criteria for Seed Data Common Pitfall #1 General Criteria for Seed Data Common Pitfall #2 General Criteria for Seed Data Common Pitfall #3 Bootstrapping a Single Category Bootstrapping Multiple Categories General Criteria for Seed Data Commont Pitfall #4 Outline Evaluation The Road Ahead: A Paradigm Shift? Need #1: Cross-Domain Evaluations Need #2: Seeding Robustness Evaluations Standardizing Seeding? If you annotate it, they will come Problem Child: Sentiment Analysis Conclusions Manual annotation efforts have primarily focused on the need for large amounts of training data. As the need for training data decreases, we have the opportunity to shift the focus to evaluation data. The NLP community has a great need for more extensive evaluation data! from both a practical and scientific perspective We would benefit from more analysis of cross-domain performance. Will an algorithm behave consistently across domains? Can we characterize the types of domains that a given technique will (or will not) perform well on? Are the learning curves similar across domains? Does the stopping criterion behave similarly across domains? Bootstrapping algorithms are sensitive to their