【key】Perspectives on Crowdsourcing Annotations for NLP.pdfVIP

Perspectives on Crowdsourcing Annotations for Natural Language Processing1 Aobo Wang Cong Duy Vu Hoang Min-Yen Kan Computing 1, 13 Computing Drive National University of Singapore Singapore 117417 wangaobo,hcdvu,.sg July 24, 2010 1The authors gratefully acknowledge the support of the China-Singapore Institute of Digital Media’s support of this work by the “Co-training NLP systems and Language Learners” grant R 252-002-372-490. Abstract Crowdsourcing has emerged as a new method for obtaining annotations for training models for machine learning. While many variants of this process exist, they largely differ in their method of motivating subjects to contribute and the scale of their appli- cations. To date, however, there has yet to be a study that helps a practitioner to decide what form an annotation application should take to best reach its objectives within the constraints of a project. We first provide a faceted analysis of existing crowdsourc- ing annotation applications. We then use our analysis to discuss our recommendations on how practitioners can take advantage of crowdsourcing and discuss our view on potential opportunities in this area. 0.1 Introduction It is an accepted tradition in natural language processing (NLP) to use annotated cor- pora to obtain machine learned models for performing many tasks: machine trans- lation, parsing, and summarization. Given that machine learners can only perform tasks as good as their input annotation, much work in annotation centered on defining high quality standards that were reliable and reproducible, and finding appropriately trained personnel to carry out such tasks. The Penn Treebank and WordNet are prob- ably the most visible exampl