Adaptive Task Assignment for Crowdsourced Classification（众包分类的自适应任务分配）.pdfVIP

Adaptive Task Assignment for Crowdsourced Classification Chien-Ju Ho, Shahin Jabbari cjho@, shahin@ University of California, Los Angeles Jennifer Wortman Vaughan jenn@ Microsoft Research, New York City and University of California, Los Angeles Abstract tively) whether or not a particular website is offensive. Crowdsourcing markets have gained popular- The availability of diverse workers willing to complete ity as a tool for inexpensively collecting data tasks inexpensively makes crowdsourcing markets ap- from diverse populations of workers. Classifi- pealing as tools for collecting data (Wah et al., 2011). cation tasks, in which workers provide labels Classification tasks, in which workers are asked to pro- (such as “offensive” or “not offensive”) for in- vide a binary label for an instance, are among the most stances (such as “websites”), are among the common tasks posted (Ipeirotis, 2010). Unfortunately, most common tasks posted, but due to hu- due to a mix of human error, carelessness, and fraud man error and the prevalence of spam, the — the existence of spammy workers on Mechanical labels collected are often noisy. This problem Turk is widely acknowledged — the data collected is is typically addressed by collecting labels for often noisy (Kittur et al., 2008; Wais et al., 2010). For each instance from multiple workers and com- classification tasks, this problem can be overcome by bining them in a clever way, but the question collecting