Institute of Data Science

Crowdsourcing, a term coined by Jeff Howe, is when you take a large task, break it down into smaller - micro - pieces and send it out to a large group of people to perform each microtask in parallel for a monetary reward. Microtask crowdsourcing traditionally covers a different set of scenarios. Tasks primarily rely on basic human abilities, including visual and audio cognition, as well as natural language understanding and communication (sometimes in different languages) and less on acquired skills (such as subject-matter knowledge). As such, a great share of the tasks addressed via microtask platforms like MTurk or CrowdFlower could be referred to as ‘routine’ tasks – recognizing objects in images, transcripting audio and video material and text editing. To be more efficient than traditional outsourcing (or even in-house resources), the tasks need to be highly parallelized. This means that the actual work is executed by a high number of contributors in a decentralized fashion; this not only leads to significant improvements in terms of time of delivery, but also offers a means to cross-check the accuracy of the answers (as each task is typically assigned to more than one person) and reward the workers according to their performance and productivity. In this workshop, you will not only learn the fundamentals of crowdsourcing but also get your hands dirty by designing and executing a task on CrowdFlower and getting and analyzing the results from the workers. Also, you will act as a worker on a live crowdsourcing task on annotating medical images. Finally, you will learn about the successful applications of crowdsourcing in different domains.