Workshop on Crowdsourcing
Crowdsourcing, a term coined by Jeff Howe, is when you take a large task, break it down into smaller - micro - pieces and send it out to a large group of people to perform each microtask in parallel for a monetary reward.
Microtask crowdsourcing traditionally covers a different set of scenarios. Tasks primarily rely on basic human abilities, including visual and audio cognition, as well as natural language understanding and communication (sometimes in different languages) and less on acquired skills (such as subject-matter knowledge). As such, a great share of the tasks addressed via microtask platforms like MTurk or CrowdFlower could be referred to as ‘routine’ tasks - recognizing objects in images, transcripting audio and video material and text editing.
To be more efficient than traditional outsourcing (or even in-house resources), the tasks need to be highly parallelized. This means that the actual work is executed by a high number of contributors in a decentralized fashion; this not only leads to significant improvements in terms of time of delivery, but also offers a means to cross-check the accuracy of the answers (as each task is typically assigned to more than one person) and reward the workers according to their performance and productivity.
In this workshop, you will not only learn the fundamentals of crowdsourcing but also get your hands dirty by designing and executing a task on CrowdFlower and getting and analyzing the results from the workers. Also, you will learn about the successful applications of crowdsourcing in different domains.
Programme
| Time | Topic |
|---|---|
| 09:00 - 09:15 | Welcome with coffee and tea and introduction organizers |
| 09:15 - 10:00 | Introduction to Human computation |
| 10:00 - 11:00 | Microtask crowdsourcing fundamentals 1 |
| 11:00 - 11:15 | Coffee and tea break |
| 11:15 - 12:30 | Hands-on 1: Designing a microtask on CrowdFlower |
| 12:30 - 13:30 | Lunch break |
| 13:30 - 14:30 | Microtask crowdsourcing fundamentals 2 |
| 14:30 - 15:30 | Hands-on 2: Executing a microtask on CrowdFlower |
| 15:30 - 16:00 | Coffee and tea break |
| 16:00 - 16:30 | Discussion of the results |
| 16:30 - 17:30 | Applications of crowdsourcing, summary and conclusion |
Prerequisite knowledge or skills required
None
Information
Learning objectives and goals
Learning Objectives:
Learn the fundamentals of human computation
Learn how to design and execute a hands-on crowdsourcing task on CrowdFlower
Learn the applications of crowdsourcing
Course Materials
Required materials and resources
Laptop (browser: Chrome, Firefox, Safari)
Course materials and resources
All slides are available online: https://github.com/MaastrichtU-IDS/Crowdsourcing-Workshop
Instructors
is a postdoctoral researcher at the Institute of Data Science, University of Maastricht. Her research interests include data integration, quality and analysis. She evaluated the feasibility of using crowdsourcing as a means of quality assessment for DBpedia (the structured version of Wikipedia). Now she is designing microtasks for employing workers towards quality assessment of biomedical metadata. The challenge is to design tasks such that non-experts, those who do not have a biomedical background, can contribute towards the quality assessment.
is a postdoctoral researcher at the Institute of Applied Informatics and Formal Methods (AIFB), Karlsruhe Institute of Technology (KIT), Germany. Her research interests include big data, query processing, and semantic web. Her work includes the combination of semantic web technologies with human-based computation to enhance the quality of knowledge graphs. In particular, she has applied microtask crowdsourcing to complete missing statements in knowledge graphs from different domains including Geography, History, and Life Sciences.