DataTerminology.comNEW
Also known as: Crowd intelligence
Obtaining data, labels, ideas, or solutions from a large, distributed group through open calls or marketplaces.
DESCRIPTION
Crowdsourcing harnesses collective effort to perform tasks like labeling data, solving problems, or generating designs. Platforms such as Amazon Mechanical Turk enable microtasks with scalable workforce access. Quality assurance relies on redundancy, consensus, gold questions, and worker qualification. Ethical practice requires fair compensation, privacy protection, and minimizing bias. Crowdsourcing complements expert review when volume or diversity of input is needed.
ALTERNATIVE DEFINITIONS
Wikipedia
Crowdsourcing involves outsourcing tasks to a distributed group of people.
Concise formulation emphasizing distribution of work to many contributors.
Wired (Jeff Howe)
Simply defined, crowdsourcing represents the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.
Original popularization highlighting open call and undefined large group.
Amazon Mechanical Turk
Amazon Mechanical Turk (MTurk) is a crowdsourcing marketplace that enables individuals and businesses to hire distributed workforce to perform tasks that computers are currently unable to do.
Operational definition focusing on marketplace facilitating human tasks.
RELATED TERMS
Data Labeling Human-in-the-Loop Mechanical Turk Citizen Science Open Innovation Quality Control Consensus Labeling Microtasks Gamification Labeling Platform
COMPETING TERMS
Expert Curation
KNOWLEDGE AREAS
Data Management Data Science AI Governance Data Privacy & Compliance Collaboration
TARGET AUDIENCE
AI Product Manager Data Scientist Data Steward Data Governance Manager Data Management Consultant
EXTERNAL SOURCES
- Wikipedia - Crowdsourcing
- Amazon Mechanical Turk - AWS guide to crowdsourcing
- Google Cloud - Managed data labeling service
- Labelbox - Labeling platform documentation
TOP QUESTIONS
Why use Crowdsourcing for data annotation?
It scales labeling quickly and inexpensively, providing diverse judgments to improve dataset robustness and coverage.
Category: Relevancy
How is quality controlled in Crowdsourcing?
Use gold standards, redundancy, consensus algorithms, worker qualifications, and spot checks to reduce noise and bias.
Category: Requirements
What are ethical concerns with Crowdsourcing?
Fair pay, transparency, privacy, and avoidance of exploitative tasks are core responsibilities when engaging crowd workers.
Category: Risks
What tasks suit Crowdsourcing?
Microtasks like image labeling, text classification, sentiment analysis, and data cleansing are commonly crowdsourced.
Category: Example
What’s a common mistake in Crowdsourcing?
Skipping pilot tests and quality controls can lead to low-quality outputs and wasted budget.
Category: Mistakes
How does Crowdsourcing compare to Expert Curation?
Crowds provide scale and diversity; experts deliver depth and accuracy. Many workflows blend both.
Category: Opinion
How to reduce bias in crowdsourced labels?
Diversify contributors, randomize tasks, and monitor inter-annotator agreement to detect skew.
Category: Requirements
What is the risk of poor task design in Crowdsourcing?
Ambiguous instructions increase variance, reduce reliability, and may introduce systemic errors.
Category: Risks
Where to start with Crowdsourcing?
Begin with a small pilot on MTurk or a labeling platform, define clear guidelines, and evaluate quality metrics before scaling.
Category: Learning