Clik here to view.
Wisdom of small crowds, part 1: how to aggregate Turker judgments for...
[ This article is part of a series, Wisdom of Small Crowds, which focuses on crowdsourcing methodology for Amazon Mechanical Turk-like systems. ] We use Turkers to classify all sorts of data, by having...
View ArticleClik here to view.
Wisdom of small crowds, part 2: individual workloads and rates
[ Update: see also another visualization of this. ] AMT’s great new interface makes it easy to download completion times for individual worker assignments. Therefore, it’s easy to visualize :) For a...
View ArticleClik here to view.
Wisdom of small crowds, part 3: another worker visualization
This is a follow-up to the previous post on individual workloads and rates. Here are the submission times and durations for every worker on the same graph. Each worker is one horizontal line. An...
View ArticleClik here to view.
AMT is fast, cheap, and good for machine learning data
Update 9/19: Final PDF version has been uploaded. See also the comments below for updates — our released data is already being used by others! We recently teamed up with Rion Snow, Prof. Dan Jurafsky,...
View ArticleEMNLP Slides
Rion Snow presented the paper, “Cheap and Fast – But is it Good?” at EMNLP last week. Here are the slides from the talk: Rls For Emnlp 2008 View SlideShare presentation or Upload your own.
View ArticleClik here to view.
Crowdsifter: More Efficient Content Filtering
“I know it when I see it.” — Justice Potter Stewart We have been running Crowdsifter, our content moderation product backed by Amazon’s Mechanical Turk for a while and we wanted to share some quality...
View ArticleClik here to view.
Visions and revisions
Writing is easy. Just sit in front of a typewriter, open up a vein and bleed it out drop by drop. – Red Smith Underwood Five Typewriter When I was in college, a professor I respected said that one of...
View ArticleClik here to view.
Good work knows no boundaries
Can the quality of crowdsourced work be linked to geography? In short, the answer is no. Here’s why. Our team looked at whether including a workforce from a specific region helped or hurt the...
View ArticleClik here to view.
Oscar Fever: The Sequel!
The votes are in from our Oscar crowdsourcing experiment, and the crowd successfully picked the winners of 14 of the academy awards. For reference, Roger Ebert got 15 predictions correct so we’d have...
View Article
