Mechanical Turk: what's it all about? · Dec 3, 05:06 PM by Ken King
I first became aware of Amazon’s Mechanical Turk service when it was mentioned in this article about peer production on Union Square Ventures’ blog. We thought it might be an interesting way to source labour for our clients’ transcription needs, so I set out to experiment with it a couple of weeks ago.
I looked around for a bit, but couldn’t find any HITs (Human Intelligence Tasks, their term for things that humans can do better than computers) other than those posted for A9, which consist of photo recognition tasks. Essentially, you are shown five pictures and asked to identify which one best represents the business name or street address listed at the top of the page. The promised payment for successful completion of this task is 3ยข U.S.
I put in about 10 minutes my first time out, and got mixed results. A large proportion (almost half) of my submitted tasks were rejected, and I spent a lot of time trying to scrutinize some pretty inscrutable pictures. I figured out some simple rules to get more efficient about it, though, and took them into my next work session:
- Preview the task
Before accepting a HIT, look over the images to see if the “right” answer will be easy to discern. If not, skip to the next HIT. - Focus on retail businesses
It’s a lot easier to recognize and identify a photo showing a retailer’s storefront than it is to look for a street address for a dentist’s office in suite 123 at 4536 Main Street. The pictures on A9 were shot with a video camera out the side of a moving vehicle, so reading street addresses is a pain in the ass and a waste of time. - Skip HITs with multiple “right” answers
The algorithm for determining the correct answer is a majority rules process based on multiple users’ responses. Working in that framework, you’re not just trying to find the right answer, you’re trying to game the system by figuring out which answer most people would choose. You’re penalized for “wrong” answers both by having wasted your time answering them, and in your qualification rating which might prevent you from getting access to more lucrative gigs down the road.
The results
With those thoughts in mind, I committed a solid half-hour to working at these image recognition tasks as fast as I could. Over that time, I managed to submit 25 HITs, of which 7 ended up being rejected despite my best efforts to avoid any ambiguous photos. Even if I managed to eliminate the rejected submissions, 50 HITs/hour = $1.50/hour.
At that rate, I can’t be bothered working on a larger sample size – has anyone else got any experience to share?
UPDATE
Okay, apparently I suck. ;-)
The users of this mturk board are getting to as many as 800 HITs in an hour, with reasonably small rejection rates. I bow to their superior skills and tools (there’s much talk in the forum about a script that speeds up the processing), but even at that blistering pace the pay is $24/hr.
