Bots on Amazon’s task service may be souring psych studies

It’s not uncommon for scientists to use Amazon’s Mechanical Turk for research — they can quickly gather survey data in return for a small outlay to participants. Academics are quickly discovering the limits of the task service with the modern bot-happy internet, however. Numerous psychology researchers have reported a sharp rise in the number of junk responses to their questions, many of which appear to come from bots. The offenders not only spam nonsensical answers to questions (such as “nice” or “good”), but tend to originate from the same location. And while anti-bot measures like captchas are in use, they aren’t always reliable.

There’s a chance that these could be people copying and pasting text, or using some form of automation to find the surveys (which Amazon allows). However, Amazon wouldn’t provide evidence one way or the other. The company told Wired in a statement that it had “automated and manual mechanisms” to fight bots, but it wouldn’t comment on bot activity levels or specific incidents.

Whatever the cause, the psychology study problems underscore deeper issues with Mechanical Turk. Bots are a reality on the service, and Amazon doesn’t always appear to acknowledge them even after multiple reports. And then there’s Mechanical Turk’s low per-task pay. While that can help scientists worried that large rewards could skew results, it encourages completing tasks en masse rather than providing high-quality answers. It may be increasingly difficult to conduct meaningful surveys on Amazon’s service, at least without tossing out many of the responses.

Via: Wired, New Scientist

Source: (Max) Hui Bai

Post Author: martin

Martin is an enthusiastic programmer, a webdeveloper and a young entrepreneur. He is intereted into computers for a long time. In the age of 10 he has programmed his first website and since then he has been working on web technologies until now. He is the Founder and Editor-in-Chief of and Online Magazines. His colleagues appreciate him as a passionate workhorse, a fan of new technologies, an eternal optimist and a dreamer, but especially the soul of the team for whom he can do anything in the world.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.