Does the Science of Human Behavior Only Show Us What We Want to See? - Pacific Standard: The Science of Society

Why scepticism matters:

 “It’s only a scientific truth if it happens in the real world,” Simmons says. “Tell me what I have to do to replicate your findings, write me a recipe, and if you can’t do that—if you can only show this in Princeton undergraduates—then I don’t care anymore.”

This challenge is the idea behind the Reproducibility Project, a scientific collaboration directed by Brian Nosek, 41, a forceful and charismatic research psychologist at the University of Virginia. Nosek’s interest in replications partly began with what he describes as a beautiful experiment. He was testing whether political extremists—often described metaphorically as thinking in black and white—really do see the world that way. He recruited nearly 2,000 volunteers online, sorted them by political stance, and asked them to match two shades of gray. In what he calls a “stunning” result, political moderates were significantly better at this than those on the far right or left. The 60 Minutes segment practically writes itself. For a young researcher like Nosek, and his grad student, Matt Motyl, it was a coup.

Then they made what many would consider a fatal mistake. Even though they had no reason to doubt the result, Nosek and Motyl repeated the experiment—and nature, in its infinite capriciousness, took away what it had just given. “Our immediate reaction was ‘why the #&@! did we do a direct replication?’” they later wrote.

Nosek’s ordeal got him wondering: How would other researchers’ experiments stack up in a similar test? Thus was born the Reproducibility Project. The purpose of the effort is not to identify the flaws in other researchers’ work, Nosek emphasizes, but to arrive at a baseline estimate of how much of what we think we know about human psychology meets Bacon’s elementary test. In one sample of 13 experimental results chosen for replication, 10 were fully reproducible, one had ambiguous results, and two failed to replicate...

Does the Science of Human Behavior Only Show Us What We Want to See? - Pacific Standard: The Science of Society: