Wednesday, March 24, 2010

Voodoo and Type II: Debate between Piotr Winkielman and Matt Lieberman


"Voodoo correlations in social neuroscience" was the original title of a paper that first caused a stir in late December 2008, when a manuscript accepted by Perspectives on Psychological Science was made available on the authors' websites. Vul, Harris, Winkielman, and Pashler produced a "bombshell of a paper" that questioned the implausibly high correlations observed in some fMRI studies in the field of Social Neuroscience. Ed Vul et al. surveyed the authors of 54 papers to determine the analytic methods used. All but three of the authors responded to the survey, and 54% admitted to using faulty methods to obtain their results:
More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases.
For background reading I suggest starting with Voodoo Correlations in Social Neuroscience. Given the paper's inflammatory title and its naming of names, the accused researchers did not take the criticism lying down (see Voodoo Schadenfreude).

Here we have a public debate between Dr. Piotr Winkielman, one of the authors of the Voodoo paper (Vul et al. 2009, PDF), and Dr. Matthew Lieberman, one of the accused (rebuttal: Lieberman et al. 2009, PDF) at the 2009 meeting of the Society of Experimental Social Psychologists. Dr. Lieberman has made these videos and papers available on his website, and I thank him for drawing my attention to them.

The Voodoo Debate Continues...


video

Piotr Winkielman, opening remarks (21:12)



video

Matt Lieberman, opening remarks (19:03) [better view of slides]



video

Piotr Winkielman, rebuttal (10:22)



video

Matt Lieberman, rebuttal (9:40) [better view of slides]


Lieberman did strike back (Lieberman et al., 2009), and attacked Vul et al. for cherry picking their data and for inappropriate use of statistics:
However, they imply that post hoc reporting of correlations constitutes an invalid inferential procedure, when in fact it is a descriptive procedure that is entirely valid. In addition, the quantitative claims that give their arguments the appearance of statistical rigor are based on problematic assumptions. Thus, it is ironic that Vul et al.’s article—which critiques social neuroscience as having achieved popularity in prominent journals and the press due to shaky statistical reasoning—itself achieved popularity based on problematic claims about the process of
statistical inference.

Additional Reading:

Voodoo correlations in social brain studies

Voodoo Gurus

"Voodoo Correlations" in fMRI - Whose voodoo?

The paper formerly known as "Voodoo Correlations in Social Neuroscience"


References

Lieberman, M., Berkman, E., & Wager, T. (2009). Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul et al. (2009) Perspectives on Psychological Science, 4 (3), 299-307 DOI: 10.1111/j.1745-6924.2009.01128.x

Vul E, Harris C, Winkielman P, Pashler H (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.

Subscribe to Post Comments [Atom]

2 Comments:

At March 25, 2010 4:01 PM, Blogger Johan said...

If you only care about where, as Matt Lieberman insists here and elsewhere, why plot the correlation scatter in the first place? The fMRI blob conveys all the information to support an inference of 'where.'

I think this is the heart of this tired debate - it's not that the studies that Vul et al attacked really committed a statistical error in a strong sense of the word. It's more that in the write-up, a lot of them lean pretty heavy on some extremely clean correlation plots. The same problem arises anytime you report uncorrected statistics (e.g. T at the peak for a standard contrast) - it's just harder to get into Science with a T of 8, compared to a brain-behaviour correlation of .8 (because even a very naive audience can form the false impression that a correlation of .8 is 'big').

So really it's back to the old problem of sensationalism and over-selling your case, something that scientists in all disciplines often fall victim to. For instance, suppose you were to include the term 'voodoo' in your ms title, or if you make sure to include the studies that particularly annoy you in your critical meta analysis...

 
At March 28, 2010 4:32 PM, Anonymous Anonymous said...

But but... Where is our hero Vul? Shouldn't he be the one standing there defending the paper?

 

Post a Comment

Links to this post:

Create a Link

<< Home

eXTReMe Tracker