Voodoo and Type II: Debate between Piotr Winkielman and Matt Lieberman

"Voodoo correlations in social neuroscience" was the original title of a paper that first caused a stir in late December 2008, when a manuscript accepted by Perspectives on Psychological Science was made available on the authors' websites. Vul, Harris, Winkielman, and Pashler produced a "bombshell of a paper" that questioned the implausibly high correlations observed in some fMRI studies in the field of Social Neuroscience. Ed Vul et al. surveyed the authors of 54 papers to determine the analytic methods used. All but three of the authors responded to the survey, and 54% admitted to using faulty methods to obtain their results:

More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases.

For background reading I suggest starting with Voodoo Correlations in Social Neuroscience. Given the paper's inflammatory title and its naming of names, the accused researchers did not take the criticism lying down (see Voodoo Schadenfreude).

Here we have a public debate between Dr. Piotr Winkielman, one of the authors of the Voodoo paper (Vul et al. 2009, PDF), and Dr. Matthew Lieberman, one of the accused (rebuttal: Lieberman et al. 2009, PDF) at the 2009 meeting of the Society of Experimental Social Psychologists. Dr. Lieberman has made these videos and papers available on his website, and I thank him for drawing my attention to them.

The Voodoo Debate Continues...



Piotr Winkielman, opening remarks (21:12)





Matt Lieberman, opening remarks (19:03) [better view of slides]





Piotr Winkielman, rebuttal (10:22)





Matt Lieberman, rebuttal (9:40) [better view of slides]


Lieberman did strike back (Lieberman et al., 2009), and attacked Vul et al. for cherry picking their data and for inappropriate use of statistics:

However, they imply that post hoc reporting of correlations constitutes an invalid inferential procedure, when in fact it is a descriptive procedure that is entirely valid. In addition, the quantitative claims that give their arguments the appearance of statistical rigor are based on problematic assumptions. Thus, it is ironic that Vul et al.’s article—which critiques social neuroscience as having achieved popularity in prominent journals and the press due to shaky statistical reasoning—itself achieved popularity based on problematic claims about the process of
statistical inference.

Additional Reading:

Voodoo correlations in social brain studies

Voodoo Gurus

"Voodoo Correlations" in fMRI - Whose voodoo?

The paper formerly known as "Voodoo Correlations in Social Neuroscience"


References

Lieberman, M., Berkman, E., & Wager, T. (2009). Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul et al. (2009) Perspectives on Psychological Science, 4 (3), 299-307 DOI: 10.1111/j.1745-6924.2009.01128.x

Vul E, Harris C, Winkielman P, Pashler H (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.