Voodoo Counterpoint

In the spirit of American political debate shows such as Crossfire, The McLaughlin Group, Hannity & Colmes, and the classic Point/Counterpoint (both the 60 Minutes and SNL versions), The Neurocritic is pleased to present an excerpt from a rebuttal to the lively and controversial paper by Vul, Harris, Winkielman, and Pashler (PDF).

Two "anonymous commenters" tipped me off to the preliminary version of a detailed reply by some of the authors on the Vul et al. hit list. The entire document is available for download as a PDF. The abstract, main bullet points, and conclusions are reproduced below.

Rebuttal of "Voodoo Correlations in Social Neuroscience" by Vul et al. – summary information for the press

Mbemba Jabbi¹, Christian Keysers², Tania Singer³, Klaas Enno Stephan³,
(authors listed in alphabetical order)

¹ National Institute of mental Health, Bethesda, Maryland, USA.

² University Medical Center Groningen, Department of Neuroscience, University of Groningen, The Netherlands. www.bcn-nic.nl/socialbrain.html

³ Laboratory for Social and Neural Systems Research, University of Zurich, Switzerland. http://www.socialbehavior.uzh.ch/index.html

The paper by Vul et al., entitled "Voodoo correlations in social neuroscience" and accepted for publication by Perspectives on Psychological Science, claims that "a disturbingly large, and quite prominent, segment of social neuroscience research is using seriously defective research methods and producing a profusion of numbers that should not be believed." In all brevity, we here summarise conceptual shortcomings and methodological errors of this paper and explain why their criticisms are invalid. A detailed reply will be submitted to a peer reviewed scientific journal shortly.

1. The authors misunderstand the critical role of multiple comparison corrections and conflate issues pertaining to null hypothesis testing and effect size estimates, respectively.

2. The authors make strong claims on the basis of a questionable upper bound argument.

3. The authors use misleading simulations to support their claims.

4. The authors inappropriately dismiss the existence of non-significant correlations.

5. The authors' understanding of the rationale behind the use and interpretation of correlations in social neuroscience is incomplete.

6. The authors ignore that the same brain-behaviour correlations have been replicated by several independent studies and that major results in social neuroscience are not based on correlations at all.

7. The authors used an ambiguous and incomplete questionnaire.

8. The authors make flawed suggestions for data analysis.

. . .

Conclusions

In this summary, we have provided a very brief summary that exposes some of the flaws that undermine the criticisms by Vul et al. We have pointed out that brain-behaviour correlations in social neuroscience are valid, provided that one adheres to good statistical practice. It has also been emphasized that many analyses and findings in social neuroscience do not rest on brain-behaviour correlations and have been replicated several times by independent studies conducted by different laboratories. A full analysis of the Vul et al. paper and a detailed reply will be submitted to a peer-reviewed scientific journal shortly.
_____________________________________________________

A rebuttal to the rebuttal, along with commentary by The Neurocritic, all to come in the next exciting episode!

But for now, you can watch Colbert & Colmes discuss Roland Burris, who was appointed to replace Barack Obama as junior senator of Illinois.