The paper formerly known as "Voodoo Correlations in Social Neuroscience"

Voodoo no more!

The paper everyone loves (or loves to hate) has a new name.¹ Through a number of channels [The Chronicle of Higher Education via @vaughanbell, Ed Vul's website, and Neuroskeptic], The Neurocritic has learned that the "Voodoo Correlations" have been downgraded to mere "Puzzlingly High Correlations." The field of social neuroscience has been spared as well, because the full title of the paper is now "Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition" (PDF).

By now, most neuroimagers and cognitive neuroscientists have heard about that controversial (some would say inflammatory) paper by Ed Vul and colleagues, summarized in this post.² In the article, Vul et al. claimed that over half of the fMRI studies that were surveyed used faulty statistical techniques to analyze their data:

...using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample.

Needless to say, authors of the criticized papers were not pleased. Two rebuttals were released online shortly thereafter: one by Jabbi et al. (PDF) -- here's the response to that rebuttal -- and an invited reply by Lieberman et al. (PDF).

That was back in January, after the manuscript had been accepted for publication by Perspectives on Psychological Science in late December 2008. Now [finally], the paper has been officially published in the May 2009 issue of the journal, with an introduction (PDF) by Ed Diener, the editor. Also included are six Commentaries by assorted authors and a Reply to the Commentaries by Vul et al. (PDF).

I haven't had time to read all the commentaries and rebuttals yet, but the Editor's Introduction is worth a quick mention for the issues it raises about peer review and publication in these modern times.

PREPUBLICATION DISSEMINATION

As soon as I accepted the Vul et al. article, I heard from researchers about it. People around the globe saw the article on the Internet, and replies soon appeared as well. Although my plan was to publish the article with commentary, the appearance of the article on the Internet meant that researchers read the article without the accompanying commentaries and replies that I had planned to publish with it.

In some fields such as economics, it is standard practice to widely disseminate articles before they are published, whereas in much of psychology this has been discouraged. An argument in favor of dissemination is that it speeds scientific communication in a fast-paced world where journal publication is often woefully slow. An argument against dissemination of articles before publication is that readers do not have the opportunity to simultaneously see commentary and replies. ... In the Internet age, the issue of prepublication distribution becomes all the more important because an article can reach thousands of readers in a few hours. Given the ability of the Internet to communicate so broadly and quickly, we need greater discussion of this issue.

Bloggers have discussed this specific issue months ago. For example, as noted in Mind Hacks,

The paper was accepted by a peer-reviewed journal before it was released to the public. The idea that something actually has to appear in print before anyone is allowed to discuss it seems to be a little outdated (in fact, was this ever the case?).

And The Neurocritic opined that...

[The aggrieved authors] are not keeping up with the way that scientific discourse is evolving. Citing "in press" articles in the normal academic channels is a frequent event; why should bloggers, some of whom are read more widely than the authors' original papers, refrain from such a practice? Is it the "read more widely" part?

...and in The Voodoo of Peer Review I asked:

Are blogs good or bad for the enterprise of scientific peer review? At present, the system relies on anonymous referees to provide "unbiased" opinions of a paper's (or grant's) merits. For today, the discussion will focus on peer review of papers in scientific journals...

[An] article [in Seed Magazine] begins:

"Few endeavors have been affected more by the tools and evolution of the internet than science publishing. Thousands of journals are available online, and an increasing number of science bloggers are acting as translators, often using lay language to convey complex findings previously read only by fellow experts within a discipline. Now, in the wake of a new paper challenging the methodology of a young field, there is a case study for how the internet is changing the way science itself is conducted."

Really? Maybe that's true for Biological and Social Sciences, but certainly not for Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance and Statistics (see arXiv.org)...

Diener then raises the point that online bloggers and commenters may be discussing various versions of the manuscript:

Another problem that has arisen in terms of Internet “publication” of the article and the Internet replies is that different individuals will have read different versions of the article. A single reader is unlikely to read more than one version of the article and will therefore often not see later corrections and changes. Furthermore, the commentaries are to some extent replies to different versions of the article and therefore might not be entirely on-target for the final version. This makes it difficult to fully understand the arguments because comments and replies might not be to the most current versions of articles, and it is impossible to fully correct this because the back-and-forth of revisions could continue indefinitely.

So there's never a final version of the article because revisions continue indefinitely?? Or are the accepted and final versions of the manuscript so radically different [why, I might ask] that a discussion of the initially accepted version is misleading? Or instead, is it the online commenters who are "revising" the article ad infinitum? Will Diener's editorial be clarified in a future edition, thus rendering moot my confusion in this particular post?

At any rate, Diener also discusses ethical issues surrounding the questionnaire that Vul et al. distributed to the authors. Some believed they were unwitting participants in Human Subjects research and did not give their informed consent (Diener disagreed). Not surprisingly, the "article tone" was another source of contention, and here Diener agreed to change the original "Voodoo" title. Finally, some of the aggrieved authors disputed the accuracy of the entire paper, suggesting that some (if not all) of their research was incorrectly classified. But in the end, the editor defers to the readers, who will judge the article and comments and form their own opinions.

I believe that the debate can itself stimulate useful discussions about scientific practices and communication. Further discussion of the issues should now take place in journals that are focused on imaging and neuroscience, so that the readers there can judge and benefit from the ensuing discussions.

I believe that further discussion of the issues can also take place on blogs that are focused on imaging and neuroscience. So feel free to discuss at length. Leave your questions and observations in the comments section of this post!

Footnotes

¹ See The Voodoo of Peer Review for a preview of this issue.

² You can also read a quick overview at Scan Scandal Hits Social Neuroscience, and more in-depth commentary in the post Voodoo Schadenfreude. And a comprehensive list of links about the the paper is located here.



Ed Diener (2009). Editor's Introduction to Vul et al. (2009) and Comments. Perspectives on Psychological Science, 4 (3).

Complete List of References

(from PERSPECTIVES ON PSYCHOLOGICAL SCIENCE, Vol. 4, Issue No. 3 · May 2009)

Editor's Introduction to Vul et al. (2009) and Comments
Ed Diener

Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler

Commentary on Vul et al.'s (2009) "Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition"
Thomas E. Nichols and Jean-Baptist Poline

Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power--Commentary on Vul et al. (2009)
Tal Yarkoni

Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul et al. (2009)
Matthew D. Lieberman, Elliot T. Berkman, and Tor D. Wager

Discussion of "Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition" by Vul et al. (2009)
Nicole A. Lazar

Correlations and Multiple Comparisons in Functional Imaging: A Statistical Perspective (Commentary on Vul et al., 2009)
Martin A. Lindquist and Andrew Gelman

Understanding the Mind by Measuring the Brain: Lessons From Measuring Behavior (Commentary on Vul et al., 2009)
Lisa Feldman Barrett

Reply to Comments on "Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition"
Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler