Voodoo Correlations: Two Years Later

The end of 2008 brought us the tabloid headline, Scan Scandal Hits Social Neuroscience. As initially reported by Mind Hacks, a new "bombshell of a paper" (Vul et al., 2009) questioned the implausibly high correlations observed in some fMRI studies in Social Neuroscience. A new look at the analytic methods revealed that over half of the sampled papers used faulty techniques to obtain their results.

-from Voodoo Correlations in Social Neuroscience, by The Neurocritic

The paper by Vul, Harris, Winkielman, and Pashler made its initial appearance on Ed Vul's website once it was accepted for publication by Perspectives in Psychological Sciences. Originally titled "Voodoo Correlations in Social Neuroscience", it was eventually renamed the more globally palatable "Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition" at the request of the editor (after much consternation from the criticized authors). This paper sparked intense debate in the field of functional neuroimaging, much of which occurred in the blogosphere (at least initially).¹

Are blogs good or bad for the enterprise of scientific peer review? At present, the system relies on anonymous referees to provide "unbiased" opinions of a paper's (or grant's) merits. For today, the discussion will focus on peer review of papers in scientific journals.

-from The Voodoo of Peer Review, by The Neurocritic

Many of the aggrieved researchers in the neuroimaging community were appalled that bloggers were discussing Vul's accepted paper before it was "properly" published (and before they had time to comment themselves). But two research groups quickly issued replies:

Two rebuttals were released online shortly thereafter: one by Jabbi et al. (PDF) and an invited reply by Lieberman et al. (PDF).

What's the problem here? It's that bloggers were writing about it! That authors and anonymous commenters somehow sullied their ideological purity by entering the free-wheeling, fast-moving world of the blogosphere. But in the modern era, why wait 5 months for a paper to be "officially" published before you're allowed to discuss it? And despite what the critics of Voodoo say, Vul et al.'s paper was not plastered all over the popular press (unlike many of the Voodoo findings themselves)...

The only other mainstream media exposure has been from Sharon Begley of Newsweek, who covered the issue in her blog (i.e., The 'Voodoo' Science of Brain Imaging and More on Brain Voodoo) and in one of her magazine columns. But many are dubious. According to Seed [That Voodoo That Scientists Do]:

Two groups of neuroimaging scientists, shocked by the speed with which this paper was being publicly disseminated, wrote rebuttals and posted them in the comments section of several blogs, including Begley's. Vul followed up in kind, linking to a rebuttal of the rebuttals in the comment sections of several blogs. This kind of scientific discourse — which typically takes place in the front matter of scholarly journals or over the course of several conferences — developed at a breakneck pace, months before the findings were officially published, and among the usual chaos of blog comments: inane banter, tangents, and valid opinions from the greater public.

The usual chaos of blog comments? Hello?? How about anonymous referees for journals? Are they never ever guilty of reviews filled with inane banter and tangents? We've all had exposure -- whether from our bosses, advisors, or colleagues or through our own experience -- to rude and nasty and ill-informed reviewers. And many journal editors do not rein them in. The Neurocritic has been a proponent of completely open peer review, where the identity of the authors and the reviewers is known (see Anonymous Peer Review Means Never Having to Say You're Sorry, Peer Review Trial and Debate at Nature, and Double-Blind Bind). That way, Dr. Nasty can't hide behind the shield of anonymity when making those dumb-ass comments.

-from The Voodoo of Peer Review, by The Neurocritic

This issue is relevant again today because of the fallout over the infamous arsenic paper (Wolfe-Simon et al., 2010), which claimed it had isolated a bacterium that can substitute arsenic for phosphorus. Publication of the paper was preceded by a Sphinxlike press release: "NASA will hold a news conference at 2 p.m. EST on Thursday, Dec. 2, to discuss an astrobiology finding that will impact the search for evidence of extraterrestrial life." After the paper appeared online in Science, negative reactions from qualified and prominent scientists were swift. One of the most visible (and withering) critiques was written by Dr. Rosie Redfield, a microbiologist at the University of British Columbia:

Arsenic-associated bacteria (NASA's claims)

Here's a detailed review of the new paper from NASA claiming to have isolated a bacterium that substitutes arsenic for phosphorus on its macromolecules and metabolites. ... Basically, it doesn't present ANY convincing evidence that arsenic has been incorporated into DNA (or any other biological molecule).

. . .

Bottom line: Lots of flim-flam, but very little reliable information. The mass spec measurements may be very well done (I lack expertise here), but their value is severely compromised by the poor quality of the inputs. If this data was presented by a PhD student at their committee meeting, I'd send them back to the bench to do more cleanup and controls.

CBC News covered the critical backlash and NASA's reply, which was anti-blog:

NASA's arsenic microbe science slammed

. . .

Debate shouldn't be in media: NASA

When NASA spokesman Dwayne Brown was asked about public criticisms of the paper in the blogosphere, he noted that the article was peer-reviewed and published in one of the most prestigious scientific journals. He added that Wolfe-Simon will not be responding to individual criticisms, as the agency doesn't feel it is appropriate to debate the science using the media and bloggers. Instead, it believes that should be done in scientific publications.

Redfield said the reason she posted the review on her blog is partly because scientific publications such as Science — and the debates therein — are typically behind a paywall and inaccessible to the public.

"I blog openly…to bring this stuff more into the open where everybody can see it," she said.

Redfield has now posted a draft of her official letter to Science.

For full coverage of the matter, I recommend Is That Arsenic-Loving Bug — Formerly an Alien — a Dog? and The Wrong Stuff: NASA Dismisses Arsenic Critique Because Critical Priest Not Standing on Altar by David Dobbs, "This Paper Should Not Have Been Published" by Carl Zimmer, and An arsenic bacteria postmortem: NASA responds, tries to pit blogs vs. “credible media organizations” by Ivan Oransky (for starters).


Returning now to Voodoo Correlations... In the November 2010 issue of Perspectives in Psychological Sciences (online December 7, 2010), outgoing editor Ed Diener has assembled an fMRI Special Section looking back at those heady days and forward into the future:

Neuroimaging: Voodoo, New Phrenology, or Scientific Breakthrough? Introduction to Special Section on fMRI

Ed Diener

In response to the widespread interest following the publication of Vul et al (2009), Perspectives Editor Ed Diener invited researchers to contribute articles for a special section on fMRI, discussing the promises and issues facing neuroimaging.

Mistreating Psychology in the Decades of the Brain

Gregory A. Miller

Scientists tend to consider psychology-biology relationships in two distinct ways: by assuming that psychological phenomena can be fully explained in terms of biological events and by treating them as if they exist in separate realms. These approaches hold up scientific progress and have important implications for clinical practice and policy decisions (e.g., allocating research funds).

Brain Imaging, Cognitive Processes, and Brain Networks

Brian D. Gonsalves and Neal J. Cohen

The growth of neuroimaging research has led to reflection on what those techniques can actually tell us about cognitive processes. When used in combination with other cognitive neuroscience methods, neuroimaging has promise for making important advancements. For example, neuroimaging studies on memory have raised questions not only about the regions involved with memory but also about component cognitive processes (e.g., the role of different attention subsystems in memory retrieval), and this has resulted in more theorizing about the interactions of memory and attention.

Mapping Mental Function to Brain Structure: How Can Cognitive Neuroimaging Succeed?

Russell A. Poldrack

To understand the anatomy of mental functions, researchers may to need to move away from commonly used brain mapping strategies and begin searching for selective associations. This will require more emphasis on the structure of cognitive processes, which may be achieved through development of formal ontologies (e.g., the Cognitive Atlas) that will describe the "parts" and processes of the mind. Using these ontologies in combination with large-scale data mining approaches may more directly relate mental processes and brain function.

The Appeal of the Brain in the Popular Press

Diane M. Beck

Why do people like the brain so much? Brain-related articles in the press, especially ones about fMRI research, tend to be very popular with the general public, but many of these articles may result in misinterpretations of the science. Part of the popularity may be attributed to their deceptively simple message: Perform an action and a certain area lights up. In addition, people are more confident in "biological" images than in the behavioral phenomena on which the images are based. In order to maintain trust with the public, scientists have a responsibility to provide the press with descriptions of research and interpretations of results research that are clear, relevant, and scientifically accurate.

Frontiers in Human Neuroscience: The Golden Triangle and Beyond

Jean Decety and John Cacioppo

The development of neuroimaging has created an opportunity to address old questions about brain function and behavior in new ways and also to uncover new questions. The knowledge that emerges from neuroimaging studies is more likely to be beneficial when combined with techniques and analyses that break down complex constructs into structures and processes, measures that gauge neural events across different times, and animal studies.

Bridging Psychological and Biological Science: The Good, Bad, and Ugly

Arthur P. Shimamura

The advent of functional neuroimaging has brought both praise and criticism to the field of psychological science. Although most studies relying on fMRI are correlative, they do offer some clues about the biology underlying psychological processes. However, it is not sufficient to show which area of the brain is involved in a particular cognitive process; rather theories need to address "how?" questions (e.g., How does the hippocampus contribute to remembering?) in order to best bridge psychological and biological science.



Footnote

¹ Also see Mind Hacks, BPS Research Digest, and Neuroskeptic.

References

Vul E, Harris C, Winkielman P, Pashler H (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science 4(3), 274-290.

Wolfe-Simon F, Blum JS, Kulp TR, Gordon GW, Hoeft SE, Pett-Ridge J, Stolz JF, Webb SM, Weber PK, Davies PC, Anbar AD, & Oremland RS (2010). A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus. Science.

Thanks to Sandra of Channel N for alerting me to the special issue.