Thursday, January 15, 2009

Voodoo Schadenfreude


Voodoo doll, by Sickboy

Most hip researchers in cognitive neuroscience and human brain imaging have already heard about the critical new journal article with the incendiary title: "Voodoo Correlations in Social Neuroscience" (Vul et al., in press - PDF). If you haven't, you can read a comprehensive summary here and a micro version here.

Avenging Voodoo Schadenfreude

Nature News ran a piece on the debate and the burgeoning backlash from an angry mob of researchers whose methods were derided as fatally flawed. Some of these authors (and perhaps some Nature editors) were miffed that bloggers wrote about the preprint when it was first made available to the public, as if that somehow violates the scientific method:
The swift rebuttal was prompted by scientists' alarm at the speed with which the accusations have spread through the community. The provocative title — 'Voodoo correlations in social neuroscience' — and iconoclastic tone have attracted coverage on many blogs, including that of Newsweek. Those attacked say they have not had the chance to argue their case in the normal academic channels.

"I first heard about this when I got a call from a journalist," comments neuroscientist Tania Singer of the University of Zurich, Switzerland, whose papers on empathy are listed as examples of bad analytical practice. "I was shocked — this is not the way that scientific discourse should take place." Singer says she asked for a discussion with the authors when she received the questionnaire, to clarify the type of information needed, but got no reply.
Based on the statements above, it would seem that Dr. Singer and her colleagues (Jabbi, Keysers, and Stephan) are not keeping up with the way that scientific discourse is evolving. [See Mind Hacks on this point as well.] Citing "in press" articles in the normal academic channels is a frequent event; why should bloggers, some of whom are read more widely than the authors' original papers, refrain from such a practice? Is it the "read more widely" part? To their credit, however, they commented in blogs and publicized the link to a preliminary version of their detailed reply.....although calling it "summary information for the press" assumes that "the press" is extremely knowledgeable about neuroimaging methodology and statistical analysis.

To learn more about the evolution of scientific discourse, let me briefly introduce you to the world of social media (e.g., FriendFeed, Facebook, and even Twitter). You can join the discussion at FriendFeed's Science 2.0 Room, which is "For people interested in Science 2.0 and Open Science, especially the use of online tools to do science in new ways." Although one needs a Facebook account to view these, Facebook groups include Neuroscience ROCKS (4,006 members), Neuroscience and Brain Studies (3,194 members), and Cognitive Neuroscience Society (2,147 members). Some social neuroscientists are, well, social enough to get it, because Columbia University Social Cognitive Affective Neuroscience Lab has a posse (79 fans, albeit inactive ones). And believe it or not, NIH now has Twitter feeds for press releases and funding announcements. As for individuals who are powerhouse resources on the future of scientific communication, I would recommend reading BoraZ (blog, Twitter), who is organizing the Jan. 16-18 ScienceOnline’09 conference, and Björn Brembs on scientific publishing and misuse of the Impact Factor.

All is not puppies and flowers in the world of science social media, however. Proponents rarely acknowledge that many companies and institutions block access to these sites, so at present their usefulness is limited for many in the scientific community. A more obvious issue is that these sites can turn into an enormous time sink.

Now back to Nature News and the voodoo backlash. In an ironic twist, one of the 'red listed' papers (Singer et al., 2006), published in Nature, was publicized as a study on Schadenfreude. Here's the Editor's Summary [which was covered by The Neurocritic three years ago]:
I feel your pain

Humans have the capacity to empathize with the pain of others, but we don't empathize in all circumstances. An experiment on human volunteers playing an economic game looked at the conditional nature of our sympathy, and the results show that fairness of social interactions is key to the empathic neural response. Both men and women empathized with the pain of cooperative people. But if people are selfish, empathic responses were absent, at least in men. And it seems that physical harm might even be considered a good outcome — perhaps the first neuroscientific evidence for schadenfreude.
Nature and Science have a long history of issuing overblown press releases that extrapolate the findings of a single, quite flawed [if you side with Vul et al.] neuroimaging paper to yield the revelation of deep truths about human social interactions (among other things). The Nature News piece, Brain imaging studies under fire (Abbott, 2009), continues:
The article is scheduled for publication in September, alongside one or more replies. But the accused scientists are concerned that the impression now being established through media reports will be hard to shake after the nine-month delay. "We are not worried about our close colleagues, who will understand the arguments. We are worried that the whole enterprise of social neuroscience falls into disrepute," says neuroscientist Chris Frith of University College London, whose Nature paper [Singer et al., 2006] on response to perceived fairness was called into question.
So media reports heavily promoted the field, and media reports will unduly tarnish the field.1

NewScientist provides a clear instance of this, in what is surely a textbook exemplar of a pot-kettle moment.
Doubts raised over brain scan findings

14 January by Jim Giles

SOME of the hottest results in the nascent field of social neuroscience, in which emotions and behavioural traits are linked to activity in a particular region of the brain, may be inflated and in some cases entirely spurious.
But one doesn't have to look very far to find NewScientist headlines like these (I just searched the archives of this blog):
Watching the brain 'switch off' self-awareness

Do games prime brain for violence?

Starving is like ecstasy use for anorexia sufferers

Mirror neurons control erection response to porn

Source of ‘optimism’ found in the brain
So the NS editorial below comes across as a wee bit hypocritical, even though it eventually acknowledges their own role in promoting "sexy-sounding" brain scan results.
Editorial: What were the neuroscientists thinking?

14 January 2009

IT IS two centuries since the birth of Charles Darwin, but even now his advice can be spot on. The great man attempted a little neuroscience in The Expressions of the Emotions in Man and Animals, published in 1872, in which he discussed the link between facial expressions and the brain. "Our present subject is very obscure," Darwin warned in his book, "and it is always advisable to perceive clearly our ignorance."

Modern-day neuroscience might benefit from adopting a similar stance. The field has produced some wonderful science, including endless technicolor images of the brain at work and headline-grabbing papers about the areas that "light up" when registering emotions. Researchers charted those sad spots that winked on in women mourning the end of a relationship, the areas that got fired up when thinking about infidelity, or those that surged in arachnophobes when they thought they were about to see a spider. The subjective subject of feelings seemed at last to be becoming objective.

Now it seems that a good chunk of the papers in this field contain exaggerated claims, according to an analysis which suggests that "voodoo correlations" often inflate the link between brain areas and particular behaviours.

Some of the resulting headlines appeared in New Scientist, so we have to eat a little humble pie and resolve that next time a sexy-sounding brain scan result appears we will strive to apply a little more scepticism to our coverage.
Um, no joke guys.

On the other hand, Sharon Begley at Newsweek is one science writer who hasn't been entirely convinced by the colorful brain images. On March 10, 2008, she wrote:

Brain-imaging studies have proliferated so mindlessly (no pun intended) that neuroscientists should have to wear a badge pleading, “stop me before I scan again.” I mean, does it really add to the sum total of human knowledge to learn that the brain’s emotion regions become active when people listen to candidates for president? Or that the reward circuitry in the brains of drug addicts become active when they see drug paraphernalia?

Therefore, her recent commentary on the brouhaha does not come across as an opinion that was invented yesterday:
The 'Voodoo' Science of Brain Imaging

If you are a fan of science news, then odds are you are also intrigued by brain imaging, the technique that produces those colorful pictures of brains “lit up” with activity, showing which regions are behind which behaviors, thoughts and emotions. So maybe you remember these recent hits... [gives many examples here] . . . the list goes on and on and on. And now a bombshell has fallen on dozens of such studies: according to a team of well-respected scientists, they amount to little more than voodoo science.

The neuroscience blogosphere is crackling with—so far—glee over the upcoming paper, which rips apart an entire field: the use of brain imaging in social neuroscience.....

Before concluding, I will state that I am not a complete neuroimaging nihilist. For examples of this view, see Coltheart, 2006 and especially van Orden and Paap, 1997 (as quoted by Coltheart):
What has functional neuroimaging told us about the mind so far? Nothing, and it never will: the nature of cognition is such that this technique in principle cannot provide evidence about the nature of cognition.
So no, I am not a Jerry Fodor Functionalist. I do believe that learning about human brain function is essential to learing about "the mind," that the latter can be reduced to the former, that fMRI can have something useful to say, and (more broadly, in case any anti-psychiatry types are listening) that psychiatric disorders are indeed caused by faulty brain function. But there's still a lot about fMRI as a technique that we don't really know. The best-practice statistical procedures for analyzing functional images is obviously a contentious issue; there is no consensus at this point. Our knowledge of what the BOLD signal is measuring, exactly, is not very clear either [see the recent announcement in J. Neurosci. that "BOLD Signals Do Not Always Reflect Neural Activity."] The critics among us2 are not trying to trash the entire field of social neuroscience (or neuroimaging in general). Some of us are taking concrete steps to open a dialogue and improve its methodology, while others are trying to rein in runaway interpretations.


ADDENDUM: via Pieces of Me, I've just discovered the link to PsyBlog's detailed discussion of the Coltheart paper: Can Cognitive Neuroscience Tell Us Anything About the Mind?

Footnote

1 It isn't even necessary to quote the appropriate metaphorical expression here.

2 By "us" I mean scientists: people who are students and post-docs and colleagues of esteemed investigators like Dr. Frith.

References

Abbott A (2009). News: Brain imaging studies under fire. Social neuroscientists criticized for exaggerating links between brain activity and emotions. Nature 457:245.

Jabbi M, Keysers C, Singer T, Stephan KE. (in preparation). Rebuttal of "Voodoo Correlations in Social Neuroscience" by Vul et al. – summary information for the press. PDF

Singer T, Seymour B, O'doherty JP, Stephan KE, Dolan RJ, Frith CD. (2006) Empathic neural responses are modulated by the perceived fairness of others. Nature 439:466-9.

Vul E, Harris C, Winkielman P, Pashler H (2009). Voodoo Correlations in Social Neuroscience. Perspectives on Psychological Science, in press. PDF

Subscribe to Post Comments [Atom]

17 Comments:

At January 16, 2009 12:34 AM, Anonymous Anonymous said...

thank-you

 
At January 16, 2009 12:55 AM, Blogger The Neurocritic said...

You're welcome.

 
At January 16, 2009 2:35 AM, Anonymous Anonymous said...

Much of that article went over my head as I'm uber-tired right now, will have to spend my lunch hour digesting it and going through the links. Just wanted to say that's a cracking image you used as the header, impeccable taste :)

 
At January 16, 2009 3:25 AM, Blogger Neuroskeptic said...

Great overview, but I'm starting to feel that the whole Voodoo-gate affair is just another instance of the overblown buzz surrounding the original flawed papers. Oh,, the irony.

"Mirror neurons control erection response to porn" is of course nonsense but so are the reactions to Vul paper to the effect that it proves that "fMRI is crap". It's exactly the same phenomeon - non-experts glance at a paper which they don't fully understand, but like the gist of, and write it up as if it were the final word on the matter, with no caveats.

In this case, the caveats being that even if the Vul et. al. methodology is entirely sound, all they set out to show was that some of the correlation numbers in one genre of fMRI paper were probably too high. Most fMRI papers never calculate such numbers, and even according to Vul et. al., half of those that do so, do it right and still get quite high numbers (green list).

So Sharon Begley's gleeful claim that Vul "rips apart an entire field: the use of brain imaging in social neuroscience....." is just so much hot air. I'm always pleased to see a fellow neurocurmudgeon but that's simply not true.

Vul et. al. is a great paper. But the implications are subtle and they take expertise to grasp - like all papers. So far as I can see the best commentators on the paper have been neuroscience bloggers, who have such expertise. And I think the "red list" authors have much less to fear from bloggers than from Newsweek writers.

 
At January 16, 2009 9:51 PM, Anonymous Anonymous said...

I think the "normal academic channels" that the authors are referring to is the ability to have a reply or comment published alongside the original Voodoo article once it actually is published.

They are, somewhat, right in being miffed that they are being tried and hanged without being given the opportunity to give an official response to the accusations leveled against them.

 
At January 16, 2009 11:12 PM, Blogger The Neurocritic said...

Dr. alf,

I can see your point. In my view, the journal is at fault for blindsiding the criticized authors. However, it does seem they'll be publishing rebuttals as well. The Nature News article says:

The article is scheduled for publication in September, alongside one or more replies.

So there will be an opportunity to have the dissenting comments heard. Jabbi et al. have made a draft of their rebuttal publicly available (PDF). I've linked to it many times and did an entire post on it. I don't know whether they plan to submit it to Perspectives or not.

BPS Research Digest interviewed Matt Lieberman, who said that

...he and his colleagues have drafted a robust reply to these methodological accusations, which will be published in Perspectives on Psychological Science alongside the Pashler paper.

Bloggers have been writing about a paper that has been accepted for publication. It's neither our fault nor Vul's that the criticized authors didn't know the content of the paper until they heard about it from the press. The journal's editors and reviewers knew what they were doing when they accepted the paper. Interested parties can contact the editor (Ed Diener)

 
At January 17, 2009 1:50 AM, Blogger The Neurocritic said...

cannonballjones - Yes, the artist has done some amazing work.

Neuroskeptic - I disagree on some of your points. The Voodoo affair hasn't made it past the Newsweek blog in terms of media coverage. Was it covered in any other major outlets? Were there press releases plastered all over the place, segments on TV news shows? I don't think so (or at least, I haven't found them, so please correct me if I'm wrong). It's too esoteric; the non-independence error is not a very sexy topic.

Using a different headline than your example, try a search on optimism brain and after Nature and NewScientist you'll find foxnews, newsweek (the magazine, not the blog), abc.net.au, www.nhs.uk, reuters, sfgate, slashdot, latimes, etc. on the first two pages.

As far as ripping apart goes, the paper draws this not-so-subtle conclusion...

To sum up, then, we are led to conclude that a disturbingly large, and quite prominent, segment of social neuroscience research is using seriously defective research methods and producing a profusion of numbers that should not be believed.

...so it's not far-fetched that a reporter would call this "ripping apart an entire field." Some biologists and psychologists have been saying that fMRI is crap before this paper, so Sharon Begley is not smacking down "the use of brain imaging in social neuroscience" all on her own.

 
At January 19, 2009 6:43 AM, Anonymous Anonymous said...

the ability to have a reply or comment published alongside the original Voodoo article once it actually is published
can I just point out that this is complete tosh ? There is no academic standard anywhere that says you have a right of reply; although I grant that it is common if you write a letter criticising an individual paper.
However, this is a paper criticising a methodology; there are many papers which show particular methodologies to be flawed. I do not recall seeing that all papers using the flawed methodology get a right of reply. There are also well known papers that characterise e.g. shoddy statistics in medical journals, and again, no right of reply.

It is also worth noting that it is difficult to accept authors complaining about a questionnaire. If their paper is so badly, or so tersely, written that you cannot work out what their methods are, that is in and of itself a serious problem. Complaining about the questionnaire is shooting the messenger, who is bringing the news that their methods section is inadequate.

per

 
At January 19, 2009 5:02 PM, Blogger Yohan said...

Good post! I'm currently reading "What we can do and what we cannot do with fMRI" (Logothetis, 2006). Perhaps we need more in the popular media about what fMRI can do.

(Just playing devil's advocate. In our department "fMRI" is usually a bad word.)

 
At January 19, 2009 5:19 PM, Blogger Yohan said...

[It was Logothetis 2008. Sorry!]

 
At January 19, 2009 5:25 PM, Blogger The Neurocritic said...

No problem, Han. I know you meant this paper.

 
At January 19, 2009 5:31 PM, Blogger The Neurocritic said...

per said,

It is also worth noting that it is difficult to accept authors complaining about a questionnaire. If their paper is so badly, or so tersely, written that you cannot work out what their methods are, that is in and of itself a serious problem.

Oh Snap!

 
At January 20, 2009 1:23 PM, Anonymous Anonymous said...

The sensationalism surrounding this paper may do more bad than good.

I was recently contacted by an NIH program officer giving me a "headsup" that one of my papers may be "bogus" [program officer's word] because it is on the red-list of Vul et al. Turns out it's someone else who shares my initials. But this is worrisome because the officer's impression of me weighs into funding decisions.

 
At January 20, 2009 1:54 PM, Blogger The Neurocritic said...

Bloggers are affecting funding decisions? I didn't realize we were so powerful...

 
At January 25, 2009 8:16 AM, Blogger Unknown said...

> Anonymous:
Cheer up! By chance, you may get the funding of that quasi homonym!

Besides, has anyone here heard of someone who might have written about "[his/her] 10 favorite fMRI-papers"?

Karl Friston once asked for something similar to prepare a conference he gave with Coltheart, but I didn't follow what came out of it (https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind0806&L=SPM&P=R53378)

 
At January 27, 2009 10:30 PM, Blogger Elliot said...

It turns out that the original paper had deep statistical and methodological flaws that invalidate many if not all of the conclusions. These flaws are laid out in the invited reply:
http://www.scn.ucla.edu/pdf/LiebermanBerkmanWager(invitedreply).pdf

 
At January 28, 2009 1:20 AM, Blogger The Neurocritic said...

Elliot - I did link to the Lieberman et al. rebuttal in a new post.

 

Post a Comment

<< Home

eXTReMe Tracker