The Voodoo of Peer Review

Are blogs good or bad for the enterprise of scientific peer review? At present, the system relies on anonymous referees to provide "unbiased" opinions of a paper's (or grant's) merits. For today, the discussion will focus on peer review of papers in scientific journals.

In Seed Magazine:

That Voodoo That Scientists Do

When findings are debated online, as with a yet to be released paper that calls out the field of social neuroscience, who wins?

by Jon Bardin • Posted February 23, 2009 11:55 AM

I think science wins!

The notion that "in press" papers are off-limits to online discussion is very quaint... Discussing (and citing) such manuscripts in the "normal academic channels" is a frequent event; why should bloggers refrain from such a practice?

Bardin's article begins:

Few endeavors have been affected more by the tools and evolution of the internet than science publishing. Thousands of journals are available online, and an increasing number of science bloggers are acting as translators, often using lay language to convey complex findings previously read only by fellow experts within a discipline. Now, in the wake of a new paper challenging the methodology of a young field, there is a case study for how the internet is changing the way science itself is conducted.

Really? Maybe that's true for Biological and Social Sciences, but certainly not for Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance and Statistics (see arXiv.org, which was started in 1991 by Paul Ginsparg). Given that the field of academic physics adopted an electronic publishing model very early on, what's the role of physics journals? Well, circa 1994...

...It is ordinarily claimed that journals play two intellectual roles: a) to communicate research information, and b) to validate this information for the purpose of job and grant allocation.

As I've explained, the role of journals as communicators of information has long since been supplanted in certain fields of physics, so let's consider their other role. Having queried a number of colleagues concerning the criteria they use in evaluating job applicants and grant proposals, it turns out that the otherwise unqualified number of published papers is too coarse a criterion and plays essentially no role. Researchers are typically familiar with the research in their own field, and must in any event independently evaluate it together with letters of recommendation from trusted sources. Recent activity levels of candidates were mentioned as a criterion, but that too is independent of publication per se: "hot preprints" on a CV can be as important as any publication.

But what about peer review??

Although the arXiv is not peer-reviewed, a collection of moderators for each area review the submissions and may recategorize any that are deemed off-topic...

. . .

The lack of peer-review, while a concern to some, is not considered a hindrance to those who use the arXiv. Many authors exercise care in what they post. A majority of the e-prints are also submitted to journals for publication, but some work, including some very influential papers, remain purely as e-prints and are never published in a peer-reviewed journal.

So brand new manuscripts in those fields are placed in a public archive before they are peer-reviewed. It's not a new idea, and it's much more radical than discussing an already-reviewed manuscript that has been accepted for publication.

And that's the paper we've all come to know and love (or hate): the one by Ed Vul and colleagues on Voodoo Correlations in Social Neuroscience (PDF), summarized in this post. In their article, Vul et al. claimed that over half of the fMRI studies that were surveyed used faulty statistical techniques to analyze their data. Two rebuttals were released online shortly thereafter: one by Jabbi et al. (in preparation, PDF) and an invited reply by Lieberman et al. (submitted, PDF).

What's the problem here? It's that bloggers were writing about it! That authors and anonymous commenters somehow sullied their ideological purity by entering the free-wheeling, fast-moving world of the blogosphere. But in the modern era, why wait 5 months for a paper to be "officially" published before you're allowed to discuss it? And despite what the critics of Voodoo say, Vul et al.'s paper was not plastered all over the popular press (unlike many of the Voodoo findings themselves¹). In one of the higher-profile popular outlets, Scientific American published dueling interviews with Matthew Lieberman (left) and Ed Vul (right), with Voodoo Correlations: Have the Results of Some Brain Scanning Experiments Been Overstated? vs. In Defense of the Value of Social Neuroscience.



The only other mainstream media exposure has been from Sharon Begley of Newsweek, who covered the issue in her blog (i.e., The 'Voodoo' Science of Brain Imaging and More on Brain Voodoo) and in one of her magazine columns. But many are dubious. According to Seed:

Two groups of neuroimaging scientists, shocked by the speed with which this paper was being publicly disseminated, wrote rebuttals and posted them in the comments section of several blogs, including Begley's. Vul followed up in kind, linking to a rebuttal of the rebuttals in the comment sections of several blogs. This kind of scientific discourse — which typically takes place in the front matter of scholarly journals or over the course of several conferences — developed at a breakneck pace, months before the findings were officially published, and among the usual chaos of blog comments: inane banter, tangents, and valid opinions from the greater public.

The usual chaos of blog comments? Hello?? How about anonymous referees for journals? Are they never ever guilty of reviews filled with inane banter and tangents? We've all had exposure -- whether from our bosses, advisors, or colleagues or through our own experience -- to rude and nasty and ill-informed reviewers. And many journal editors do not rein them in. The Neurocritic has been a proponent of completely open peer review, where the identity of the authors and the reviewers is known (see Anonymous Peer Review Means Never Having to Say You're Sorry, Peer Review Trial and Debate at Nature, and Double-Blind Bind). That way, Dr. Nasty can't hide behind the shield of anonymity when making those dumb-ass comments. In the meantime, here's another possible solution, from a poll by Talking Brains:

Is it ethical for an author to post anonymous reviews of her/his paper online?

Yes	38 (48%)
No	32 (40%)
Not sure	9 (11%)

Votes so far: 79 [Poll closed ]

Back to the chaos (and the use of loaded language):

Tor Wager, a Columbia University cognitive neuroscientist, whose work was not mentioned in Vul's paper but who helped prepare one of the rebuttals, says that it was important to respond both publicly and swiftly. "The public and the news media operate on sound bites, and the real scientific issues are quite complex." His complaints focus not only on the content of Vul's paper, but also on the authors' diction — specifically, the title, and its use of "voodoo."

"When the conversation gets complex — and with statistics it always is — many blog readers will form opinions based on very simple things," says Wager. "Like words such as 'voodoo correlations.' There's no reason to use such loaded words when making a statistical argument. The argument should be able to stand on its own."

And so Ed Diener, the editor of Perspectives in Psychological Science, has stricken the offending word from the title..........

But Dr. Wager is no stranger to the media, since he has positioned his own work for widespread media coverage:

• Meta-analysis work featured in The Economist (Dec 2006)
• Interview on NPR’s Talk of the Nation (with Ira Flatow, 2007)
• TV Interview: CNN (2004); on placebo effects
• Interview for NPR’s Radio Lab series (2007); on placebo effects
• Placebo work featured in article in Wired Magazine
• Placebo work featured in article in National Geographic
• News coverage of placebo work: MSNBC, BBC, LA Times, Reuters, other news services

...and the social neuroscientists are allowed to benefit from all the fawning press coverage that overstates their results, and from the increases in funding that follow. But when one -- ONE -- paper dares to criticize some of the methodology, that's dangerous. AND IT MUST BE STOPPED! Because we all know that bloggers influence funding decisions.

But...

Vul has a different take. Once their paper had passed peer review, Vul and his colleagues argue, it was the public's right to read about it, and respond to it, however they chose — especially given that it sought to reveal flaws in publicly funded research that gets widespread media coverage.

Needless to say, I agree with this point, but I've also publicized the rebuttals as well (as noted in the Seed article):

Comment wars
Starting January 12, anonymous commenters on the Neurocritic's blog posted rebuttals from Vul's detractors; and Vul followed with rebuttals of the rebuttals.

Go bloggers! And now it's official: Nature declares that It's good to blog!

More researchers should engage with the blogosphere, including authors of papers in press.

Is blogging a part of science, journalism or public discourse? In fact it may be all of these — an ambiguity that can sometimes leave scientists feeling uncertain about the rules of the game.

...but only up to a point:

The blogosphere differs from mass media and specialized media in many respects, but the same considerations apply in disseminating new scientific results there. Authors of papers in press have the right to correct misrepresentations and to point to results that will appear in a paper. But a full discussion should await the paper's publication.

Nonetheless, they end on a positive note:

Indeed, researchers would do well to blog more than they do. The experience of journals such as Cell and PLoS ONE, which allow people to comment on papers online, suggests that researchers are very reluctant to engage in such forums. But the blogosphere tends to be less inhibited, and technical discussions there seem likely to increase.

Moreover, there are societal debates that have much to gain from the uncensored voices of researchers. A good blogging website consumes much of the spare time of the one or several fully committed scientists that write and moderate it. But it can make a difference to the quality and integrity of public discussion.

Many of us believe that is true. And with that, it's time for me to go to bed.