Thursday, February 07, 2008

Double-Blind Bind

An editorial on peer review was published today in Nature, and the text is reprinted in their Peer-to-Peer blog. The editors are soliciting comments and suggestions, and so far the count is up to 25 48.
Working double-blind

Should there be author anonymity in peer review?

Double-blind peer review, in which both authors and referees are anonymous, is apparently much revered, if not much practised. The Publishing Research Consortium (PRC) has assessed attitudes towards peer review among 3,000 academics in an international survey across the sciences and humanities. ...71% have confidence in double-blind peer review and 56% prefer it to other forms of review. Support is highest with those who have experienced it (the humanities and social sciences) or where it is perceived to do the most good (among female authors). The least enthusiastic group is editors. So is it time for editors, and those at Nature in particular, to reconsider their position?

If referees know the authors' identities, it may leave the latter vulnerable to biases about them or their previous work, their gender, their nationality or their being new to an area of research. But the PRC survey supports the contention of Nature and others that identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique). Knowing author identities also makes it easier to compare the new manuscript with the authors' previously published work, to ensure that a true advance is being reported. And knowing rather than guessing the identities of authors encourages reviewers to raise potential conflicts of interest to the editors.
The Neurocritic has been a proponent of completely open peer review, where the identity of the authors and the reviewers is known (see Anonymous Peer Review Means Never Having to Say You're Sorry and Peer Review Trial and Debate at Nature).

Commenter Nina Papavasiliou puts forth another interesting proposal:
Is double blind (or even open) review a cure-all? Not necessarily, and there are definitely good reasons to argue for/against both options (including but not limited to the reasons you mention). But here's another thought, that comes from being on NIH review panels: What if review is changed by having each reviewer identity and their comments blogged to one another before a decision on the paper is reached? That will a) stop reviewers from making silly comments (from asking for ridiculous experiments to killing the paper with a couple of unsubstantiated lines like "I see nothing new here" - as they will not be hiding behind anonymity in their reviewer group) and b) may even increase the quality of papers (as reviewers could, in real time, convince one another and reach consensus prior to a decision).
She also describes the disadvantages of being a junior investigator, one who would benefit from double-blind review.

Peer review can be more or less biased depending on field. I "live" in a field where it is indeed extremely biased but not against sex - rather there is direct bias (both by editors and also by reviewers) against junior faculty combined with a relatively free ride for senior faculty. Of course, this is anecdotal, but a study on the issue broken down by field will, I think, be far more informative.

Many junior faculty in my field have direct experience of this bias in peer review. How else would you call it when a paper with a set of data but junior authors is rejected by the journal (e.g. Nature) without review, when the IDENTICAL results (authored by very senior authors) are reviewed by the same Editor and published with much fanfare? (This has happened to me twice, and if you are interested I can substantiate with exact publications, dates etc).

CLARIFICATION: Dr. Papavasiliou doesn't think that being female is as great a disadvantage in her field as seniority. However, a recent paper showed that in another field (ecology and evolutionary biology), double-blind review benefits women to a significant extent (Budden et al., 2008 -- read a good summary here).

References

Budden A, Tregenza T, Aarssen L, Koricheva J, Leimu R, Lortie C. (2008). Women, Science and Writing. Trends in Ecology & Evolution 23:4-6.

Editorial. Working double-blind. Nature 451; 605-606; 2008.

Subscribe to Post Comments [Atom]

4 Comments:

At February 11, 2008 9:46 AM, Anonymous Nina Papavasiliou said...

Hi -
Just to clarify: it is not that I think that there is/isn't sex bias in my field; but I think that age bias trumps it.

Nina P.

 
At February 11, 2008 9:55 AM, Blogger The Neurocritic said...

Hi Nina P,

Thanks for the clarification. I fixed the post to reflect your comment.

 
At February 17, 2008 10:42 PM, Blogger d said...

I have a question for you that isn't directly related to this post but is perhaps in the same philosophical ballpark - first, I should say that I 'm an undergrad at a large Northeastern university, where I work as a research
assistant in the lab of a well known cognitive scientist, one of whose studies you've commented on within the last few months. I'm a regular reader of your blog and find your commentary interesting and enlightening...my
question is this: why do you remain anonymous? My guess is that
you must remain anonymous because you're in the field, and your candor wouldn't be well received by those whose studies you comment on, perhaps some of whom are colleagues and/or acquantainces. I don't ask to be critical of your choice to remain anonymous but because I'm curious why its necessary...

thanks,
dave j

 
At February 18, 2008 11:07 AM, Blogger The Neurocritic said...

Hi Dave J,

Thanks for your comment. You're correct about why I maintain anonymity: it's because I fear retaliation. As long as journals and funding agencies employ anonymous reviewers, I've decided to remain anonymous as well. From what I've heard, reviewers have been able to get away with nasty, insulting, ill-conceived attacks on good solid research they don't happen to like, mostly because it conflicts with their own results or contradicts their favorite theory. If a reviewer's identity were out in the open, they'd have to be more accountable for their remarks.

I've always thought of this blog as a new and different form of anonymous peer review...

Thanks for reading.

Regards,
The Neurocritic

 

Post a Comment

Links to this post:

Create a Link

<< Home

eXTReMe Tracker