Thursday, April 17, 2014

Post-Publication Peer Review, Systemic Changes to Biomedical Research, NIH R01A7s, and Social Media

My, that title sure is a mouthful, isn’t it? That's because in the span of a few short days, we’ve seen the following:

(1) An invited review in Neuron on the role of blogs, Twitter, Facebook, and online forums such as PubPeer, PubMed Commons, and journal comments in shaping discussions about published research papers.

(2) A commentary in PNAS about the sad structural state of biomedical research in the U.S. and  suggestions for change.

(3) A MAJOR revision in how NIH (National Institutes of Health) reviews research grants.

These communiques intersect because of how rapidly we can discuss them, who is allowed to comment, and where the debate takes place. Let me take these in turn.

The Vacuum Shouts Back: Postpublication Peer Review on Social Media

(1) An invited review in Neuron covered the role of blogs, Twitter, Facebook, and online forums such as PubPeer, PubMed Commons, and journal comments (like at PLOS ONE) in shaping discussions about published research papers (by Dr. Zen Faulkes):
Postpublication peer review through traditional scientific publishing is like kabuki theater: a slow, rehearsed drama in which the viewer must recognize the subtle profundities of performers wearing deliberately ambiguous masks.

Postpublication peer review on social media is like the mosh pit at a punk rock show. It’s fast, uncoordinated, a lot less subtle, more in your face, and involves a few more risks.

I thank Dr. Faulkes (“Dr. Zen”) for mentioning this blog and its cousin, The Neurocomplimenter.  I started the latter in response to overt dismissals of neuroscience as providing anything useful to our understanding of human cognition (and to emphasize the importance of studying brain function, more broadly). I see it as “a new project designed to counter gratuitous anti-neuroscience sentiment. It’s part of my campaign to combat pop neurobashing profiteers.”

Lately, the compliments have been sparse (perhaps due to my pessimistic nature) but I will try to highlight the positives anon. Meanwhile, the critical element soldiers on. Dr. Zen nailed my reason for starting an anonymous blog right on the head: professional peer review is anonymous, so why should bloggers identify themselves? Many do, of course, and some researchers sign their reviews.
Regardless, there are strong traditions for using both anonymity and pseudonyms in science (Neuroskeptic, 2013), not the least of which is journal peer review itself. It is a little audacious for authors and editors to decry the negative effects of “anonymous bloggers” when essentially every journal practices anonymous peer review. Bloggers are often easier to identify than journal reviewers. We still don’t know who reviewed Wolfe-Simon et al. (2011)) for Science. But we know Rosie Redfield critiqued it on her blog (, which ultimately led to a paper that failed to replicate key claims of the original paper (Reaves et al., 2012).

The voodoo correlations neuroimaging brouhaha is another major example of post-publication peer review (that went a little pre-publication due to a preprint going public), much to the dismay of the journal editor and some of the implicated researchers (see The Voodoo of Peer Review).

Dr. Zen also made the important distinction between completely anonymous commenters and those who identify as a pseudonym, a point that I often fail to make, e.g. Anonymous Peer Review Means Never Having to Say You're Sorry: “The Neurocritic is happy to provide a new form of anonymous peer review, free of charge.”

Rescuing US biomedical research from its systemic flaws

(2) A commentary in PNAS relayed the sad structural state of biomedical research in the U.S. and made [some unrealistic, some potentially helpful] suggestions for change (by four distinguished scientists in positions of power):
There is a no more worrisome consequence of the hypercompetitive culture of biomedical science than the pall it is casting on early careers of graduate students, postdoctoral fellows, and young investigators.

The commentary discusses the unsustainable growth of the biomedical research enterprise and the overly competitive yet conservative culture it has spawned. This win-at-all costs mentality squelches creativity and collaboration, and concentrates resources in the hands of fewer and fewer investigators. The average age for landing one’s first tenure-track position has risen to 37. And some estimates project that fewer than 8% of new PhDs will get tenure, with the figure an abysmal 0.45% according to one report.

The “post-doc apocalypse”1 has been widely discussed on social media (and in the popular press) for some time now. Potential solutions have been expressed by researchers at all levels, but these voices have been scattered. Those of us who are not in certain informal social circles may overlook a site of discussion somewhere, but who has time to search for it.

On the other hand, specific responses to the Alberts et al. PNAS article have already appeared on PubPeer (which allows anonymity) and PubMed Commons (which does not). Professor Dorothy Bishop made some trenchant points on the three major solutions proposed by the authors. You can read these at PubMed Commons and click on links to blog posts, where she expands on the topics of academic workload, grant review, and evaluation.

Here's my worthless $0.02 on the matter, from a position of no power and no influence: I agree with Prof. Bishop that the “Predictable and Stable Funding of Science” solution is a pipe dream...
We encourage Congressional appropriators and the executive branch to consider adding a 5-y projected fiscal plan to the current budgetary process. This plan would be updated each year, at the same time that annual appropriation bills are written.

Ha ha ha ha ha! Have they forgotten the Great Budget Sequestration of 2013 already? Getting the political parties to agree (or factions within parties) seems impossible to me. Even if all four authors win plum positions in Congress or the White House, good luck with that.

Perhaps more plausible are recommendations for downsizing the future workforce to reduce the glut of under- and unemployed junior scientists. However, they have to persuade graduate programs to admit fewer students and ban principal investigators from funding students through research grants (rather than through training grants). Some funding sources/agencies already forbid PIs from paying student stipends, and Alberts et al. propose that NIH should move towards this model.

They also suggest “Broadening the career paths for young scientists” so they're not looked down upon as failures if they don't become a clone of their advisor at MIT. This would also require changing the minds of Bob Graybeards everywhere. Good luck with that.

Another modest proposal is to force universities to pay their faculty. Here, however, they have to convince Chancellors and Deans of medical schools to forgo the “perverse incentives” of “soft money” positions, where the institution benefits from the indirect costs awarded to the university in conjunction with salary money paid by NIH. This would do away with a whole army of productive researchers. What a great idea.

Perhaps these unemployed “soft money” faculty could apply for the proposed Staff Scientist positions, displacing all the post-docs who are supposed to be in line for them. What a great idea. Good luck with that.

To be fair, let's see if the proposed Staff Scientist position might be a good idea. Post-post-docs could move into higher paying jobs that would better prepare them to run their own labs. This is because they will actually run their supervisor's lab, at much lower pay and with no prestige:
We believe that staff scientists can and should play increasingly important roles in the biomedical workforce. Within individual laboratories, they can oversee the day-to-day work of the laboratory, taking on some of the administrative burdens that now tend to fall on the shoulders of the laboratory head; orient and train new members of the laboratory; manage large equipment and common facilities; and perform scientific projects independently or in collaboration with other members of the group. Within institutions, they can serve as leaders and technical experts in core laboratories serving multiple investigators and even multiple institutions.

Or here's an idea: there could actually be a place in the system for individuals who may not want to run their own lab! These people could conduct their own independent research and help with the grant application process, if so inclined. We'd call them Staff Scientists. We'd hire different people to take on the administrative burdens and call them Lab Managers. Still others could be hired as “technical experts in core laboratories serving multiple investigators and even multiple institutions.” We'd call them Technical Experts.

But this would be prohibitively expensive. Back to cheap graduate student labor...

The most important point is that I can lobby all I want for the Snarky Policy Consultant position, since this is my blog.

NIH and AHRQ Announce Updated Policy for Application Submission

(3) A MAJOR, MAJOR revision in how the NIH reviews research grants was released today (April 17, 2014). At first I thought it was an April Fool's joke (and that I must have been dreaming for the last 16 days, which would explain a lot of things).
Effective immediately … following an unsuccessful resubmission (A1) application, applicants may submit the same idea as a new (A0) application for the next appropriate due date. The NIH and AHRQ will not assess the similarity of the science in the new (A0) application to any previously reviewed submission when accepting an application for review.

In essence, this does away with the “two strikes” rule – which meant that rejected proposals were barred from being submitted again sans a complete overhaul.

Within minutes of this announcement, Drug Monkey had 159 tweets and a blog post on this new policy, which would allow investigators to submit multiple revisions of basically the same grant (hence the R01 A7, a grant funded on the eighth try). But it wouldn't really be considered the same grant, so the numbers only go up to A1 (one amendment), thereby avoiding the stigma of A7. Thank you, NIH, that's very considerate.

And here we are again on social media, providing immediate feedback and discussion of important issues that impact the biomedical research enterprise. Can I get a full-time job doing that? We'd call it Blogger-in-Residence.


1 I refuse to use the neologism because of what Google will turn up. And this is nothing new; it's been brewing since the early 90s: “From the early 1990s, every labor economist who has studied the pipeline for the biomedical workforce has proclaimed it to be broken.”

Subscribe to Post Comments [Atom]


At April 19, 2014 3:30 AM, Anonymous Anonymous said...

Allowing unlimited resubmissions of the same grant seems good, at first sight. However, this won't only allow YOU to resubmit many times, but also everybody else. So, I predict that there will be a huge spike of less polished grant submissions and resubmissions and a corresponding increase in noise in the system. We'll see.

At April 19, 2014 1:05 PM, Blogger The Neurocritic said...

Seems like it'll be a huge burden on grant reviewers. Perhaps NIH should mandate that the grant reviewing should be commensurate with the amount of grant money received. So if you're awarded $1,000,000 in direct costs per year, you have to review something like 100 grants.

Sounds fair, eh?

At April 20, 2014 3:43 AM, Anonymous Anonymous said...

That's right, it could easily lead to a doubling of applications. Who does not have a couple of grants in their drawer that were rejected? Since it costs nothing to resubmit, then there is nothing to lose. One can always get lucky, given that the general perception is that the system is random anyway. The idea of having people with bigger grants review more grants sounds fair. However, the people who already dominate the field would end up dominating it even more.
Perhaps at some point NIH will start using an automated scoring system! Some Universities already use it for essays:

At April 20, 2014 4:58 AM, Anonymous Anonymous said...

Perhaps to deal with the flood of grants, they'll adopt this strategy:

At April 20, 2014 11:09 AM, Blogger The Neurocritic said...

To offset the undue influence of senior investigators, each grant will also be reviewed by five post-docs. And it's only a matter of time before the Dada Engine will be adapted to generate random NIH grant proposals.

All applicants and reviewers will attend the 1st ANNUAL CONFERENCE OF UNIVERSAL REJECTION. Applicants may find the Optimizing Your Grant Proposal to Ensure Rejection to be useful and life-negating, while reviewers will hone their rejection skills at the Optimizing Rejection Talent session.

At August 05, 2014 6:51 AM, Anonymous Joanne Gaudet said...

Very interesting post and comments on post-publication peer review. In my research on journal peer review, this type of review is the most likely to contribute to rational decision-making - when referees are not anonymous (full accountability) and when editorial documents are freely available (full transparency in judgement). Following are a few preprints for further reflection...

Socio-historical: – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part I. uO Research. Pp. 1-24. – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part II. uO Research. Pp. 1-20.

an abridged version – Gaudet, J. 2014. Journal peer review as scientific object of study. uO Research. Pp. 1-11.

Contemporary shaping: – Gaudet, J. 2014. All that glitters is not gold: The shaping of contemporary journal peer review at scientific and medical journals. uO Research. Pp. 1-23. – Gaudet, J. 2014. An end to ‘God-like’ scientific knowledge? How non-anonymous referees and open review alter meanings for scientific knowledge. uO Research. Pp. 1-12.

Empirically investigating purported resistance to new ideas at journal peer review: – Gaudet, J. 2014. How pre- publication journal peer review (re)produces ignorance at scientific and medical journals: a case study. uO Research. Pp. 1-67.

At August 06, 2014 9:27 AM, Blogger The Neurocritic said...

Thanks for the list of articles!


Post a Comment

Links to this post:

Create a Link

<< Home

eXTReMe Tracker